The 99.4% Accurate Model That Caught Zero Fraud

Let me tell you about the fraud detection model I ever built. It achieved 99.4% accuracy. It required no training, no GPU, no feature engineering, no hyperparameter tuning, no 2am Slack messages about drifting distributions.

Here’s the model, in production-ready Python:

def lazy_model(transaction):    return "not fraud"

That’s it. That’s the model. It looks at a transaction with amount, location, time, whether the card was swiped from a different continent five minutes after being swiped at Super Market, and confidently declares: not fraud. every single time.

And it’s right 99.4% of the time.

It also catches exactly zero fraud cases, ever, in the history of the universe.

Welcome to the accuracy trap!

The setup

Say you’ve got 20,000 transactions. Fraud, thankfully, is rare, only about 0.6% of them, or 120 transactions, are actually fraudulent. The other 19,880 are perfectly boring, legitimate purchases of coffee, SaaS subscriptions, and regrettable late-night impulse buys.

This is a wildly imbalanced dataset. And imbalanced data is where accuracy quietly turns into a liability rather than a virtue. If 99.4% of your labels are “not fraud,” then a model that just shouts “NOT FRAUD” at everything walks in wearing a 99.4% accuracy badge, and does precisely nothing useful.

Accuracy, as a metric, is asking one question: “How often was I correct overall?” That’s a perfectly fine question when your classes are balanced and every mistake costs about the same. It is a catastrophically bad question when 99.4% of the answer key says the same thing and the 0.6% you’re ignoring is the entire reason you built the model.

Meet the Lazy Model’s report card

Here’s what the Lazy Model’s evaluation looks like:

Metric	Score	Translation
Accuracy	99.4%	Looks fantastic on the dashboard
Precision	0.0%	Never raises an alert, so it’s never “right” about fraud either
Recall	0.0%	Catches nothing. The money just… leaves
F1 Score	0.0%	No balance to speak of, because there’s nothing to balance

This is the part where, if you only glanced at the accuracy number in a stakeholder deck, you’d approve this model for production and go get lunch. The dashboard is green. Everyone’s happy. The fraud team, meanwhile, is quietly wondering why the “AI-powered” system hasn’t flagged a single transaction in three weeks.

So what should we actually be asking?

Two better questions, and they matter more than accuracy the moment your positive class gets rare or your mistakes stop being equally priced:

Precision asks: “Of everything I flagged as fraud, how much was actually fraud?”

Precision = TP / (TP + FP)

Low precision means your fraud investigation queue is full of confused, innocent customers who just tried to buy plane tickets and got blocked. High precision means when the model raises its hand, it’s usually right, and your fraud analysts aren’t drowning in noise.

Recall asks: “Of all the actual fraud out there, how much did I catch?”

Recall = TP / (TP + FN)

This is the “did the money walk out the door” metric. Low recall means fraud is slipping through the cracks you didn’t know you had. In rare-event detection, recall is often the metric that determines whether your model earns its keep or just earns you an awkward postmortem.

The confusion matrix, decoded for people who hate confusion matrices:

• True Negative: normal transaction, correctly approved. Nobody notices, which is the point.
• False Positive: a real customer wrongly blocked. Annoying, generates support tickets, but survivable.
• False Negative: fraud that slipped through undetected. This is usually the expensive one.
• True Positive: fraud, caught. The whole reason you’re doing this.

Now let’s build a better model

Time to retire the Lazy Model and bring in logistic regression, given a few real features to work with transaction amount, hour of day, recent transaction velocity, and one important setting:

clf = LogisticRegression(    class_weight="balanced", max_iter=1000)

class_weight="balanced" is the line doing the real work here. It tells the model: hey, I know fraud is only 0.6% of what you’ll see, but don’t just learn to ignore it because it’s rare pay attention.

Here’s where things get interesting:

	Lazy Model	Real Model
Accuracy	99.4%	85.3%
Recall	0.0%	77.8%

Look at that table for a second. On the headline metric accuracy the Lazy Model wins. It’s not close. 99.4% versus 85.3%. If your only KPI is accuracy, you ship the model that does nothing.

But the real model catches roughly 78% of actual fraud. The “worse” model, by the metric everyone defaults to, is the one actually doing the job.

This is the entire lesson in one table: the model with lower accuracy is the better fraud fighter.

Precision and recall are in a tug-of-war

Most classifiers don’t output a hard yes/no they output a probability, and a threshold decides where “fraud” starts. Slide that threshold around and you get a tradeoff:

• Loose threshold: catch more fraud (recall goes up), but more false alarms (precision goes down).
• Strict threshold: fewer false alarms (precision goes up), but more subtle fraud slips through (recall goes down).

There is no mathematically perfect threshold. There’s only the threshold that matches your business’s tolerance for the two kinds of pain: annoying a legitimate customer, versus letting fraud through. A bank might set one threshold for a $12 coffee purchase and a completely different one for a $40,000 wire transfer. This is why threshold tuning isn’t purely a modeling exercise, it’s a conversation with risk and ops, not just a line in a notebook.

The cheat sheet, for your next model review

Metric	Question it answers	Watch out when…
Accuracy	How often am I right overall?	Classes are imbalanced
Precision	When I flag something, am I right?	False alarms are expensive
Recall	Of all real positives, how much did I catch?	Missed cases are expensive
F1 Score	How balanced are precision and recall?	You need one summary number

The takeaway

A model that’s right 99.4% of the time can still be completely, uselessly wrong about the one thing you built it for.

Accuracy isn’t a bad metric. it’s an incomplete one, and imbalanced problems (fraud, churn, disease screening, defect detection anywhere the interesting class is rare) expose that incompleteness ruthlessly. Before you ship anything, ask which mistake is more expensive for your business, and pick metrics that actually reflect that not just the number that looks best on the dashboard.