Naive Bayes as a classifier

Bayes' theorem can be turned into a working classifier. Naive Bayes has been filtering spam for decades and is still strong as a fast baseline. It directly applies everything we know about conditional probability and the base rate.

The email = the selected words. Click to add/remove a word:

odds = 0.30/0.70 × 15.0(prize) × 5.0(urgent)

P(spam | email) = 97.0% → spam

Prior share of spam P(spam)30%

Each spam word (LR>1) pushes the probability up; a "business" word (LR<1) pushes it down. The words multiply independently — hence "naive". The base rate P(spam) is the starting odds: when spam is rare, even strong words do not make the email spam for sure.

The task: an email with words — spam or not? Bayes flips the question. We want P(spam | words), but what is measurable is the reverse: P(words | spam) — how often these words appear in spam. Bayes' theorem links them through the base rate P(spam) — how frequent spam is at all.

you can flip beats with ← → keys

What it means

Naive Bayes is an excellent baseline for text (spam, sentiment, topics): fast, cheap, needs little data. It is a sensible starting point before reaching for heavy models.

Its weakness is exactly the correlated features: it estimates probabilities with bias (often overconfident), so as a SOURCE of probabilities it is worse than as a DECISION-making classifier. If you need calibrated probabilities — calibrate them separately.

Where it shows up

Classic spam filters (Bayesian filtering) are the direct application: the "spam" probability is updated by the email's words. The same trick works for simple classification of reviews and news by topic.

In medicine and diagnostics, the same "update the base rate with observations" logic underlies probabilistic screening rules.

Definitions

Naive Bayes

a classifier built on Bayes' theorem with the assumption of feature independence given the class.

Independence assumption

features (words) are treated as independent within a class — almost always false, but convenient and it often works.

Feature likelihoodP(word | class)

how often a feature occurs in a given class; estimated as shares from the training data.

Smoothing

a small addition to the counts so a word unseen in training does not zero out the whole product.

When the method lies (assumptions)

The "naivety" is the assumption that features are independent within a class. When features duplicate each other (email length and word count), the classifier counts one piece of evidence twice and becomes overconfident: its "probabilities" cannot be read literally.

A word never seen in a class during training gives zero probability and zeroes out the whole product — you need smoothing (Laplace correction). And predictions depend on base rates: under a new class balance they drift.