Naive Bayes

Tags: #◇Computer Science #◇CSINTSY #◇Naive Bayes #◇Problem Solving #◇M4

---

◇Overview

Naive Bayes is a statistical supervised learning model that leverages Bayes' Theorem for classification. The core assumption that makes the model "naive" is the assumption of conditional independence between all input features given the target class. Even though features are rarely completely independent in real life, this assumption greatly simplifies computation and often works surprisingly well.

Bayes' Theorem Framework:
Prior: The probability of a target class occurring, $P(\text{Class})$.
Likelihood: The probability of observing the features given a target class.
Evidence/Marginal: The total probability of observing the features, which is often ignored during classification since it's the same across all target classes.
Posterior: The final computed probability of a class given the features.

◇Important Formulas

Bayes' Theorem:
$P(T | F) = \frac{P(F | T)P(T)}{P(F)}$
Where $T$ is the target class and $F$ is the observed features.

Naive Bayes Assumption (Feature Independence):
For features $X_1, X_2, \dots, X_d$:
$P(X_1, X_2, \dots, X_d | T) = P(X_1 | T) \times P(X_2 | T) \times \dots \times P(X_d | T)$

Continuous Features (Gaussian Naive Bayes):
For numerical features, the likelihood is computed using the probability density function of a normal distribution defined by the mean $\mu$ and standard deviation $\sigma$ of the feature for that class.

◇Worked Examples

Categorical Classification:
Given training data for student grades and study hours predicting a pass/fail outcome.

1. Compute Prior for passing: $P(Pass) = \frac{N_{pass}}{N_{total}}$
2. Compute Likelihood of features for a Pass: $P(\text{Math=Bad} | Pass) \times P(\text{Study} \ge 4 | Pass)$
3. Calculate posterior score by multiplying Prior and Likelihood. Repeat for the Fail class. Predict the class with the highest score.

◇References & Navigation

• Previous Topic: ◇Decision Trees
• Next Topic: ◇K-Means Clustering
• Related Notes: ◇Naive Bayes, ◇Bayesian Inference