Bias is the systematic error of an estimator: whether it tends to over/under estimate the true parameter across repeated samples.

For estimator $\hat\theta$ of a parameter $\theta$, bias is defined as:

$$ \operatorname{Bias}(\hat{\theta})

\mathbb{E}[\hat{\theta}] - \theta $$

where:

• if bias is $0$, the estimator is unbiased
• if bias is positive, it overestimates on average
• if bias is negative, it underestimates on average

For example, the sample mean $\bar X$ is unbiased for the population mean $\mu$:

$$ \Bbb E[\bar X]=\mu \implies \text{Bias}(\bar X)=0 $$

Bias concerns average behavior over many possible samples, not the error of one particular estimate. An estimator can be biased yet useful if its lower variance producers a lower overall mean squared error:

$$ \text{MSE}(\hat\theta)=\text{Var}(\hat\theta)+\text{Bias}(\theta)^2 $$

References

Mean squared error

Estimator

Variance