Maximum likelihood estimation of normal distribution

The probability density function of normal distribution is: $$f\left(x\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$$

Support we have the following n i.i.d observations: $x_1,x_2,\dots ,x_n$. Because they are independent, the probability that we have observed these data are: $$f(x_1,x_2,\dots ,x_n|\sigma ,\mu )=\prod _{i=1}^n\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}}=(\frac{1}{\sigma \sqrt{2\pi }}{)}^n{e}^{-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2}$$

$$\begin{array}{cc}\log \left(f\right(x_1,x_2,\dots ,x_n|\sigma ,\mu \left)\right)& =\log \left(\right(\frac{1}{\sigma \sqrt{2\pi }}{)}^n{e}^{-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2})\\ & =n\log \frac{1}{\sigma \sqrt{2\pi }}-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2\\ & =-\frac{n}{2}\log \left(2\pi \right)-n\log \sigma -\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2\end{array}$$

Let’s call $\log \left(f\right(x_1,x_2,\dots ,x_n|\sigma ,\mu \left)\right)$ as $L,$ then let: $$\frac{dL}{d\mu }=-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2{\mid }_{\mu }=0$$ solve this equation, we get $$\frac{1}{2{\sigma }^2}\sum _{i=1}^n(2\hat{\mu }-2x_i)=0$$

Because ${\sigma }^2$ should be larger than zero, $$\hat{\mu }=\frac{{\sum }_{i=1}^n{x}_i}{n}$$

Similarly, let $$\frac{dL}{d\sigma }=-\frac{n}{\sigma }+\sum _{i=1}^n(x_i-\mu {)}^2{\sigma }^{-3}=0$$

I realized that it would be better to get the maximum likelihood estimator of ${\sigma }^2$ instead of $\sigma$. Thus

$${\hat{\sigma }}^2=\frac{{\sum }_{i=1}^n(x_i-\hat{\mu }{)}^2}{n}$$

But this MLE of ${\sigma }^2$ is biased. A point estimateor $\hat{\theta }$ is said to be an unbiased estimator of $\theta$ is $E\left(\hat{\theta }\right)=\theta$ for every possible value of $\theta$. If $\hat{\theta }$ is not unbiased, the difference $E\left(\hat{\theta }\right)-\theta$is called the bias of $\hat{\theta }$.

We know that $${\sigma }^2=Var\left(X\right)=E\left(X^2\right)-\left(E\right(X){)}^2\Rightarrow E(X^2)=Var(X)+(E\left(X\right){)}^2$$

Then $$\begin{array}{cc}E\left({\hat{\sigma }}^2\right)& =\frac{1}{n}E\left({\sum }_{i=1}^n\right(x_i-\hat{\mu }{)}^2)\\ & =\frac{1}{n}E(\sum x_i^2-n{\hat{\mu }}^2)\\ & =\frac{1}{n}E(\sum x_i^2-\frac{(\sum x_i{)}^2}{n})\\ & =\frac{1}{n}\left\{\sum E\left(x_i^2\right)-\frac{1}{n}E\left[(\sum x_i{)}^2\right]\right\}\\ & =\frac{1}{n}\left\{\sum ({\sigma }^2+{\mu }^2)-\frac{1}{n}\left[n{\sigma }^2+(n\mu {)}^2\right]\right\}\\ & =\frac{1}{n}\left\{n{\sigma }^2+n{\mu }^2-{\sigma }^2-n{\mu }^2\right\}\\ & =\frac{n-1}{n}{\sigma }^2\\ & \ne {\sigma }^2\end{array}$$

Bias is $E\left({\sigma }^2\right)-{\sigma }^2=-\frac{{\sigma }^2}{n}$. In fact the unbiased estimator of ${\sigma }^2$ is $s^2=\frac{{\sum }_{i=1}^n(x_i-\hat{\mu }{)}^2}{n-1}$. But the fact that $s^2$ is unbiased does not imply that $s$ is unbiased for estimating $\sigma$. The expected value of the square root is not the square root of the expected value. Fortunately, the biase of $s$ is small unless the sample size is very small. Thus there are good reasons to use $s$ as an estimator of $\sigma$.