Xjyuk

I already understood the mathematical definition of a consistent estimator. Correct me if I'm wrong:

$W_n$ is an consistent estimator for $theta$ if $forall epsilon<0$

$$lim_ntoinfty P(|W_n - theta|> epsilon) = 0, quad foralltheta in Theta$$

Where, $Theta$ is the Parametric Space.  But I want to understand the need for an estimator to be consistent. Why an estimator that is not consistent is bad? Could you give me some examples?

I accept simulations in R or python.

Consider $n = 10,000$ observations from the standard Cauchy distribution,
which is the same as Student's t distribution with 1 degree of freedom.
The tails of this distribution are sufficiently heavy that it has no
mean; the distribution is centered at its median $eta = 0.$

A sequence of sample means $A_j = frac 1j sum_i=1^j X_i$ is not consistent
for the center of the Cauchy distribution. Roughly speaking, the difficulty
is that very extreme observations $X_i$ (positive or negative) occur with
sufficient regularity that there is no chance for $A_j$ to converge to $eta = 0.$ (The $A_j$ are not just slow to converge, they don't ever converge. The distribution of $A_j$ is again standard Cauchy [proof].)

By contrast, at any one step in a continuing sampling process, about half
of the observations $X_i$ will lie on either side of $eta,$ so that the sequence $H_j$ of sample medians does converge to $eta.$

This lack of convergence of $A_j$ and convergence of $H_h$ is illustrated
by the following simulation.

set.seed(2019) # for reproducibility
n = 10000; x = rt(n, 1); j = 1:n
a = cumsum(x)/j
h = numeric(n)
for (i in 1:n) 
 h[i] = median(x[1:i]) 
par(mfrow=c(1,2))
 plot(j,a, type="l", ylim=c(-5,5), lwd=2,
 main="Trace of Sample Mean")
 abline(h=0, col="green2")
 k = j[abs(x)>1000] 
 abline(v=k, col="red", lty="dotted")
 plot(j,h, type="l", ylim=c(-5,5), lwd=2,
 main="Trace of Sample Median")
 abline(h=0, col="green2") 
par(mfrow=c(1,1))

Here is a list of steps at which $|X_i| > 1000.$ You can see the effect
of some of these extreme observations on the running averages in the plot at left (at the vertical red dotted lines).

k = j[abs(x)>1000]
rbind(k, round(x[k]))
 [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
k 291 898 1293 1602 2547 5472 6079 9158
 -5440 2502 5421 -2231 1635 -2644 -10194 -3137

Consistency in important in estimation: In sampling from a Cauchy population, the sample mean of a sample of $n = 10,000$ observations is no better for estimating the center $eta$ than just one observation. By contrast, the consistent sample median converges to $eta$ so larger samples produce better estimates.

If the estimator is not consistent, it won't converge to the true value in probability. In other words, there is always a probability that your estimator and true value will have a difference, no matter how many data points you have. This is actually bad, because even if you collect immense amount of data, your estimate will always have a positive probability of being some $epsilon>0$ different from the true value. Practically, you can consider this situation as if you're using an estimator of a quantity such that even surveying all the population, instead of a small sample of it, won't help you.

A really simple of example of why it's important to think of consistency, which I don't think gets enough attention, is that of an over-simplified model.

As a theoretical example, suppose you wanted to fit a linear regression model on some data, in which the true effects were actually non-linear. Then your predictions cannot be consistent for the true mean for all combinations of covariates, while a more flexible may be able to.