LOS k: Discuss the issues regarding selection of the appropriate sample size, data-mining bias, sample selection bias, survivorship bias, look-ahead bias, and time-period bias.

A larger sample reduces the sampling error and the standard deviation of the sample statistic around its population value. However, this does not imply that the sample should be as large as possible, or that the sampling error must be as small as can be achieved. Larger samples might contain observations that come from a different population, in which case they would not necessarily improve the estimates of the population parameters. Cost also increases with the sample size. When the cost of increasing the sample size is greater than the value of the extra precision gained, increasing the sample size is not appropriate.

When you are sampling from a:	and the sample size is small, use a:	and the sample size is large, use a:
Normal distribution with a known variance	z-statistic	z-statistic
Normal distribution with an unknown variance	t-statistic	t-statistic*
Nonnormal distribution with a known variance	not available	z-statistic
Nonnormal distribution with an unknown variance	not available	t-statistic*

*The z-statistic is theoretically acceptable here, but use of the t-statistic is more conservative.

There is no sample statistic for non-normal distributions with unknown variance for small samples, but the t-statistic is used when the sample size is large.

Studies of the performance of mutual fund managers often suffer from survivorship bias as poorly performing funds are closed down and are not included in the sample.

Data mining is the process where the same data is used with different methods until the desired results are obtained.

The practice of data mining involves analyzing the same data so as to detect a pattern, which may not replicate in other data sets, also known as torturing the data until it confesses.

Data mining occurs when the analyst continually uses the same database to search for patterns or trading rules until he finds one that works. If you are reading research that suggests a profitable trading strategy, make sure you heed the following warning signs of data mining:

Evidence that the author used many variables (most unreported) until he found ones that were significant.

The lack of any economic theory that is consistent with the empirical results.

If we try to draw any conclusions from an analysis of a mutual fund database with survivorship bias, we overestimate the average mutual fund return, because we don’t include the poorer-performing funds that dropped out. A larger sample size from a database with survivorship bias will still result in a biased estimate.

Time-period bias can result if the time period over which the data is gathered is either too short because the results may reflect phenomenon specific to that time period, or if a change occurred during the time frame that would result in two different return distributions. In this case the time period sampled is probably not large enough to draw any conclusions about the long-term relative performance of value and growth stocks, even if the sample size within that time period is large.

Look-ahead bias occurs when the analyst uses historical data that was not publicly available at the time being studied. Survivorship bias is a form of sample selection bias in which the observations in the sample are biased because the elements of the sample that survived until the sample was taken are different than the elements that dropped out of the population.

The study suffers from look-ahead bias because traders at the beginning of the year would not be able to know the book value changes. Financial statements usually take 60 to 90 days to be completed and released.

Evidence that the researcher used data mining is that he was defensive about the lack of economic theory consistent with his results and that he used the same database of data for all his tests. One way to avoid data mining is to test the trading rule on out-of-sample data.  Sample selection bias occurs when some data is systematically excluded from the analysis, usually because it is not available. Here, the researcher excluded stocks for which he could not determine the CEO’s birthday. Time-period bias can result if the time period is too short or too long. Here, it is likely that the period was too short since the researcher used a cut-off date of the month before the latest market correction. Note: this could be an additional example of data mining.

We are not given enough information to determine if the researcher is guilty of look-ahead bias (which occurs when the analyst uses historical data that was not publicly available at the time being studied).

Data mining refers to the extensive review of the same database searching for patterns.

The best way to avoid data mining is to test a potentially profitable trading rule on a data set different than the one you used to develop the rule (out-of-sample data). A larger sample size won’t prevent data mining, and you can still data mine a database free of survivorship bias.