CAIA Chapter 7: Hypothesis Testing in Alternative Investments

is the behavior that the analyst assumes would be true if the null hypothesis is rejected. The null and the alternative hypotheses are usually stated in such a way that they are mutually exclusive.

which is a function of observed values of the random variables and typically has a known distribution under the null hypothesis. The test statistic is the variable that is analyzed to make an inference with regard to rejecting or failing to reject the nul

are often used interchangeably to indicate the probability that a result may be due to randomness, in the case of the significance level, or not due to randomness, in the case of the confidence level (sig=1-5%; conf=95-99%)

indicated that there is only a 2% chance that the estimated value would occur given the assumption that the null hypothesis is true

describes the extent to which a variable in an economic model has a meaningful impact on another variable in a practical sense. One can be very confident about significance but size and dispersion of parameter may indicate that parameter has minor impact

false positive is when an analyst makes the mistake of falsely rejecting a true null hypothesis

also known as false negative is failling to reject the null when it is falese

eg. examine the probability that a trader has actually cheated, given that we have rejected the null

is a fake or false indication that there is a true underlying relationship between two variables when there is only a temporary and coincidental association

is a distortion in relevant sample characteristics from the characteristics of the population, caused by the sampling method of selection or inclusion

if the bias originates from the decision of fund managers to report or not to report their returns, than the bias is refered to as a self-selection bias

is a common problem in investment databases where the sample is limited to those observations that continue to exist through the end of the period of study

typcially refers to the vigorous use of data to uncover valid relationships. The idea is that by using a variety of well-designed statistical tests and exploring a number of data sources, analysts may uncover previously missed relationships

or data snooping refers to the overuse and misuse of statistical tests to identify historical patterns

is the use of historical data to test a strategy that is selected subsequent to the observation of the data. Can be valid however, backtesting with data dredging using too many hypothetical strategies, can generate false indications of future returns

is usisng too many parameters to fit a model very closely to data over some past time frame. Models that have been overfit may have a smaller chance to fit future data than a model using fewer and more generalized parameters

typically refers to the insertion of an actual trading record of an investment into a database.

or instant history bias, is when the funds, returns and strategies being added to the data set are not representative of the universe of fund managers, fund returns and fund strategies

is the concept of extracting only those results that support a particular viewpoint

is an observation that is much further from the mean than almost all other observations. Outliers tend to have large impacts on results, and an exceptionally unsusual outlier may severely distort the measurement of economic tendencies.

reflects when one variable's corelation with another variable is because the one variable's value partially or fully determines the value of the other variable