SRaI

• Corr(Yj, Yk) = Cov(Yj, Yk) / sqrt(Var(Yj) * Var(Yk))

\(E(Y) = E_X(E(Y|X))\\Var(Y) = E_X(Var(Y|X) + Var_X(E(Y|X))\)

• sum of any distributed random variable converges to normal distribution
  • for n -> infinity (asymptotically)
• conditions
  • i.i.d.
  • mean given
  • finite variance
• example: random-walk: Yn = sum(x_i) 
  • Z_n = (Y_n) / sqrt(n) ~ N(0, sigma^2)
  • with Y_n  sum(x_i)

• \(M_Y(t) = E(e^{t Y})\)
• \(\frac{\partial^kM_Y(t)}{(\partial t)^k} = E(Y^k) \) evaluated at t = 0
• \(K_Y(t) = log(M_Y(t))\)
  • first derivative w.r.t. t: E(Y) = first moment
  • 2nd derivative w.r.t. t = Var(Y) = 2nd moment
  • as long as moments are finite

• y (data) is realisation of Y with Y~F(y; theta) (model)
• theta unknown
• \(f_\theta (\vartheta|y) = \frac{(\prod f(y_i; ~\vartheta))f_\theta(\vartheta)}{\int{\prod f(y;~\vartheta)f_\theta(\vartheta)d\vartheta}}\)
  • posterior = all information about theta
  • likelihood: information in data
  • prior: knowledge about theta before observing the data
  • denominator: normalisation constant (f(y), independent of theta) 

• \(L(\theta;y) = \prod f(y_i; \theta)\)
• \(l(\theta;y) = \sum log~f(y_i; \theta)\)

Characteristics

• plausibility of parameter-values theta, given data y
• posterior proportional to Likelihood * prior

• posterior mean estimate
  • theta_hat = E(posterior)
  • requires (numerical) integration (E(.))
• posterior mode estimate
  • theta_hat = max(posterior)
  • first derivative required
• ML-estimation
  • theta_hat = max(L(theta; y)) (set first derivative w.r.t. theta of loglikelihood to zero)
  • assumes flat/constant/non-informative prior
• posterior mode estimate = ML-estimation if prior is flat

• gamma = g(theta), theta = g^-1(gamma)
• gamma_ML = g(theta_ML) >> no reestimation required after transformation
• g(.) bijective transformation function
• counts also for Variance s.t. gamma_hat - gamma ~ N(0, derivative of theta wrt gamma * inv(I(theta)) * derivative of theta wrt gamma)

• L: set of t   x  parameter-space -> R+
• L(t, theta) = (t-theta)^2 >> min
• L(theta, theta) = 0

• t(y) = theta_hat depens on sample
  • risk = expected loss (R(t, theta) = E(L(.))
  • estimate theta_hat s.t. R(.) is minimised
• theta unknown (risk and loss depend on theta)
  • minimax approach: theta_hat = chose t(y) of theta with maximal risk s.t. risk is minimized
  • bayes risk: minimise Expectation of risk w.r.t. theta
  • posterior bayes risk: minimise Expectation of Loss given the data

• R(t, theta) = E((t(y) - theta)^2) = expectation of squared loss
  • = Var(t(y)) + Bias^2(t, theta)
  • = stochastic error (variability of t) + systematic error

• Bias(t, theta) = E(t(y)) - theta
• aim: asymptotically unbiased estimate s.t. Bias = 0 for whole parameter space

• compares distributions
• not symmetric
• log(f(y; theta)) - log(f(y; t))) = log(f(y; theta) / f(y; t))
  • theta -> true distribution
  • t -> estimated distribution
• 0 if t = theta else >= 0
• KL(t, theta) = \(\int log\frac{f(y; \theta)}{f(y; t)} f(y; \theta)dy\)
• E(KL) = R(t, theta) = integral(I(theta) * (t - theta)^2 * prod(f(y;theta))
  • fisher-information independent of data
  • >> min(KL-risk) approximates min(MSE)

• quality of estimate t(y)

• all information about theta in t() 
  • idea: replace data y by t(y)
• if f(y | t(y) ; theta) is independent of theta
• problem: hard to show

• sufficient if f(y; theta) = h(y) * g(t(y); theta)
  • h() and g() non-negative
  • >> h() independent of theta
  • >> g() depends on y only through t()
• weak statement as y is already sufficient

• t() is sufficient AND
• it exists t*() s.t. t() = h(t*(.))

• quantify information in data
• theta_hat is consistent if MSE(theta_hat, theta) -> 0 for n -> inf
  • >> theta_hat -> theta (asymptotically)

• lower limit of MSE for given n
• holds only for fisher regular distributions
• for unbiased theta_hat: MSE() >= inv(I(theta))
  • Var(theta_hat) >= inverse of fisher-information (= in best case)
• for biased theta: MSE() >= Bias()^2 + (. = 1 if bias = 0)^2/I(theta)
  • smaller variance possible, but larger bias then

• quantify uncertainty of theta_hat
• interval estimate instead of point estimate
  • P(theta in CI) >= 1-alpha (-> =)

• g(y; theta) independent of theta
• i.e. x = (theta_hat - theta) / sqrt(Var(theta_hat)) ~ N(0,1) 
  • pivotal statistics by using CLT
  • P(z_alpha/2 <= x <= z_1-alpha/2) = 1-alpha
  • CI = [theta_hat +- z_1-alpha/2 * sqrt(Var(theta_hat))]
    • problem: Var(theta_hat) depends on theta >> circle >> estimate Var(theta_hat)

• bayesian approach using the posterior
• P(theta in CI | y) ) = integral from left bound to right bound over posterior >= 1-alpha
  • integral of posterior from -inf to left bound = integral of posterior from right bound to inf = alpha/2
  • >> cut left and right probability mass of alpha/2 

• HDI(y) = {theta; posterior >= c)
• c s.t. integral over posterior with all thetas of HDI = 1-alpha
  • >> cut from top (posterior of left and right bound same)

• different approaches/reasoning, similar results

• Expectation(- 2nd derivative of log-likelihood w.r.t. theta)
  • Expectation of observed information
• measures the amount of information that y carries about theta (increases with n)
  • variance of score
  • reciprocal variance of estimate (lowest possible variance of unbiased estimator)
  • >> central role in asypmtotical theory of ML-estimation
  • >> can be used for tests (i.e. Wald-test)

• support of y is independent of theta
• parameter-space of theta is open
• f(y; theta) is twice differentiable wrt theta
• integration and differentiation are exchangeable

• fisher regularity
• Y_i ~ f(y; theta) iid

• first derivative of l(theta; y) w.r.t. theta = s(theta; y)
• is used for ML-estimation: s(theta_hat) = 0 (max of l(theta;y))
• s(theta; y) ~ N(0, I(theta))
  • approx.
  • proof
    • 1st Barlett identity: 1 = integral(f(y; theta)) -> diff() -> E(s) = 0
    • 2nd Barlett identity: 0 = E(s) -> diff -> E(s^2) = I(theta) = Var(s)

• l(theta; y) depends on sample (maxima around theta)
  • aim: quantify expectation and uncertainty of theta_hat
• >> asymptotic normality of theta_hat
  • theta_hat ~ N(theta, inv(I(theta)) (approx.)
  • proof:
    • define s_n, I_n etc.
    • TLS around s_n at theta
    • >> theta_hat - theta = - s_n / s'_n = inv(I_n) * s_n
    • theta_hat - theta ~ N(0, inv(I_n)) as s_n ~ N(0, I_n)

• theta_t+1 = theta_t - s / s' = theta_t + s / I
• 0) theta_t = theta_zero
• 1) theta_t+1 = theta_t + s(theta_t) / I(theta_t)
• 2) stop if ||theta_t+1 - theta_t|| < d
• 3) theta_ML = theta_t+1
• >> can end up in local optimum
  • different theta_zero
  • adapt step-size

• idea: answer questions based on data
• type-one error: "H1" | H0, reject H0 even it is true
• type-two error: "H0" | H1, accept H0 even it is false
• problem: errors are complementary: no type1 error by always "H0" i.e.

• P("H1" | H0) <= alpha (bound for type one error)
• P(y_bar > c | mu <= mu_zero) <= alpha
  • >> pivotal statistic >> c = mu_zero + z_1-alpha * sigma / sqrt(n)
    • with z_1-alpha = 1-alpha quantile of N(0, 1)

• one sided test: H0: mu <= mu_zero, H1: mu > mu_zero
  • "H1" <> y_bar > c
• two-sided test: H0: mu = mu_zero, H1: mu != mu_zero
  • "H1" <> |y_bar - mu_zero| > c
• testing theta: H0: theta in theta-zero-space, H1: theta not in theta-zero-space
• problem: for calculation of c, sigma is required (mostly unknown)
  • >> estimate sigma_hat >> (y_hat - mu_zero) / (sigma_hat / sqrt(n)) ~ t_n-1 distributed

• all test: H0: theta = theta_zero, H1: theta != theta_zero
• Wald-test
  • "H1" <> |theta_hat - theta_zero| > c (large deviation of thetas speak against H0)
  • c = z_1-alpha/2 * sqrt(inv(I(theta_zero))) >> theta_hat instead of theta_zero used as theta_hat ~ theta_zero under H0
  • >> estimation of theta_hat and variance of estimate required
• Score-test
  • "H1" <> |s(theta_zero; y)| > c (large score speaks against H0)
  • c = z_1-alpha/2 * sqrt(I(theta_zero))
  • >> score of theta_zero is enough, no theta_hat required
• likelihood-ratio test
  • "H1" <> lr(theta, theta_hat) > c (large lr speaks against H0)
  • lr() = 2(l(theta_hat) - l(theta)) ~ X^2_p
    • lr(theta, theta_hat) = l(theta_hat) - l(theta) >= 0
    • l(theta) = TSE at theta_hat = l(theta_hat) - 0.5 s^2 / I
    • lr(,) = 0.5 s^2/I
  • c = X^2_p,1-alpha

• no statement about type 2 error yet
• power = P("H1" | H1) = 1 - P(type2 error)
  • = 1 - PHI(z_1-alpha + (mu_zero - mu) / (sigma / sqrt(n)))
  • increasing n increases the power, decreases type2 error
• aim: maximal power while maintaining alpha

• aim: construct optimal test: significant alpha-test + higher power than regual PSI(y)
• "H1" <> l(theta_zero) - l(theta_1) <= c
  • a = f(y; theta_zero) / f(y; theta_1) <= exp(c) <= k
  • PHI(y) = 1 if a <= k, else 0
• proof:
  • P(PSI(y) = 1; theta_one) <= P(PHI(y) = 1; theta_one) >> larger power
  • three case (a < k, a > k, a = k)
  • always: power(PHI) - power (PSI) >= P(PSI(y) = 1 - PHI(y) = 1| H0))