PM
PM@LMU
PM@LMU
-
- 1 / 103
-
Lernkarten
How to train SVMs in primal?
- min 1/n * sum(0.5 || theta ||2 + nCL(y(i), f(x(i))) is not differentiable, no gradient methods (1), if gradient-based methods >> expensive because of sum (2)
- solution:
- use smooth version of hinge loss
- SGD: randomly choose one summand (subgradient at theta(t))
How to train SVM in the dual? What is intuitive, what is the problem and how to solve it?
- intuitive: use coordinate ascent: pick and optimize one variable in each iteration i.e. by linesearch (optimize alphai with alphai != j fixed
- problem: constraint sum(alpha_i * y_i) = 0
- solution: sequential minimal optimization (SMO): optimize dual problem w.r.t. alpha_i, alpha_j with fixing alpha_k, k != i, j
Why is interpretable ML useful? (6)
- powerful models usually intransparent (black boxex) i.e. bagged/boosted trees, kernelised SVMs, deep NN
- for debugging models
- increasing trust in models
- if models have huge impact (decisions about humans)
- if loss function doesn't cover constraints like fairness
- gaining insights (science)
Name interpretable models. (3) What are intrinsically interpretable models? (1)
- decision trees
- decision rules
- linear regression (and GLM)
- iterpretable due to simple structure (weighted sume of tree structure)
- (but may get difficult with many features, complex interactions)
What is the idea of model-agnostic methods? (2)
- for blackbox-models
- underlying model can be exchanged >> explaination independent to model type
How can post-hoc interpretation methods be characterised? (2)
- applied after training
- work for back-box-models
What is global and local interpretability? Name examples (2 + 2)
- global interpretability methods
- explain entire input space
- i.e. Permutation feature importance, partial dependence plot
- local interpretability methods
- explain single prediction y_hat_i
- i.e. ICE, LIME
What is the object of analysis in interpretable ML? Give examples for each.
- trained model by fixed model i.e. permutation feature importance
- learning process by refitting (variability in learning) i.e. PIMP
How can importance be measured? (2) Name an examples (1 + 3)
- feature importance: contribution to prediction performance / variance (compare hypothesis tests in regression)
- permutation feature importance
- feature effects: average relationship between features and predictions (compare theta in regression)
- partial dependence plots, ICE, ALE
What is the goal of permutation feature importance?
How is achieved?
- quantify increase of loss after permuting feature's values
- estimate error_test
- for each feature x_j: shuffle feature (permutation)
- >> estimate error_test >> compare (difference or factor)
- average importance over multiple repetitions
What are the drawbacks of permutation feature importance? (2)
- unstable results >> average results
- permutation regardless of correlations to other features (>> unrealistic combinations of feature values)
What is the main idea of PIMP (2)
- compute distribution of importances under H0 that feature has no influence
- sample under H0 by permuting y-vector model-retraining
Explain how PIMP works in detail
- train unpermuted model, compute FI_j
- for m in 1:reps
- permute y
- retrain model
- compute feature importane FI_j_m
- fit distribution of feature importance of FI_j_m for all m
- calculate p-value: compare FI_j to estimated distribution
What is the general idea of ICE?
- individidual conditional expectation: effect of features x_s on prediction for single observation conditional on complementary features x_c
- fix x_c
- iterate over grid for x_s, fit a model and calculate prediction
What is a partial dependence plot (DPD)?
What is important at analysis of it?
What is a general problem here?
- calculate average for each grid-value (over all observations) at ICE
- = (average model prediction for certain grid value)
- >> take marginal distribution into account at interpretations of rare ranges
- extrapolation issues: unlikely combinations of grid-values are taken into account (unrealistic combinations of feature values)
What is the general idea of ALE? (2) What is it's main advantage?
- ALE = Accumulated local effects
- estimate feature effects (for various orders (first order: main effect, 2nd order: interaction between features))
- partial derivative w.r.t. feature, integrate it w.r.t. feature
- >> effects of other features are removed
- + doesn't suffer when features are correlated
What is the idea of functional decomposition? What can it be used for? (3)
- any prediction function (f) can be decomposed
- intercept
- + 1st order effects (regular linear model)
- + 2nd order effects (interactions)
- ...
- + p-th order effects (ML)
- can be used for
- extract main effects
- find lower-dimensional approximation of model
- contribution to variance of prediction function (>> functional ANOVA)
How can functional ANOVA be characterised? (3 + 1)
- zero means
- orthogonality
- variance decomposition (as sum of single variances)
- >> estimation of single components comparable to PDP
What is the general idea of LIME?
- LIME = local interpretable model-agnostic explanations
- fit a local interpretable (surrogate) model to explain y_hat_i
Explain the main steps of LIME
- choose instance of interest (x)
- get predictions for all instances
- weight samples by proximity to x
- fit weighted, sparse, interpretable model (i.e. LASSO, CART)
What are the disadvantages of LIME? (2)
- how to define locality (proximity)
- instability of explanations
What is the general idea and problem with non-linear separation?
- mapping data into a high-dimensional by feature map
- problem: how to find a good feature map? (>> many combinations possible >> computational/memory problems)
- i.e. monimonial feature extraction: (d + p - 1) over p for degree d
What is the shortcut of the kernel-trick?
- normal:
- map features with feature map
- calculate inner product of mapped features
- >> applicable as long as computable
- kernel trick:
- replace inner products by k(x, x')
What is the intuitive meaning of a kernel?
- similarity representation
What are characteristics of the hilbert space?
- geometry: distances, angles calculatable
- analysis: convergence, limits
Name the influences of the parameters in KSVMs (C, sigma, d). Explain RBF sigest heuristic.
- C:
- small C = large margin
- large C = strong penalization of violations >> wiggled hyperplane
- sigma:
- large sigma = smooth hyperplane
- small sigma = wiggled hyperplane (overfitting)
- >> RBF sigest heuristic: calculate pairwise distances ||x - x'||, estimate for sigma based on quantiles of distances i.e. (q0.1 + q0.9) / 2
- d:
- 1: linear hyperplane
- > 1: non-linearity introduced
- d = 10: not useful
What can be kernelized?
- everything as long as it can be expressed in terms of inner products
- also applicable on strings, graphs, trees (similarity measure needs to be defined)
What is the idea of benchmark-experiments? (3)
- different algorithms
- on different datasets
- ranked by performance
- (sychronised splitting required)
What is the main idea at testing in case of ML?
- are differences in generalisations errors significant
- H0: GEs_hat equivalent
- H1: GEs_hat not equivalent
Charaterise the following tests:
McNemar
two-matched-sampled t-test
Friedman-test
McNemar
- non-parametric
- based on contingency table (nominal or binomial output)
two-matched-sampled t-test
- parametric: approximative normality
- assumptions: iid samples, equal variances
Friedman-test
- H0: average ranks equals for all
What is a ROC-Curve? AUC?
- for each threshold plot TPR, FPR >> line
- AUC: Integral of ROC (between 0 and 1)
When Precision-Recall curves are useful?
- useful if n- >> n+ (imbalanced classes)
- ROC depends on FPR (gets really small as TN are high) >> FP no big influence
- >> balanced: ROC, imbalanced: PR-curves
What are cost curves and why are they useful?
- cost for missclassification introduced >> determine best classifier
- provides more practical information than regular ROC-curves (in which setting a classifier is superior)
What is discrimination?
- ability to perfect separate
What is the general definition of calibration?
What is calibration in the large and small?
What is a calibration plot / reliability plot?
- predicted probability ~ observed outcome for any grouping
- large: regarding full sample
- small: regarding subsamples
- grouped by quantiles (predicted probability (x)vs. observed proportion in that group (y))
- algorithms often biased
What is the relation of calibration and discrimination?
- good calibration doesn't imply good discrimination and vice versa
Relate ROC and calibration. How do changing of rankings and monotonic transformations influence these?
- ROC only measure discrimination (TPR, FPR)
- changing of ranking: influence of ROC, no influence on calibration (if within groups)
- transformations: no influence on ROC (ranking unchanged), affects calibration
Explain how calibration can be used for models.
State two methods for this.
- calibration = postprocessing of predictions (on unseen data)
- platt's scaling: logistic regression on output w.r.t. true labels (parametric) (y = logit(beta_hat * y_hat))
- isotonic regression: piecewise-constant-non-decreasing function instead of logistic regression (adjust violations of monotonicity constraint)
What is the general idea of boosting?
- homogeneous ensemble method (sequential)
- weak classifiers sequentially applied on modified data
What is a weak / strong classifier? What is the main assumption of boosting?
- weak: slightly better than random guessing
- strong: low error, high confidence
- assumption: existence of weak learner implies existence of strong learner
