PM
PM@LMU
PM@LMU
-
- 1 / 103
-
Cartes-fiches
What is the general idea of AdaBoost?
- base learner sequentially applied on weighted training data
- missclassifications recieve higher weight in next iteration
What are the advantages of trees? (6)
- robust to outliers
- categorical features applicable
- can handle missing values
- no impact of monotone transformations
- quickly fitted
- built-in feature selection
Why does gradient boosting overfit (2), how to handle this? (3)
- reasons
- aggressive loss minimization
- large M
- solutions:
- early stopping: limit M
- limit depth of trees
- shorten step length beta_m each iteration by learning rate / shrinkage parameter v
- interaction: large M, small v; small M large v
What is the idea of stochastic gradient boosting? (2)
- integrate advantages of bagging (reduce variance of week learners)
- >> random subsampling in each iteration
How can gradient boosting be adapted to further problem domains (i.e. regression, count data, survival data)? (2)
- choosing suitable loss-function (differentiable!)
- combining different base-learners
What is the main disadvantage of gradient boosting? What is the solution?
- lack of interpretability (caused by stumps)
- solution: componentwise boosting
Why do we need fair base-learners? How can it be achieved?
- complex base-learners would be prefered over simple ones (lower loss, better fit)
- >> regularisation for each base-learner
What is the basic idea of Gaussian processes?
- instead of searchin for theta in space, search directly in H (space of allowed functions)
- draw functions from prior distribution, after observing data: remove inconsistent functions
How can we specify a distribution over functions? (3)
- finite domain set: h = [h(x_1), ..., h(x_n)] = n-dimensional vector
- h ~ N(mu, sigma)
- drawing a function by drawing from N(mu, sigma)
What is the characteristics of the covariance function in GPs? (2)
- covariance function = kernel
- similarity measure: covariance = closeness = similarity
- if x, x' close = k(x, x') is large = high correlation of f(x), f(x')
- if x, x' is far = k(x, x') small = low correlation of f(x), f(x')
What is the implication of stationary and isotropic at kernels of GPs
- stationary: invariance in translations: k(x, x + d) = k(0, d) for same x
- isotropic: invariance to rotations (implies stationary)
Why GPs are a spatial models?
- correlation K depends on dist(x, x')
- close datapoints = high correlation = high similarity = high contribution to prediction (maximum a-posteriori)
- uncertainty increases with distance (posterior variance is zero for design-points)
How is the training for GPs characterised?
- training = matrix-multiplication
- >> training = model-selection (i.e. choose K and its hyperparameter) >> ML-estimation
How can the inversion of K at GPs be reduced (O(n3))? (4)
- use sparse K-function
- subsampling: O(m^3)
- combining estimates of subsamples: O(n*m^2)
- using low-rank approximation of K
How can a GP be defined? (2)
- K defined shape of GP
- derive K explicitly by inner products OR
- choosing kernel directly
Name the runtime in context of GPs: solve K^-1, compute K, predictions of mean and variance
- O(n^3)
- O(n^2)
- O(n)
- O(n^2)
What are standard approaches for optimization?
- analytic solution >> local stationary point
- gradient-based methods (differential function required)
- numerical solution (if derivative-free >> i.e. Nelder-Mead, simulated annealing, evelotionary algorithms)
Name naive approaches for back-box optimization (5). Name advantages and disadvantages.
- empirical knowledge / manual tuning: select lambda by expert knowledge
- + good outcome for known problems
- - inefficient, bad reproducability, suboptimal solution
- searching
- grid-search: exhaustive search over grid
- random-search: uniform-sampled search over grid
- latin hypercube sampling: randomly sampled, uniqueness in each dimension
- + easy, intuitive, parallelisation
- - considers all (irrelevant) areas (inefficient)
- black-box-optimization
- + focus on relevant regions
- - inefficient (many evaluations)
What is the main idea of optimization with surrogate modeling?
- derive properties about objective function by evaluations of given lambda
Why is GP a good choice for surrogate modeling? (4)
- non-linear
- flexible
- uncertainty modeled
- spatial dependencies
What is a good initialisation for the inital lambdas at surrogate modeling? How to choose m_init?
- lambdas chosen with latin hypercube sampling
- m_init = 2d, 4d, 10d...
What is meant by cooldown in context of infill criterias? (2)
- start with high inital tao (exploration of regions with highest variance)
- decrease tao (exploitation of interesting areas)
What is the general idea of the reinterpolation procedure?
- clean noise from model to properly apply the infill criteron
