PM
PM@LMU
PM@LMU
-
- 1 / 103
-
Lernkarten
Which tasks require human labor at ML? (4)
What is different for deep learning? (2)
- preprcessing
- feature engineering
- chosing model-family
- optimization of hyperparameter
- automatic feature engineering
- results even more sensitive to architecture, hyperparameters
What is the core idea of AutoML?
- automatic end-to-end ML systems
- turn manual exploration to computation x 100
What are the main goals of AutoML? (3)
- efficient exploration (tedious aspects)
- democratization (high availability, open source platforms...)
- improving algorithms
Name techniques of AutoML (7)
- grid/random search
- bayesian optimization (iterative best configuration based on surrogate model)
- evolutionary algorithms (evolve population of configurations)
- meta-learning (predict useful pipelines / architectures)
- transfer-learning (reuse good solutions)
- reinforcement learning (better configurations by reward signal)
- multi-fidelity methods (extrapolate learning curves i.e. Hyperband)
Distinguish between hyperparameters and model parameters. Relate it to pipeline configuration.
- model parameter: result of training
- hyperparameter: input for computation (regions of interest)
- pipeline: precrocessing, feature construction, performance measure, hyperparameter >> to be optimized
Name possible approaches for pipeline configuration (4)
- stable performance: algorithm is insensitive w.r.t. parameter changes
- constant default: find constant configuration by benchmarking (data independent)
- dynamtic default: find constant configuration by benchmarking (data driven) i.e. mtry = p/3
- ex-post determination: go backwards in full-path computation
How can a pipeline configuration be tested?
- pipeline = blackbox >> CV
Define hyperparameter /types) and how they are used by the inducer
- can be continous (i.e. depth of tree), discrete, categorical (which kernel), dependent (kernel parameters)
- Inducer I takes dataset D and hyperparameter configuration lambda and returns trained model (ERM)
Explain what is meant by bi-level optimization. (2)
- finding best model f (find theta by ERM) is nested in...
- outer hyperparameter optimization problem (search for inducer parameter lambda by minimzing generalization error; 2nd level problem)
Name problems of hyperparameter optimization (4)
- derivative free (black-box problem)
- optimization is stochastic (>> resampling)
- computationally expensive evaluation of target function
- complexity by categorical / dependent parameters
What are alternatives for surrogate models. Compare it to SMBO (sequential model-based optimization)
- Random regression forest
- mixed input spaces
- includes uncertainty estimate (variance of trees or bootstrapping)
- SMBO: smoother variance structure, BUT: problems with many categorical hyperparameters
How can learning curves be used at training?
- relative order of configuration in early iterations is insightful
- training every single configuration can be wasteful (i.e. configurations with high predictive error after some iterations)
- >> focus on promising configurations
What is successive halving? (4)
What is the main drawback? (3)
- train all configurations for small inital budget
- remove worst half, double budget
- train until budget is wasted
- repeat 2., 3. until one configuration remains
- user has to specify the inital # configurations (many for few iterations or few for many interations)
- for fixed budget (B) and increasing n: B/n (budget for configuration) decreases (>> inference of picking) (n versus B/n problem)
- relative ordering may vary
What is BOHB?
- Bayesian optimization & hyperband
- combines BO and hyperband (configurations at successive halving sampled from surrogate model)
What is the main idea of meta-learning?
How can it be used today?
- transfer knowledge of previous tasks to new tasks (with small amount of data)
- describe task by features (meta-features) >> return predicted performance, training time, algorithms by similarity
- use result of meta-learning as initial design (prior knowledge ie. sampling from parameter distribution) >> to be improved
How can learning be defined? How can it be improved?
- from Experience E
- w.r.t. task T
- and performance measure P
- >> improved by E
What is the goal of supervised models?
- find f from H s.t. f(x) ~ y (or in case of classification: s(f(x)) ~ y)
What are the main components of ML? (3) Give examples each.
- representation: specification of H
- i.e. Decision trees, NN, Set of rules, linear functions...
- cost-function: evaluate good and bad classifiers (= inner loss; to be optimized during training/fitting)
- i.e. SSE, MSE, MC, likelihood, information gain...
- optimization: effieciently search in H for google models
- depends on cost-function (analytically, numerically)
- i.e. gradient descent, quadratic programming, genetic alg.
What is the main goal of generalisation?
What is the relation between training and test error?
- generalize beyond Dtrain
- f_hat learned from Dtrain s.t. expected training-error <= expected test-error
What is the difference between inner and outer loss? (2+1)
- inner: used during optimization
- outer: estimate generalisation error
- >> should be the same, but for i.e. AUC not possible (AUC not optimizable)
Why is it better to return continuous output at classification? (3)
- easier to optimize
- more information contained
- transformation to class-labels easy
What is a decision boundary?
What are boundaries?
How is it defined (mathematically)
What is the distance from a point to the hyperplane?
- hyperplane partitioning X into g regions (g-1 dimensional linear subspace)
- boundaries = ties (exists i != j s.t. fi(x) = fj(x))
- defined by normal vector theta (|| theta || = 1), perpendicular to hyperplane with offset theta0 (>> f(x) = thetaT x + theta0 = 0)
- d(f, x) = | f(x) | / || theta ||
What is over- and underfitting? What does it depend on?
- under: no sufficient low errortrain
- over: large delta of errortrain and errortest (noise modelled)
- depends on capacit of learner (complexity of H, i.e. # parameters)
Explain the variance-bias tradeoff
- low capacity: high bias, low variance
- high capacity: low bias, high variance
- see bias variance decomposition
What is the No Free lunch theorem?
What does this mean for ML?
- there exists no best algorithm (averaged over all possible problems)
- only on specific set of problems
- choosing best out of many algorithms is again an algorithm
- >> narrowing problems (subset only) by assumptions / prior expert knowledge
How do we find the best model f_hat?
- by pointwise evaluation of the const-function L(y(i), f(x(i))) >> minimized expected loss
How to choose a loss-function? (2)
- statistical properties (distribution of error, robustness)
- optimization complexity (i.e. smootness, convexity, closed-form, differentiable)
Explain the relationship of frequency of losses and the value of loss
- high loss: not frequent
- low loss: frequent
- maximum @ L(.) = 0
State the general aim in information theory.
Relate the probability to oberserve an event to the information gain.
- quantify amount of informtion when r.v. is observed
- if p is small >> I(p) is large (surprise), if p is large >> I(p) is small (expected)
What is the aim of source coding?
- efficient storage of information i.e. string as binary code
What is the meaning of cross-entropy?
- average increase of code-length from going from optimal (p(x)) to non-optimal (q(x)) code
What is the meaning of joint entropy?
- = total uncertainty in two variables
- = entropy of joint distribution (p(x, y))
What is the meaning of mutual information?
- reduction of uncertainty in y after learning x
How to define a good code book? How can this be done?
- error-correcting-codes (ECOC)
- row separation: dissimilar code words for classes
- column separation:
- uncorrelated (similar classifiers >> similar errors)
- not complementary (same errors, not too large distance)
- not equal (same errors, not too small distane)
- >> random hill climbing
What can be done against overfitting? (4)
How is this related to regularisation?
- more data (increase n)
- better data (less noise)
- less complex models
- optimize less
- >> aim: reduce generalisation error
- >> regularisation: maximize fit (performance), minimize complexity (sparsity) >> find sweet spot
What is the influence of regularisation on bias and variance? Explain in case of linear regression.
- bias increased
- varince decreased
- (XTX)-1 requires full rank, highly correlated features increase variance >> regularisation
Describe early stopping. (2)
Name pros and cons. (1 + 2)
- iterative optimization: generalisation error increases after some iterations >> stop, use previous theta
- similar effect like regularisation
- + simple effective method
- - evaluation of GE each iteration (expensive)
- - copying theta after each iteration
What is the general idea of SVMs?
- data linear separable (i.e. small n, large p)
- best classifier: maximize safety margin
What is the difference between hard (1) and soft-margin SVMs (3)? How are they related? (1)
- hard-margin SVM:
- support vector y(i) * f(x(i)) == 1, solution independent of non-SVMs
- soft-margin SVM:
- data not linear separable (large margin for most samples)
- slack-variable = margin-violation
- support vector: y(i) * f(x(i)) <= 1
- relation: hard SVM is special-case of soft SVM (C -> infinity s.t. margin-violations -> 0)
What does the hinge-loss measure? What are other choices for loss in case of SVMs? (3)
- violations of margin
- hinge loss
- squared hinge loss
- huber loss (smoothed hinge loss)
