Last time

Ensemble learning

Combining multiple classifiers

• Use majority rule or weighted majority for final classification
• Use mean or weighted mean or median for regression
• Can use same hypothesis class over multiple datasets
• Or different hypothesis classes over same dataset

How to stratify dataset?

• Bootstrap resampling/Bagging
• Resample uniformly at random with replacement to create new datasets

AdaBoost

• Consider a sequence of weak learners which can operate on a dataset
• Optimizes the sequence of weak learners by weighting them
• For each classifier, if it classifies correctly, increase weight multiplying by $e^{\beta t}$ and $e^{-\beta t}$ if classified incorrectly
• Note that $\beta$ is inversely proportional to the error

Applying AdaBoost to decision stumps

• Will weight based on simple lines, can create much more interesting decision boundary

What is the loss function for AdaBoost?

• Exponential loss: $\ell^{\mathrm{EXP}}(y;\,h(x))=e^{-yh(x)}$