Skip to content

Independent AI & deep-tech news

Machine Learning

An Ultimate Guide To Ensemble Learning

An Ultimate Guide To Ensemble Learning
An Ultimate Guide To Ensemble Learning
On this page
  1. Key takeaways
  2. What is ensemble learning?
  3. Why use an ensemble instead of a single model?
  4. How do bias and variance relate to ensembles?
  5. How do ensembles improve performance?
  6. How should you decide whether to try an ensemble?
  7. Frequently asked questions
  8. Conclusion

Ensemble learning combines the predictions of two or more models to get a result that is usually better or more stable than any single model. Ensembles are a go-to technique when predictive performance matters most, but they are not always the right choice.

Key takeaways

  • An ensemble is a model that combines the predictions of two or more models, called ensemble members.
  • Ensembles offer two main benefits: better average performance and more robust predictions.
  • The performance gain usually comes from reducing the variance part of prediction error.
  • An ensemble can perform no better than, or even worse than, its best member, so it must be tested.
  • Ensembles add complexity and computational cost.

What is ensemble learning?

An ensemble is a machine learning model that combines the predictions from two or more models. The models inside it are called ensemble members. They can be of the same type or of different types, and they may or may not be trained on the same data.

The members’ predictions can be combined with simple statistics, such as the mode or the mean. More sophisticated methods can learn how much to trust each member and under which circumstances.

Research on ensemble methods grew in the late 1990s, and their use became widespread in the late 2000s as they proved highly successful in machine learning.

Why use an ensemble instead of a single model?

There are two main reasons, which are related.

Benefit Meaning
Performance An ensemble can make better predictions and reach better performance than any of its individual members.
Robustness An ensemble reduces the spread of predictions and of model performance.

The trade-off is cost. Ensembles add complexity and computing effort, because several models must be built and maintained instead of one. Beginners in applied machine learning sometimes assume an ensemble is always the best option, which is not true.

How do bias and variance relate to ensembles?

A machine learning model learns a mapping from inputs to outputs, for regression or classification. It learns this mapping from examples in a training dataset, and it is evaluated on a test dataset that was not used during training.

The errors a model makes are often described in terms of two properties.

  • Bias measures how well the model can capture the mapping between inputs and outputs. It reflects the model’s inflexibility, meaning the strength of its assumptions about the form of that mapping.
  • Variance measures how much the model’s performance changes when it is fit on different training data. It captures the effect of the particular dataset on the model.

Bias and variance are connected. Ideally a model would have both low bias and low variance, but that is hard to achieve in practice, and finding such a model is the central goal of applied machine learning on a given prediction problem. Reducing bias usually increases variance, and reducing variance usually increases bias.

How do ensembles improve performance?

Reducing the variance component of the prediction error improves predictive performance. Ensemble learning is used mainly for this purpose, aiming for better results such as lower error in regression or higher accuracy in classification. This is the approach used in machine learning competitions.

An ensemble is only worth using if it performs better on average than every one of its members. If it does not, the best individual member should be used instead. The comparison should use the average error or accuracy of each model on a test set.

It is important to recognize that an ensemble that reduces variance changes the distribution of errors. It does not simply narrow its range. Expecting it to beat every single model automatically is a common mistake among beginners.

When does an ensemble fail to help?

  • It can perform no better than its best member. This happens when one top-performing model dominates and the other members add nothing, or when the ensemble cannot combine their contributions effectively.
  • It can perform worse than its best member. This happens when one strong model’s predictions are made worse by one or more weaker models, and the ensemble cannot limit their influence.

For this reason, it is wise to try several ensemble techniques and compare their behavior, just as you would compare any other machine learning models.

How should you decide whether to try an ensemble?

  • Establish the performance of the best single model on a test set.
  • Build an ensemble from several members and measure its average performance the same way.
  • Check whether the gain justifies the extra complexity and computing cost.
  • If the ensemble does not beat the best member, keep the single model.

Frequently asked questions

What is an ensemble in machine learning?

It is a model that combines the predictions of two or more other models. The combined output is typically computed with a statistic such as the mean or the mode, or with a more refined method.

Do ensembles always beat single models?

No. An ensemble can match or even underperform its best member. Always compare it with the best individual model on test data.

What do ensembles reduce?

They mainly reduce the variance component of prediction error. That improves average performance and makes predictions more stable.

What is the downside of ensembles?

They require more time, expertise and computing resources, because multiple models have to be trained and maintained.

Conclusion

Ensemble learning is a strong way to improve predictive performance and robustness, mainly by reducing variance. It is not guaranteed to win, and its cost in complexity is real. Test ensembles against the best single model before committing to one.

Written by

Manika Sharma

16 articles on DeepTechBytes