Decision Trees and Random Forests
Interpretable rule-based models built by recursive splitting, and the forests of randomized trees that made them competitive.
A decision tree is a machine learning model that predicts by recursively partitioning the feature space into regions and assigning a prediction to each region. Internally it is a flowchart: each internal node tests one feature ("age ≤ 30?", "income > 50k?"), each branch follows the outcome, and each leaf holds a prediction — a class label for classification, a mean value for regression. Any input is routed from the root to a leaf, and the path is the model's explanation.
The tree is grown from data by choosing, at every node, the split that best separates the training examples. The standard criteria are the Gini impurity and the entropy used in the classic ID3, C4.5, and CART algorithms; both measure how mixed the classes are, and the chosen split is the one that reduces impurity most. Recursion continues until a stopping rule fires — maximum depth, minimum samples per leaf, or no informative split left. The result is a model that handles mixed numeric and categorical features, captures nonlinearities, and needs little preprocessing: trees are among the few models that work reasonably on raw, messy tables.
Single trees have two well-known weaknesses. They are high-variance: a small change in the training data can produce a completely different tree, and deep trees overfit badly unless pruned or limited. They are also greedy — the locally best split can be globally suboptimal. Interpretability is their great strength: a small tree is human-readable, which is why trees appear in credit scoring, medical triage rules, and anywhere an explanation is required.
Random forests fix the variance problem by combining many trees. Leo Breiman's 2001 formulation uses two sources of randomness: each tree is trained on a bootstrap sample of the data (bagging), and each split considers only a random subset of features. The trees are therefore decorrelated — they make different errors — and averaging turns their noisy individual predictions into a stable, accurate ensemble. Forests are remarkably robust: they need little tuning, resist overfitting as the number of trees grows, estimate feature importance and out-of-bag error for free, and were for years the default answer to "what model should I try first?" on tabular data.
Boosting is the complementary ensemble strategy: instead of building trees in parallel, it builds them sequentially, each correcting the errors of the previous — AdaBoost, then gradient boosting, then the highly optimized XGBoost, LightGBM, and CatBoost implementations that dominate machine learning competitions on tabular data. The price of the forest's accuracy is the loss of the single tree's neat explanation, and feature-importance rankings from forests must be read with care when features are correlated. Interpretability, robustness, and competition-grade accuracy are the reasons trees and their ensembles remain central to applied machine learning decades after their invention.
Tags
decision trees ensemble methods machine learning