The Theory
So what's a Decision Tree Regressor and how does it work?
It's a regression model so it returns continuous numerical values. It gets a prediction by asking a series of yes/no questions about the chosen features, building a binary tree structure where each question splits the data further until it reaches a prediction at the leaf node.
It learns by choosing which questions to ask
So for a given feature, a "split" is a threshold value that divides the data into two groups, those above and those below.
So say we wanted to make predictions on house price and we have features like the number of rooms, the size of the land, etc. Then at each node the algorithm tries every possible split across every feature and picks the one that minimises the mean squared error (MSE) of the resulting groups i.e. the split that produces two groups where house prices within each group are as similar to each other as possible ( it sums the MSE from each set of two groups and picks the set with the lowest combined MSE).
Where,
$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2$$
example:
| Split 1: Rooms < 2 | Group A [1] | Group B [2, 3, 4, 5] |
| Split 2: Rooms < 3 | Group A [1, 2] | Group B [3, 4, 5] |
| Split 3: Rooms < 4 | Group A [1, 2, 3] | Group B [4, 5] |
| Split 4: Rooms < 5 | Group A [1, 2, 3,4] | Group B [5] |
What value does the leaf node actually hold?
This is the key bit. During training, multiple houses end up at each leaf node. The predicted value stored at that leaf is simply the mean house price of all training houses that landed there.
How do we know if our model is any good?
A reasonable way to validate the model is to compare, on average, how close a prediction is to the actual value. We will use Mean Absolute Error (MAE).
$$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$
At this point you may be wondering why are we using MSE for training and MAE for validation. Can't we just pick one metric and use it for splitting the tree and validating it?
We use MSE for training because it's faster than MAE. During training, the tree has to calculate errors for millions of potential splits. Because MSE uses squared numbers, it unlocks a clever mathematical shortcut (variance reduction) that lets the computer calculate splits using a simple running tally.
$$\sum (y_i - \bar{y})^2 = \sum y_i^2 - \frac{(\sum y_i)^2}{n}$$
Look closely at the right side of the identity. There are no subtractions between individual data points and the mean. Instead, the entire formula is built out of only three components, The sum of all squared target values, the sum of all target values (which is then squared) and the count n.
The computer can keep a running tally of the sums, the squared sums, and the counts as it slides through split points. It never has to look back at individual data points to calculate the error for a group.
We validate with MAE because humans understand it: MSE leaves you with an error in "squared units" (like squared euros), which would probably raise a few eyebrows at your local bank.
Limitations to consider.
Decision Tree tends to overfit because it memorises the training data rather than learning general patterns. This may result in highly specific "rules" that only apply to historical data, causing it to struggle to generalise.
When you train a decision tree, it uses a greedy algorithm. This means at every single split, it only cares about making the absolute perfect decision for the data right in front of it at that exact moment. It doesn't look ahead, and it doesn't plan for the future.
If you don't stop it (by setting limits like max_depth), the tree will keep asking questions until it has separated every single unique case in your training set into its own tiny leaf node.
This is exactly why Ensemble Methods like XGBoost exist - it builds hundreds of shallow trees and combines them, which is far more robust. Think of a Decision Tree as the foundation concept that XGBoost is built on top of.
The Code
Learning path:
Explore the data and identify prediction target
Transform - Clean the data
Select features and Model type
Train the model - find patterns from the data
Predict and Evaluate the model
Optimise the model
Step 1: Load and Explore the data
Using Pandas to read from a csv file and return a pandas DataFrame.
import pandas as pd
housing_df = pd.read_csv('./datasets/housing_data.csv')
print(f"Statistical Summary \n {housing_df.describe()}")
print(f"Dataset Features \n {housing_df.columns}")
Output
Statistical Summary
Rooms Price ... Longtitude
Propertycount
count 13580.000000 1.358000e+04 ... 13580.000000 13580.000000
mean 2.937997 1.075684e+06 ... 144.995216 7454.417378
std 0.955748 6.393107e+05 ... 0.103916 4378.581772
min 1.000000 8.500000e+04 ... 144.431810 249.000000
25% 2.000000 6.500000e+05 ... 144.929600 4380.000000
50% 3.000000 9.030000e+05 ... 145.000100 6555.000000
75% 3.000000 1.330000e+06 ... 145.058305 10331.000000
max 10.000000 9.000000e+06 ... 145.526350 21650.000000
Output
Dataset Features
Index(['Suburb', 'Address', 'Rooms', 'Type', 'Price', 'Method', 'SellerG', 'Date', 'Distance', 'Postcode', 'Bedroom2', 'Bathroom', 'Car', 'Landsize', 'BuildingArea', 'YearBuilt', 'CouncilArea', 'Lattitude', 'Longtitude', 'Regionname', 'Propertycount'], dtype='object')
(i) Remove any rows that have NaN (missing) value ( axis: 0 -> rows, 1 -> columns )
clean_housing_df = housing_df.dropna(axis=0)
(ii) Select the prediction target 'y' and the features 'X'.
y = clean_housing_df.Price
features = ['Rooms', 'Bathroom', 'Landsize', 'Longtitude', 'Lattitude']
X = clean_housing_df[features]
(iii) Split the data into a training set and validation set
from sklearn.model_selection import train_test_split
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)
Step 3: Select Model type and train.
(i) Scikit-learn Decision Tree Regressor ( random state = 1. If the split results in a tie then breaking is deterministic )
from sklearn.tree import DecisionTreeRegressor
housing_regressor_model = DecisionTreeRegressor(random_state=1)
(ii) Fit the model
housing_regressor_model.fit(train_X, train_y)
Step 4: Predict
predicted_home_prices = housing_regressor_model.predict(val_X)
print(predicted_home_prices)
Output
[ 503000. 1857000. 760000. ... 4200000. 800000. 1785000.]
Step 5: Validate
Using the Mean Absolute Error metric as discussed above.
from sklearn.metrics import mean_absolute_error
mae = mean_absolute_error(val_y, predicted_home_prices)
print(mae)
Output
251698.44673983214
Step 6: Optimise
How can we minimise the MAE?
By using a process known as Hyperparameter Optimisation .
We can tune our decision tree using the max_leaf_nodes parameter, which is passed to the model before training. This setting allows us to specify the upper limit of leaf nodes the model is allowed to create, helping us control the balance between underfitting (too few leaves) and overfitting (too many leaves).
Let see if we can find a value for max_leaf_nodes that minimise the MAE.
mae_scores = []
candidate_levels = [10, 100, 200, 400, 500, 1000, 5000]
for max_leafs in candidate_levels:
# Initialize and fit the model with the current candidate size
model = DecisionTreeRegressor(max_leaf_nodes=max_leafs, random_state=0)
model.fit(train_X, train_y)
# Predict and evaluate
preds_val = model.predict(val_X)
mae = mean_absolute_error(val_y, preds_val)
# Store the result as a sublist: [error, node_count]
mae_scores.append([mae, max_leafs])
print(f"Max leaf nodes: {max_leafs} | MAE: {mae:.2f}")
# Find the sublist with the lowest MAE score
smallest_mae = min(mae_scores, key=lambda x: x[0])
print(f"\nThe optimal choice for max_leaf_nodes is: {smallest_mae[1]} (MAE: {smallest_mae[0]:.2f})")
Output
Max leaf nodes: 10 | MAE: 320570.21
Max leaf nodes: 100 | MAE: 256533.25
Max leaf nodes: 200 | MAE: 244453.42
Max leaf nodes: 400 | MAE: 243238.90
Max leaf nodes: 500 | MAE: 241920.26
Max leaf nodes: 1000 | MAE: 243230.84
Max leaf nodes: 5000 | MAE: 256050.02
The optimal choice for max_leaf_nodes is: 500 (MAE: 241920.26)