Machine Learning Without the Mystique: Supervised, Unsupervised and Overfitting

Machine learning has a reputation for being either magic or maths too heavy to bother with. In practice, for most finance and reporting work, it is neither. It is a way of fitting a rule to historical data, then checking whether that rule still holds on data it has not seen. Everything else is detail.

This post covers the three ideas I think are worth having straight before touching a library: the difference between supervised and unsupervised learning, why we split data into training and test sets, and what overfitting looks like when you measure it. There is a short scikit-learn example with made-up data to make it concrete.

What machine learning actually is

A conventional program is a set of rules a person wrote down. A model is a set of rules that was estimated from examples. If you have ever fitted a trendline in Excel, you have done a small version of this: you gave the software points and it produced a formula you can apply to new inputs. Machine learning generalises that to many inputs and flexible shapes, such as decision trees, ensembles or neural networks.

The useful mental model is that a model is a compressed summary of past data, shaped by the question you asked. It does not understand your business. It finds patterns that were present in the sample, whether or not they are causal and whether or not they will continue.

Supervised learning: you have the answer column

In supervised learning, every training row comes with a known outcome, called the label or target. The model learns the mapping from the inputs to that outcome. If the outcome is a category (fraud or not fraud, default or no default), it is classification. If it is a number (next month’s claims, a bond’s spread), it is regression.

The key requirement is a trustworthy label column. In investment data work, that is often the hard part. Labels such as “this trade was an error” may be recorded late, inconsistently, or only for cases someone happened to investigate. A model trained on that inherits the same gaps.

Unsupervised learning: no answer column

Unsupervised learning has inputs only. The task is to find structure: group similar rows together (clustering), or flag rows that look unlike the rest (anomaly detection). Typical uses are segmenting accounts by behaviour or surfacing odd records for a data quality review. scikit-learn’s clustering guide compares the main methods.

The catch is that nothing tells you whether the groups are right. A clustering is useful if a human looks at it and finds it meaningful, which makes review part of the method rather than an afterthought.

Train/test splits: marking your own homework is not allowed

If you measure a model on the same rows it learned from, the score tells you how well it memorised, not how well it will perform on tomorrow’s data. The scikit-learn documentation describes this as a methodological mistake: a model that simply repeated the labels it had seen would score perfectly and be useless on new rows (cross-validation guide).

The standard fix is to hold some data back. You fit on the training portion, then score once on the held-out test portion, using train_test_split. The same guide covers cross-validation, which repeats the split several times for a steadier estimate.

One caveat matters a lot in finance. A random split mixes past and future rows, so the model can effectively see tomorrow while training on yesterday. For anything time-ordered, split by date instead. scikit-learn provides TimeSeriesSplit for this, described in the same guide.

Overfitting, measured

Overfitting means the model has learned the noise in the training sample as if it were signal. The tell is a large gap between training and test performance. The example below uses synthetic data with deliberate label noise, so the numbers are illustrative and say nothing about real markets.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.cluster import KMeans

# Synthetic, noisy data: 1,000 rows, 10 features
X, y = make_classification(
    n_samples=1000, n_features=10, n_informative=3, flip_y=0.15, random_state=1
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1
)

for depth in (None, 3):
    model = DecisionTreeClassifier(max_depth=depth, random_state=1)
    model.fit(X_train, y_train)
    print(depth, round(model.score(X_train, y_train), 2), round(model.score(X_test, y_test), 2))

# Unsupervised: no labels passed in
clusters = KMeans(n_clusters=3, n_init=10, random_state=1).fit_predict(X)

When I ran this on scikit-learn 1.9.1, the output was as follows. Exact figures can shift between library versions.

The unrestricted tree gets every training row right and drops to 0.75 on the test set. The restricted tree looks worse on training data and better where it counts. That is the whole lesson: a perfect training score is a warning, not a result. The underfitting and overfitting example in the scikit-learn gallery shows the same trade-off with polynomial curves.

The last two lines of the code show the unsupervised side. KMeans receives only X, never y, and assigns each row to one of three groups. Whether those groups mean anything is a judgement for a person to make.

Where this fits in reporting and data quality work

• Rare events need the right metric. If only a few per cent of transactions are fraudulent, accuracy can look excellent for a model that flags nothing. Use precision, recall and the confusion matrix instead.
• Start with a baseline. A SQL rule or a simple threshold is often the benchmark a model has to beat, and sometimes it is the better answer.
• Validate the inputs. A model fed misaligned dates or duplicated rows will fit that damage faithfully. Automated checks in Python or SQL before training are cheaper than explaining a bad model afterwards.
• Keep the test set untouched. If you tune against it repeatedly, it stops being independent.

Further study

For a rigorous but readable next step, An Introduction to Statistical Learning is freely available from its authors (statlearning.com), with a Python edition. Its sections on resampling and model selection cover the ground in this post in more depth.