Boosting Algorithms for Phishing Detection: Features, Tuning and Evaluation

Phishing detection is a good fit for gradient-boosted trees. The inputs are usually tabular (counts, flags, ratios and scores extracted from a message), the classes are heavily imbalanced, and attackers keep changing their tactics. This post covers what I have found matters most when tuning a boosted model for the job, and where common advice gets muddled. It was written with Bohdan Boiprav.

What boosting does

Boosting builds a strong classifier by adding weak learners one at a time, usually shallow decision trees, with each new learner concentrating on the mistakes of the ensemble so far. The three names you will meet most often differ in how they do that:

• AdaBoost increases the weight of misclassified examples, so the next tree pays more attention to them.
• Gradient boosting fits each new tree to the gradient of a loss function with respect to the current predictions. For squared error that gradient is just the residual.
• XGBoost is not a separate idea. It is a fast, regularised implementation of gradient boosting (Chen & Guestrin, 2016), as are LightGBM and scikit-learn’s histogram-based estimators.

The scikit-learn guide to ensemble methods describes AdaBoost and gradient boosting in detail.

Features usually matter more than the algorithm

A boosted model can only use what you give it. For phishing email, the useful signals tend to fall into a few groups:

• Message content: word frequencies, the presence of urgency or credential-related terms, and the ratio of links to text.
• URLs: length, number of subdomains, use of raw IP addresses, mismatch between the visible link text and the real target, and the age of the domain.
• Headers and sender: failed SPF or DKIM checks, a reply-to address that differs from the sender, and whether the sender has ever written to the recipient before.
• Attachments: file type and whether the file is an archive or macro-enabled document.

Each of these is cheap to compute and easy to explain, which matters when someone asks why a message was quarantined. New features that capture a new tactic usually move the result more than another round of tuning.

Measure the right thing

Phishing is rare compared with legitimate mail, so accuracy is almost useless: a model that flags nothing can score above 95% on accuracy if only a few percent of messages are malicious. Use precision, recall and the area under the precision-recall curve instead. In scikit-learn that last one is average_precision_score. Also split the data by time rather than at random. Phishing campaigns change, and a random split lets the model see near-duplicates of its test messages.

Tune hyperparameters with a sensible metric

The settings that matter most are the learning rate, the number of boosting iterations, the maximum tree depth and the minimum number of samples per leaf. A lower learning rate generally needs more iterations. Random search is usually enough to find a good region, and Bayesian optimisation is worth trying if each fit is slow. The example below runs on synthetic data, not real email, and shows the structure of the search.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split

# Stand-in for an email feature matrix: ~2% positives (phishing), 20 numeric features
X, y = make_classification(n_samples=6_000, n_features=20, n_informative=8,
                           weights=[0.98], flip_y=0.01, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0)

search = RandomizedSearchCV(
    HistGradientBoostingClassifier(class_weight="balanced", random_state=0),
    param_distributions={
        "learning_rate": [0.03, 0.05, 0.1, 0.2],
        "max_depth": [3, 4, 6, None],
        "max_iter": [100, 200, 400],
        "min_samples_leaf": [20, 50, 100],
    },
    n_iter=6,
    scoring="average_precision",
    cv=StratifiedKFold(3, shuffle=True, random_state=0),
    random_state=0,
)
search.fit(X_train, y_train)

print("CV average precision:  ", round(search.best_score_, 3))
print("Test average precision:", round(average_precision_score(
    y_test, search.predict_proba(X_test)[:, 1]), 3))
print("Baseline (prevalence): ", round(y_test.mean(), 3))

When I ran this, the output was a cross-validated average precision of 0.635 and a test value of 0.687, against a baseline equal to the positive rate of 0.025. Those numbers describe a synthetic dataset and say nothing about real phishing. The point of the baseline line is that average precision should always be read against the prevalence of the positive class.

Combine models carefully

Stacking trains a final model on the predictions of several base models, for example a boosted-tree model on header features and a text classifier on the message body. It can help when the base models see genuinely different information. It helps much less when you stack three boosted-tree models trained on the same columns, because their errors are highly correlated. Always generate the base-model predictions out of fold, otherwise the final model learns from leaked labels.

Where transfer learning fits

The earlier version of this post suggested that transfer learning can be used to improve boosting algorithms directly. That was imprecise. Transfer learning applies to neural networks: a language model pretrained on a large text corpus is fine-tuned on labelled phishing and legitimate emails. Boosted trees do not transfer knowledge in that way.

The two can still work together. Either use the fine-tuned language model as a classifier in its own right and feed its output score to the boosted model as one more feature, or use a pretrained model to produce text embeddings and train the trees on those alongside the hand-built features. In both cases the boosted model stays a tabular learner, and the language model supplies a text-understanding signal it could not compute itself.

Summary

• Gradient boosting is a strong default for tabular phishing features.
• New, well-chosen features and a time-based evaluation matter more than which boosting library you use.
• Judge models on precision, recall and average precision, and compare against the base rate.
• Tune the learning rate, tree depth and iteration count together.
• Stack models that see different information, and use language-model scores or embeddings as inputs rather than expecting trees to “transfer”.