Business analytics
Supervised learning
Predict a yes-or-no outcome, comparing logistic regression, a decision tree, nearest neighbours and a random forest fairly.
Supervised learning learns from rows where the answer is known, then predicts it for rows where it isn’t. Four models answer the same question here, and every one is judged only on rows it never trained on.
Your data
Choose a table, then the yes-or-no column to predict (any column with exactly two values) and the columns to predict it from. Every section below uses this choice. A table you loaded under Explore appears here too.
Compare models
Logistic regression, a decision tree, nearest neighbours and a random forest, scored the same fair way: the rows are dealt into five folds that keep the yes/no balance, and each model is trained on four folds and scored on the fifth, five times over. Every row ends up predicted by a model that never saw it.
Drawn in colour on the charts and used for feature importance below. The decision tree’s depth is set under A tree you can read.
Small dots are the five folds; the diamond is their mean, with a line one standard deviation either side. Models whose lines overlap are hard to tell apart on this data.
How many of the yes rows each model catches (up) for how many no rows it wrongly flags (across). Higher and further left is better; the diagonal is a coin flip. The figure in brackets is the AUC of all the out-of-fold predictions pooled, which can differ a little from the average of the five fold scores.
A tree you can read
A decision tree asks one yes-or-no question at a time, choosing at each step the split that leaves the two sides purest (lowest Gini impurity). A shallow tree is a set of rules a person can check; a deep one fits the training rows better and new rows worse.
These settings also drive the decision tree in Compare models above.
- Leaves (rules)
- —
- AUC across folds
- —
- AUC on its own training rows
- —
Read down from the top: go left when the answer is yes. Each leaf shows the share of its rows that were yes. Point at a leaf, or focus the tree and use the arrow keys, to see the rules that lead there.
Show the rules as text
Feature importance
Which columns does a model lean on? Permutation importance shuffles one column among the held-out rows, breaking its link to the outcome, and measures how far the AUC falls. It works the same way for every model, so the answers are comparable.
Measured on the rows the tree was grown on, so it flatters columns with many distinct values. Read it alongside the chart above.
- Two columns that carry the same information share the credit: shuffle one and the model leans on the other, so both can look minor.
- A small negative value means the column didn’t help; shuffling it changed the score by chance.
- Importance says what this model uses to predict, not what causes the outcome.
Logistic regression in depth
Logistic regression is the model you can explain line by line: each coefficient becomes an odds ratio with a confidence interval. Fitted on every row for the reading, and judged on the 30% of rows held back from a separate fit.
- Accuracy
- —
- Precision
- —
- Recall
- —
- McFadden pseudo-R²
- —
Sources
- Breiman, L., Friedman, J., Olshen, R. & Stone, C. (1984). Classification and Regression Trees. Wadsworth. Breiman, L. (2001). Random forests. Machine Learning 45, 5–32.
- Cover, T. & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13(1), 21–27. Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning, 2nd ed., ch. 7 (cross-validation) and ch. 15 (random forests). Springer.
- Fisher, A., Rudin, C. & Dominici, F. (2019). All models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research 20(177), 1–81.
- Hosmer, D. W., Lemeshow, S. & Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd ed. Wiley. McFadden, D. (1974). Conditional logit analysis of qualitative choice behavior. In Frontiers in Econometrics.
- Everything here is computed in your browser. The tree, nearest neighbours and AUC are checked against scikit-learn, and logistic regression against statsmodels, in the site’s automated tests.
- All example datasets are synthetic, generated for this page.