Outliers and impossible values · B02 · Sunflower Intelligence

Before you start

You will be able to

Distinguish impossible values, errors and genuine extreme values, and explain why impossible values are a validation problem before they are an outlier problem.
Calculate quartiles, the interquartile range, Tukey fences, z-scores and modified z-scores for a small dataset.
Explain masking and why rules built on the median and the median absolute deviation resist it.
Choose a treatment for a flagged value (correct, remove, cap, keep with robust statistics, or flag) and calculate its effect on the mean and the median.
Document an outlier decision so that someone else could reproduce it.

Review first

Missing values — Impossible values that can’t be corrected become missing values, and the treatments here build on the ones you met there.
Profiling a dataset and writing validation rules — Range checks and validation rules catch impossible values before any outlier rule is applied.
Sorting a list, finding its median, and calculating a mean and a sample standard deviation.

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

Lantern’s online shop sells coffee in 250 g bags, and most orders are for one to six bags. Ana averages the quantity column for the last twelve orders and gets 95.7 bags per order, a number that would have her ordering a warehouse. Two orders explain it. One is for 120 bags from a law firm that now buys coffee for its staff kitchens through the shop. The other is for 999 bags from a customer whose previous orders were for two; the payment taken matches nine bags. Both are unusual, but only one is wrong.

An outlier is an observation that lies an abnormal distance from the other values. The definition deliberately leaves “abnormal” to judgement: a statistical rule can flag a value as unusual, but only knowledge of the business can say what it means. That is why this lesson separates two jobs, detection and treatment, and why the first question about any flagged value is not “how do I get rid of it?” but “what is it?”.

Three kinds of unusual value

Impossible values: Values that break a rule of the world: a negative quantity, an attendance of 112%, a check-out date before the check-in date, a nightly rate of $0 or $9,999 typed as a placeholder. They are not unusual measurements; they are not measurements at all.
Errors that look possible: Values that could exist but are wrong: 999 bags typed for 9, a price entered in cents instead of dollars, a weight recorded in grams where the column holds kilograms, a decimal point in the wrong place.
Genuine extremes: Real values that are simply far from the rest: the law firm’s 120 bags, a two-week stay in one of Harborview’s Deluxe rooms, a student who studies 45 hours a week. They are often the most important rows in the file.

Impossible values come first because they are a validation problem, not an outlier problem. A range check such as “quantity must be a whole number of at least 1” or “nightly rate between $100 and $500” (Harborview’s data dictionary) catches them with certainty, without any statistics. They should be caught before you look for outliers, for a practical reason: outlier rules are built from the mean, the standard deviation or the quartiles of the column, and a handful of 9,999 placeholders distorts exactly those numbers. Once identified, an impossible value is corrected from a source document if one exists, or set to missing and handled with the methods of the previous lesson.

How unusual is unusual?

Two families of rules measure how far a value sits from the rest. The first uses quartiles: sort the data, find the values that cut off the lowest and highest quarters, and flag anything well beyond them. The second uses a typical distance from the centre: compute how many standard deviations (or a robust equivalent) a value lies from the mean (or the median). Both need care, because the values you are hunting can damage the yardstick you measure them with. Before any rule, though, look: a box plot, a histogram or a scatter plot shows in seconds whether there is one stray value, a long tail, or two separate groups mixed together.

Anatomy of a box plot. The box holds the middle half of the data; the whiskers stop at the most extreme values inside the fences; anything beyond a fence is drawn as a separate point. Most box plots don’t draw the fences themselves; they are shown here to explain where the whiskers stop.

In depth

Quartiles and the interquartile range

Sort the data. The first quartile Q1 is the value below which a quarter of the data lie, the median is the halfway point, and the third quartile Q3 is the value below which three-quarters lie. The interquartile range (IQR) is $Q_3 - Q_1$ , the width of the middle half of the data. Because it ignores the top and bottom quarters, a few extreme values barely move it.

There are several ways to compute quartiles, and statistical software offers at least nine (Hyndman and Fan, 1996). This course uses linear interpolation, the method of Excel’s QUARTILE.INC and NumPy’s default: the $p$ -th quantile of $n$ sorted values sits at position $1 + (n - 1)p$ , and when that position falls between two values you go the same fraction of the way from one to the next. For Lantern’s twelve orders, sorted: 1, 1, 2, 2, 2, 3, 3, 4, 5, 6, 120, 999.

Q1: position $1 + 11 \times 0.25 = 3.75$ , three-quarters of the way from the 3rd value (2) to the 4th (2): $Q_1 = 2$ .
Median: position $1 + 11 \times 0.5 = 6.5$ , halfway from the 6th value (3) to the 7th (3): median = 3.
Q3: position $1 + 11 \times 0.75 = 9.25$ , a quarter of the way from the 9th value (5) to the 10th (6): $Q_3 = 5 + 0.25 \times 1 = 5.25$ .
IQR: $5.25 - 2 = 3.25$ bags.

Many textbooks, OpenStax’s Introductory Statistics among them, instead take the median of the lower and upper halves of the data. That method gives Q3 = 5.5 here. The differences are usually small, but a value close to a fence can be flagged by one method and not the other, so say which method you used.

Tukey fences

John Tukey’s box plot, published in his 1977 book Exploratory Data Analysis, made one rule famous (Tukey worked with “hinges”, which are close to the quartiles). The Tukey fences sit 1.5 IQRs beyond the quartiles, and values outside them are flagged as potential outliers:

\text{lower fence} = Q_1 - 1.5 \times \text{IQR}, \qquad \text{upper fence} = Q_3 + 1.5 \times \text{IQR}

For Lantern’s orders the fences are $2 - 1.5 \times 3.25 = -2.875$ and $5.25 + 4.875 = 10.125$ bags, so 120 and 999 are flagged and nothing else is. The NIST/SEMATECH e-Handbook also describes outer fences at 3 IQRs, calling values between the inner and outer fences mild outliers and values beyond the outer fences extreme outliers. Here the upper outer fence is $5.25 + 3 \times 3.25 = 15$ , so both orders are extreme.

The multiplier 1.5 is a convention, not a law of nature. If data really came from a normal distribution, the inner fences would sit about 2.7 standard deviations from the mean ( $0.6745 + 1.5 \times 1.349 = 2.698$ , since Q1 and Q3 of a normal distribution lie 0.6745 standard deviations either side of the mean), and about 0.7% of perfectly ordinary values would fall outside them. In 10,000 clean orders that is around 70 flags. In a long-tailed column such as hotel booking totals the share is far higher, which is one reason the next lesson transforms skewed data before hunting for outliers.

Z-scores and masking

A z-score measures how many standard deviations a value lies from the mean, using the sample standard deviation $s$ (dividing by $n - 1$ ):

z_i = \dfrac{x_i - \bar{x}}{s}

A common convention flags values with $|z| > 3$ ; for normal data about 0.27% of values lie that far out. The rule is easy to explain and works well when the data are roughly normal and the outliers are few. Its weakness is that the outliers are included in the mean and standard deviation they are judged against. An extreme value pulls the mean towards itself and inflates the standard deviation, so it looks less extreme than it is, and it can hide other outliers completely. This is called masking.

Lantern’s twelve orders show it clearly. Their mean is 95.67 bags and their standard deviation is 286.47 bags, both created almost entirely by the 999. The 999 scores $z = 3.15$ , only just over the line. The genuine 120-bag order scores $z = 0.08$ : by this rule it is a perfectly ordinary order. Correct the 999 to 9 and recompute: the mean falls to 13.17, the standard deviation to 33.72, and the 120-bag order now scores $z = 3.17$ . One outlier had masked another.

Small samples make things worse. Shiffler (1988) showed that in a sample of $n$ values no z-score can exceed $(n - 1)/\sqrt{n}$ in absolute value, however extreme the value is. For $n = 12$ that ceiling is $11/\sqrt{12} = 3.18$ , which is why the 999 could only just clear 3. For $n = 10$ it is $9/\sqrt{10} = 2.85$ : in ten values or fewer, a $|z| > 3$ rule can never flag anything at all.

The modified z-score

The fix is to measure distance from a centre and a spread that outliers can’t drag around. The median is one; the median absolute deviation (MAD) is the other: take each value’s absolute distance from the median, then find the median of those distances. The modified z-score recommended by Iglewicz and Hoaglin, and described in the NIST/SEMATECH e-Handbook, is:

M_i = \dfrac{0.6745\,(x_i - \tilde{x})}{\text{MAD}}, \qquad \text{MAD} = \operatorname{median}_i\,\lvert x_i - \tilde{x} \rvert

Here $\tilde{x}$ is the median. The constant 0.6745 makes $M_i$ comparable with an ordinary z-score: for normal data the MAD is about 0.6745 standard deviations, because half of a normal distribution lies within 0.6745 standard deviations of the centre. Iglewicz and Hoaglin recommend flagging $|M_i| > 3.5$ .

For Lantern’s orders the median is 3. The absolute distances from 3, sorted, are 0, 0, 1, 1, 1, 1, 2, 2, 2, 3, 117 and 996, whose median is $(1 + 2) \div 2 = 1.5$ . So the 120-bag order scores $0.6745 \times 117 \div 1.5 = 52.6$ and the 999 scores $0.6745 \times 996 \div 1.5 = 447.9$ ; the largest ordinary order, 6 bags, scores 1.35. Correcting the 999 to 9 changes neither the median nor the MAD, so the 120 scores 52.6 either way: no masking. The rule has one blind spot: if more than half the values are identical, the MAD is 0 and the score can’t be computed. Columns with heavy ties, such as quantities where most orders are for one bag, need the IQR rule or a look at the counts instead.

Three rules applied to Lantern’s twelve orders (fictional)
Quantity	z-score	Modified z	Outside Tukey fences?
6	−0.31	1.35	No
120	0.08	52.6	Yes
999	3.15	447.9	Yes

Flagged with |z| > 3: only 999. With |modified z| > 3.5 or the 1.5 × IQR fences: 120 and 999. The values 1 to 5 are flagged by no rule.

Looking as well as computing

Rules work on one column at a time; plots show context. A box plot draws the quartiles, the fences’ reach and every flagged point, and several box plots side by side (one per room type, say) show whether a “high” rate is just a suite. A histogram shows the shape: a single stray bar far to the right, a long gradual tail, or two humps from two kinds of customer mixed in one column. A scatter plot catches values that are only unusual in combination: an order for two bags with a total of $480 has an ordinary quantity and a plausible total, yet together they are impossible.

Treatments

Once a value is flagged and understood, there are five honest things to do with it. The applied example below works out their effect on Lantern’s numbers.

Investigate and correct. Use evidence from outside the column: Lantern’s payment record shows nine bags were paid for, so 999 becomes 9. Keep the original value in a separate column or a change log.
Remove, with a stated reason. Test orders, duplicated records, and impossible values that can’t be corrected (set those to missing rather than deleting the whole row when the other columns are fine).
Cap (winsorise). Winsorising replaces values beyond a limit with the limit itself, for example the Tukey fence or the 1st and 99th percentiles. The row stays, but its pull on the mean and standard deviation is limited. It changes real values, so it suits models that are sensitive to extremes better than reports of what actually happened.
Keep, and use robust statistics. A robust statistic is one that a few extreme values can’t move far: the median, the IQR and the MAD, or a trimmed mean that drops a fixed share from each end. Genuine extremes stay in the data, and the summary describes the typical case.
Flag or separate. Add a column such as is_bulk and analyse bulk orders as their own segment, or include the flag in a model.

Documenting the decision

An outlier decision must be reproducible. Record the rule and its parameters (“1.5 × IQR on quantity, quartiles by linear interpolation”), the rows it flagged, what you found for each, what you did, and how the key statistics changed. Keep the raw values: write corrections to a new column or keep a change log instead of overwriting the export. If a reviewer asks next year why the March average is 13.2 bags rather than 95.7, the answer should take one line.

In a spreadsheet or pandas

Spreadsheet: QUARTILE.INC(range, 1) and QUARTILE.INC(range, 3) give Q1 and Q3 by linear interpolation; STANDARDIZE(x, AVERAGE(range), STDEV.S(range)) gives a z-score; the MAD is MEDIAN(ABS(range-MEDIAN(range))) entered as an array formula.
pandas: s.quantile([0.25, 0.75]) uses linear interpolation by default; (s - s.median()).abs().median() is the MAD; s.clip(upper=fence) caps values at a fence; df['is_bulk'] = df.quantity > fence creates a flag.

Formulas

Quantile by linear interpolation

\text{position} = 1 + (n - 1)p, \qquad Q_p = x_{(k)} + f\,\bigl(x_{(k+1)} - x_{(k)}\bigr)

$n$: Number of values
$p$: The proportion wanted: 0.25 for Q1, 0.5 for the median, 0.75 for Q3
$x_{(k)}$: The k-th smallest value, where k is the whole-number part of the positionunits of the variable
$f$: The fractional part of the position
$Q_p$: The p-th quantile of the sorted valuesunits of the variable

Assumes. Values sorted from smallest to largest; the same method as Excel’s QUARTILE.INC and NumPy’s default.

Other quartile methods exist and can give slightly different values; state the one you use.

Interquartile range and Tukey fences

\text{IQR} = Q_3 - Q_1, \qquad Q_1 - 1.5\,\text{IQR}, \quad Q_3 + 1.5\,\text{IQR}

$Q_1,\ Q_3$: First and third quartilesunits of the variable
$\text{IQR}$: Interquartile range: the width of the middle half of the dataunits of the variable

Assumes. The multiplier 1.5 is Tukey’s convention; outer fences use 3.

Values beyond the fences are potential outliers to investigate, not automatic errors.

Z-score

z_i = \dfrac{x_i - \bar{x}}{s}

$x_i$: The value being scoredunits of the variable
$\bar{x}$: Mean of all values, including x_iunits of the variable
$s$: Sample standard deviation (dividing by n − 1)units of the variable
$z_i$: Z-score: standard deviations x_i lies above (+) or below (−) the mean

Assumes. Roughly normal data with few outliers; |z| > 3 is a common convention.

In a sample of n values, |z| can never exceed (n − 1)/√n (Shiffler, 1988).

Modified z-score

M_i = \dfrac{0.6745\,(x_i - \tilde{x})}{\text{MAD}}

$x_i$: The value being scoredunits of the variable
$\tilde{x}$: Median of the valuesunits of the variable
$\text{MAD}$: Median absolute deviation: the median of |x_i − median|units of the variable
$0.6745$: Makes M comparable with a z-score for normal data
$M_i$: Modified z-score: distance from the median in robust standard-deviation units

Assumes. Flag |M| > 3.5 (Iglewicz and Hoaglin). Undefined when the MAD is 0.

The median and the MAD are barely moved by outliers, so the score resists masking.

Worked examples

A first example: Fences for Northfield entry scores

Fictional figures

Eleven Northfield applicants have entry scores (0–100) of 52, 58, 61, 63, 64, 66, 68, 70, 73, 96 and 4. Which scores lie outside the Tukey fences, and what should happen next?

Sort: 4, 52, 58, 61, 63, 64, 66, 68, 70, 73, 96. With $n = 11$ , positions are $1 + 10p$ .
Q1: position $1 + 10 \times 0.25 = 3.5$ , halfway from the 3rd value (58) to the 4th (61): $Q_1 = 59.5$ . Q3: position $8.5$ , halfway from 68 to 70: $Q_3 = 69$ .
IQR $= 69 - 59.5 = 9.5$ , so $1.5 \times \text{IQR} = 14.25$ .
Fences: $59.5 - 14.25 = 45.25$ and $69 + 14.25 = 83.25$ . Only 4 and 96 lie outside.

Answer. The scores 4 and 96 are flagged; the fences are 45.25 and 83.25.

What it means. The rule only says where to look. A score of 96 is within the 0–100 scale and the applicant’s school record shows top marks, so it is a genuine extreme and stays. A score of 4 is possible on the scale but far below anyone admitted; the admissions file shows 46, so it is a typing error and is corrected. Neither decision came from the fences.

A harder example: Why the z-score missed Harborview’s placeholder

Fictional figures

Ten Harborview bookings have nightly rates of $118, $125, $132, $140, $145, $150, $158, $165, $179 and $9,999. Score the $9,999 with the z-score, the Tukey fences and the modified z-score, and decide what to do with it.

Mean $= 11{,}311 \div 10 = 1{,}131.10$ ; sample standard deviation $s = 3{,}115.92$ . Both are created by the one placeholder.
$z = (9{,}999 - 1{,}131.10) \div 3{,}115.92 = 2.85$ , below 3, so the $|z| > 3$ rule flags nothing. It never could: with $n = 10$ the largest possible $|z|$ is $9/\sqrt{10} = 2.85$ , and this value is at that ceiling.
Quartiles: Q1 at position 3.25 is $132 + 0.25 \times 8 = 134$ ; Q3 at position 7.75 is $158 + 0.75 \times 7 = 163.25$ ; IQR $= 29.25$ . Upper fence $163.25 + 43.875 = 207.125$ : $9,999 is far beyond it.
Median $= (145 + 150) \div 2 = 147.5$ . Absolute distances from it, sorted: 2.5, 2.5, 7.5, 10.5, 15.5, 17.5, 22.5, 29.5, 31.5, 9,851.5, so MAD $= (15.5 + 17.5) \div 2 = 16.5$ . Modified z $= 0.6745 \times 9{,}851.5 \div 16.5 = 402.7$ , far above 3.5.
Decide: no guest paid $9,999 for a standard room, and the booking system uses 9999 as a placeholder. A range check (“rate between $100 and $500”) would have caught it before any outlier rule. Set it to missing; the other nine rates have mean $1{,}312 \div 9 = 145.78$ and median 145.

Answer. The z-score (2.85) misses the $9,999; the Tukey fences and the modified z-score (402.7) flag it. It is a placeholder, so it becomes a missing value.

What it means. Before the placeholder was removed, the “average rate” was $1,131, nearly eight times what any guest paid. The z-score failed for two reasons at once: the placeholder inflated the standard deviation it was measured against, and ten values are too few for any z-score to pass 3. Validation would have caught it first; the robust rules caught it anyway.

Applied: a business decision: Treating Lantern’s bulk order for two different questions

Fictional figures

After correcting 999 to 9, Lantern’s twelve orders are 1, 1, 2, 2, 2, 3, 3, 4, 5, 6, 9 and 120 bags; the Tukey fences are still −2.875 and 10.125, so only the 120-bag office order is flagged. Ana needs (a) the total bags to restock and (b) a typical order size to choose the box sizes for home deliveries. Compare four treatments.

Keep everything: total $= 158$ bags, mean $158 \div 12 = 13.17$ , median 3 (the 6th and 7th sorted values are both 3).
Remove the bulk order: total 38 bags, mean $38 \div 11 = 3.45$ , median 3 (the 6th of 11 values).
Cap at the upper fence: 120 becomes 10.125, total $38 + 10.125 = 48.125$ , mean $48.125 \div 12 = 4.01$ , median 3.
Before the 999 was corrected: total 1,148, mean $1{,}148 \div 12 = 95.67$ , median 3.
Match treatment to question. Restocking needs every real bag, so (a) uses all 158; capping would leave Lantern about 110 bags short. Home-delivery boxes are about household orders, so (b) uses the median of 3, or the mean of 3.45 with the office order flagged as business-to-business and analysed separately.

Answer. Restock for 158 bags; size home-delivery boxes around 3 bags. Keep the 120-bag order, flagged as a business order, rather than capping or deleting it.

What it means. Across the four treatments the mean ranges from 3.45 to 95.67 bags while the median never leaves 3. That stability is what makes the median a robust statistic, and the instability is why the treatment must follow the question. A single cleaned file can serve both questions if it keeps the genuine order and carries an is_bulk flag.

Explore

Common mistakes

“An outlier is an error, so it should be deleted.”
Rules flag values that are unusual, not values that are wrong. Lantern’s 120-bag order is real revenue; deleting it would remove 120 of the 158 bags those twelve orders sold. Investigate each flagged value with evidence from outside the column, and delete only what you can show is wrong or out of scope.
“Impossible values are just very large outliers.”
A negative quantity, a 112% attendance or a $9,999 placeholder rate fails a validation rule; it isn’t a measurement at all. Catch these with range checks first, correct or set them to missing, and only then look for statistical outliers, because impossible values distort the statistics the outlier rules use.
“|z| > 3 will catch any extreme value.”
Extreme values inflate the mean and standard deviation they are judged against (masking), and in a sample of n values no |z| can exceed (n − 1)/√n, which is below 3 for ten values or fewer. Use the modified z-score or the IQR fences, which rest on the median and quartiles.
“Capping is always safer than deleting.”
Winsorising keeps the row but changes a real value, so any total, revenue or count computed afterwards is wrong. It suits models that are sensitive to extremes; for reporting what happened, keep genuine values and summarise with the median or a flagged segment.
“Quartiles are quartiles.”
Different methods (linear interpolation, medians of halves and others) give slightly different quartiles and fences, so a value near a fence can be flagged by one tool and not another. State the method and use the same one throughout a project.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading four questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading six questions…

In practice

At Lantern, the outlier review is now a fixed step after validation. The analyst runs the quantity and order-total checks each month, looks at every flagged order, and records a one-line verdict in a log: “LC-10310, 120 bags: office customer, confirmed by account history, kept, is_bulk = 1”; “LC-10315, 999 bags: payment matches 9 bags, corrected, original kept in quantity_raw”. The monthly report shows total bags from the cleaned data and the median order size for households, so neither question is answered with the other’s number.

At Harborview the order of operations matters even more. The revenue manager first applies range checks to the nightly rate, which catches the 0 and 9,999 placeholders. Because the booking total is nights × rate, total ÷ nights recovers each true rate, so the placeholders are corrected rather than left missing. Only then does the manager look for outliers by room type, because a $400 night is extreme for a standard room and normal for a suite. Genuine high rates stay in the average daily rate, since they are real revenue; for the question “what does a typical guest pay?”, the manager reports the median. The same logic serves a battery-storage installer monitoring home systems: a sudden reading far outside a battery’s normal range may be a sensor fault to correct or a failing unit to visit, and the extreme values are the ones the maintenance team most needs to see.

Northfield’s admissions office reviews extreme entry scores the same way. A score outside the published scale is a validation failure; a score inside the scale but beyond the fences is checked against the application file. When the analyst later models first-year GPA from entry scores, she keeps the genuine extremes but compares the model with and without them, and reports both if the conclusion changes. That comparison, a sensitivity check, is the most honest answer when a few values could drive the result.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

Unusual values are impossible values (a validation problem), errors that look possible, or genuine extremes; rules flag them, evidence from outside the column decides which.
Quartiles by linear interpolation: position 1 + (n − 1)p. IQR = Q3 − Q1; Tukey fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR (a convention).
The z-score (x − mean) ÷ s is weakened by masking: outliers inflate the mean and standard deviation, and no |z| can exceed (n − 1)/√n.
The modified z-score 0.6745 (x − median) ÷ MAD, flagged above 3.5, rests on the median and MAD, so outliers can’t hide each other.
Treatments: correct, remove with a reason, cap (winsorise), keep with robust statistics, or flag; the median barely moves across them while the mean can swing widely.
Choose the treatment to fit the question, keep the raw values, and log every decision.

Terms introduced

Outlier: An observation that lies an abnormal distance from the other values in a dataset; it may be an error or a genuine extreme.
Interquartile range: The third quartile minus the first quartile: the width of the middle half of the data (IQR).
Tukey fences: Limits at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR; values outside them are flagged as potential outliers. Outer fences use 3 × IQR.
Z-score: The number of standard deviations a value lies from the mean: (x − mean) ÷ s.
Modified z-score: A robust z-score, 0.6745 × (x − median) ÷ MAD; absolute values above 3.5 are flagged as potential outliers.
Median absolute deviation: The median of the absolute distances of the values from their median (MAD); a robust measure of spread.
Winsorising: Replacing values beyond a chosen limit, such as a fence or a percentile, with the limit itself, so that extreme values keep their rows but lose most of their pull.
Masking: The effect in which outliers inflate the mean and standard deviation used to detect them, so that they, or other outliers, no longer look extreme.
Robust statistic: A summary that a few extreme values can’t move far, such as the median, the interquartile range or the median absolute deviation.

Sources

NIST/SEMATECH e-Handbook of Statistical Methods, section 7.1.6, “What are outliers in the data?”. itl.nist.gov/div898/handbook/prc/section1/prc16.htm
NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.17, “Detection of Outliers” (modified z-score of Iglewicz and Hoaglin; masking and swamping). itl.nist.gov/div898/handbook/eda/section3/eda35h.htm
Shiffler, R. E. (1988). Maximum z scores and outliers. The American Statistician, 42(1), 79–80. doi:10.1080/00031305.1988.10475530.
Illowsky, B., Dean, S. et al. (2023). Introductory Statistics 2e, section 2.3, “Measures of the Location of the Data”. OpenStax. openstax.org/books/introductory-statistics/pages/2-3-measure…
Hyndman, R. J. & Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50, 361–365; summarised in the R documentation for quantile. rdocumentation.org/packages/stats/topics/quantile
NumPy documentation: numpy.quantile (default method “linear”, Hyndman and Fan’s method 7). numpy.org/devdocs/reference/generated/numpy.quantile.html

Before you start

You will be able to

Review first

The idea

Three kinds of unusual value

How unusual is unusual?

In depth

Quartiles and the interquartile range

Tukey fences

Z-scores and masking

The modified z-score

Looking as well as computing

Treatments

Documenting the decision

In a spreadsheet or pandas

Formulas

Quantile by linear interpolation

Interquartile range and Tukey fences

Z-score

Modified z-score

Worked examples

Explore

Common mistakes

“An outlier is an error, so it should be deleted.”

“Impossible values are just very large outliers.”

“|z| > 3 will catch any extreme value.”

“Capping is always safer than deleting.”

“Quartiles are quartiles.”

Guided practice

On your own

In practice

Knowledge check

Summary

Key formulas

Terms introduced

Sources