Skewed data and transformations

In depth

Measuring skewness

Skewness is measured from the cubed deviations from the mean. Cubing keeps the sign of each deviation and exaggerates large ones, so a long tail on the right produces a large positive total and a long tail on the left a large negative one. This course, like Excel’s SKEW function and pandas’ skew(), uses the adjusted Fisher–Pearson coefficient described in the NIST/SEMATECH e-Handbook:

Here $m_2$ and $m_3$ are the average squared and cubed deviations (dividing by $n$), and $g_1$ is the plain Fisher–Pearson coefficient. The factor $\sqrt{n(n-1)}/(n-2)$ corrects for small samples and approaches 1 as $n$ grows. Symmetric data give values near 0; positive values mean right skew and negative values left skew. There is no universal cut-off for “highly” skewed: compare the value before and after a transformation, and always look at the histogram.

Because it cubes deviations, skewness is itself very sensitive to extreme values. In Lantern’s six orders (worked example below), the $80 order’s cube, $45^3 = 91{,}125$, is larger than the total of all six cubes, 84,210: one order is the skewness. Fix errors and impossible values before measuring it.

The logarithm

The log transformation replaces each value $x$ with $\ln x$. It turns multiplication into addition: values that differ by the same ratio end up the same distance apart. $10, $20, $40, $80 and $160 each double the one before, so their natural logs (2.303, 2.996, 3.689, 4.382, 5.075) are equally spaced, 0.693 apart (that is, $\ln 2$). A right-skewed variable whose values vary by percentages becomes roughly symmetric. The base doesn’t matter for the shape: $\log_{10} x$ and $\log_2 x$ are just $\ln x$ multiplied by a constant.

The log has one strict requirement: every value must be positive. The log of zero is undefined (it heads to minus infinity) and so is the log of a negative number. Order totals and hotel bills are positive; study hours, units returned or days absent often include zeros; profit can be negative. Those need other tools.

Log(1 + x), the square root and the reciprocal

The log1p transformation, $\ln(1 + x)$, adds 1 before taking the log, so a zero becomes 0 instead of being undefined. For large $x$ it behaves like $\ln x$; for small $x$ it is close to $x$ itself. Programming libraries provide it as a function, log1p, which is also more accurate than computing log(1 + x) directly when $x$ is tiny. For six students reporting 0, 1, 2, 4, 9 and 30 study hours a week, the skewness falls from 2.06 to 0.54 after log(1 + x).

The “1” is not neutral, though: it is one unit of whatever the column measures. Record the same students’ study time in minutes (0, 60, 120, 240, 540, 1,800) and log(1 + x) maps the zero to 0 and everyone else to between 4.1 and 7.5, making the student who studies nothing a new extreme value: the skewness becomes −1.34. Log(1 + x) works best for counts and for units in which 1 is small compared with typical non-zero values.

The square-root transformation, $\sqrt{x}$, is milder than the log and handles zeros naturally; it is a common choice for counts such as orders per day or rooms cleaned per shift. For the study hours above it reduces the skewness from 2.06 to 1.10, less than log(1 + x) does. The reciprocal, $1/x$, is stronger than the log; it reverses the order of the values (the largest becomes the smallest), so analysts use $-1/x$ to keep the order, and it is undefined at zero. It suits rates such as “hours per order”, whose reciprocal “orders per hour” may be the more natural measure anyway.

Box–Cox: choosing the strength from the data

Log, square root and reciprocal are members of one family of power transformations. Box and Cox (1964) wrote the family so that it changes smoothly with a single parameter $\lambda$ (lambda):

With $\lambda = 1$ the shape is unchanged (the values are only shifted by 1); $\lambda = 0.5$ behaves like a square root; $\lambda = 0$ is the log; $\lambda = -1$ gives $1 - 1/x$, a reciprocal that keeps the order. The Box–Cox transformation estimates $\lambda$ from the data, choosing the value that makes the transformed data look most nearly normal: scikit-learn’s PowerTransformer uses maximum likelihood, and the NIST/SEMATECH e-Handbook’s Box–Cox normality plot picks the $\lambda$ that maximises the correlation of a normal probability plot. Like the log, Box–Cox requires strictly positive values. For data with zeros or negatives, the Yeo–Johnson transformation (Yeo and Johnson, 2000), also in PowerTransformer, extends the same idea.

For left-skewed data, one option is to reflect the values first (subtract each from a number just above the maximum), transform, and remember that the order is reversed.

Reading results after a log transformation

Two facts let you translate log-scale results back into business language. First, a difference in logs is close to a percentage change when the change is small: $\ln(42/40) = 0.0488$, close to the true 5% increase. The approximation fails for large changes: $\ln(60/40) = 0.405$, against a true increase of 50%. The exact conversion is $e^{d} - 1$. Log differences also add up neatly: a rise from $40 to $44 is $+0.0953$ and the fall back to $40 is $-0.0953$, whereas the percentages, +10% and −9.1%, don’t cancel.

Second, the back-transformed mean is a geometric mean. If you average the logs and convert the average back with $e^{x}$, you get the geometric mean, $\left(x_1 x_2 \cdots x_n\right)^{1/n}$, not the ordinary (arithmetic) mean. For $10, $20, $40, $80 and $160 the mean of the logs is 3.689 and $e^{3.689} = 40$, while the arithmetic mean is $62. The geometric mean is never larger than the arithmetic mean, and the gap grows with the skew. So a model fitted to log(order value) and converted back predicts something closer to a typical order than to an average one; anyone who needs totals, such as revenue, must account for that.

When not to transform

• When the audience needs real units. “The mean log order value is 3.49” means nothing to a manager. Report medians, quartiles and percentages in dollars; transform inside the analysis, not in the slide.
• When the original units are the point. Totals add up only in the original units: revenue is the number of orders times the arithmetic mean order value. Forecasting revenue from a geometric mean understates it.
• For tree-based models. A decision tree splits on thresholds such as “total above $400”. A log transformation keeps the order of the values, so the same split exists as “log total above ln 400” and puts the same rows on each side: nothing changes except readability.
• When the skew is mild or the model’s errors are already well behaved. In regression, what matters is the shape of the residuals, not of every input. Check them before transforming.
• To hide errors. A log makes a 999-bag typo look less extreme without making it any less wrong. Fix errors and impossible values first; transform genuine skew only.

In a spreadsheet or pandas

• Spreadsheet: LN(x), LN(1 + x) and SQRT(x) in a new column; SKEW(range) for the adjusted Fisher–Pearson coefficient; EXP(AVERAGE(log column)) or GEOMEAN(range) for the geometric mean.
• pandas: np.log(s), np.log1p(s) and np.sqrt(s) transform a column; s.skew() measures skewness; PowerTransformer(method='box-cox') from scikit-learn estimates λ (use method='yeo-johnson' when there are zeros or negatives).