These past months, I’ve been running a Reading Group for the No Bullshit Guide to Statistics and this week we finally reached Section 3.4 that explains hypothesis testing procedures like the one-sample $t$-test. Hypothesis testing is a notoriously complicated topic. Without a doubt, it is the most difficult to understand topic in the STATS 101 curriculum.

I just finished preparing the exercises for Section 3.4 and I’m filled with pride for what I managed to accomplish. I can honestly ask readers to solve these exercises knowing that they have all the necessary prerequisite knowledge and experience to answer them. After multiple rewrites and endless editing, the text has finally reached the level that I aspire to in all my writing: it explains from first principles, in full detail, using only concepts that readers understand.

Context

The first draft of the book covered the same topic using the “standard” narrative: introducing the $t$ test statistic, using the standard Student $t$-distribution to compute the $p$-value, then making a decision to reject $H_0$ or not by comparing the $p$-value to a cutoff value $\alpha$ (usually $\alpha=0.05$). I re-read and edited this initial draft many times. My main skill as an educator is my ability simulate what readers can “handle” in terms of complexity, and no matter how smoothly I explained things, I couldn’t get myself to believe that readers will be able to follow all the steps. Too many new concepts; too many moving parts; too much math; and too much procedural complexity. I wanted the No Bullshit Guide to Statistics to be different, but instead I had reproduced the same impenetrable explanations as in the hundreds of other statistics textbooks.

The problem with the standard approach

Understanding the hypothesis testing procedure requires the following pieces:

• Knowing how to load data samples (e.g. using pandas)
• Knowing how to compute sample statistics like the sample mean $\overline{\mathbf{x}}$ and the sample standard deviation $s_{\mathbf{x}}$.
• Knowing the basics of probability theory (random variables, probability distributions, expectations, probability models and their parameters).
• Knowing about location-scale probability model families and the location-scale transform $\frac{X – \mu_X}{\sigma_X}$ that allows us to “standardize” any random variable $X$ within that family (e.g. any normal random variable $N\sim\mathcal{N}(\mu,\sigma)$ can be transformed to the standard normal $\frac{N-\mu}{\sigma} = Z \sim \mathcal{N}(0,1)$, which has mean 0 and standard deviation 1).
• Knowing that probability distributions can be used to model data distributions.
• Knowing about the sampling distribution of the mean $f_{\overline{\mathbf{X}}}$, which describes the variability of the sample means we might observe from i.i.d. random samples $\textbf{X} =(X_1,X_2,\ldots,X_n)$ from a the population $X$.
• Knowing about the central limit theorem (a math formula that describes the sampling distribution of the mean).
• Knowing the formula for the standard error of the mean $\mathbf{se}_{\overline{\mathbf{x}}} = \frac{\sigma_X}{\sqrt{n}}$, which is the standard deviation of the sampling distribution of the mean $f_{\overline{\mathbf{X}}}$.
• Knowing about the plug-in principle that allows us to “plug in” the sample standard deviation $s_{\mathbf{x}}$ into the place where the population standard deviation $\sigma_X$ goes, to obtain the estimated standard error of the mean $\widehat{\mathbf{se}}_{\overline{\mathbf{x}}} = \frac{s_{\mathbf{x}}}{\sqrt{n}}$.
• Knowing about Student’s $t$-distribution and its use for probability calculations based on the estimated standard error $\widehat{\mathbf{se}}_{\overline{\mathbf{x}}}$ when the true standard error $\mathbf{se}_{\overline{\mathbf{x}}}$ is unknown.
• Knowing about pivotal transformations (a type of location-scale transform) like $T = \frac{\overline{\mathbf{X}} – \mu}{ \widehat{\mathbf{se}}_{\overline{\mathbf{x}}} }$, which allows us to do calculations with the sampling distribution of the mean in terms of the standard Student $t$-distributions $\mathcal{T}(\nu)$ with mean 0 and scale 1.
• Knowing about the logic of hypothesis testing:
  • Formulating statistical hypotheses $H_0$ and $H_A$.
  • Computing the $t$-statistic from the sample $t = \frac{\overline{\mathbf{x}} – \mu_X}{ \widehat{\mathbf{se}}_{\overline{\mathbf{x}}} }$.
  • Choosing the appropriate reference distribution to use as the sampling distribution of the test statistic under the null hypothesis (spoiler: it’s the standard $t$-distribution with $\nu = n-1$ degrees of freedom).
  • Computing the $p$-value of the observed test statistic (or a more extreme value) under the sampling distribution under $H_0$.

That’s a tall order! Asking readers to take in all these concepts at once is akin to trying to fit an entire tree into your fireplace. It simply ain’t gonna fit! The only outcome you can expect if you try to fit an entire tree—with branches, leaves, and all—into your fireplace is to set your house on fire!

No wonder most statistics students end up confused by their first contact with hypothesis testing and resort to memorizing procedures and formulas. Most students in science, social science, business, etc. leave their first STATS 101 course without a proper understanding of the logic of statistical inference.

I wasn’t about to quit though! No way I’m going to let the future generations of students down. Something must be done!

My solution

I picked up a small axe and relentlessly chopped away at the complexity. I “factored out”  all the prerequisite concepts and frontloaded their explanations in previous chapters. The overall plan was to build up the reader’s understanding of all the moving parts before they get to Section 3.4.

• Section 1.2 introduces practical data manipulation skills like load datasets using pandas.
• Section 1.3 teaches readers how to compute descriptive statistics like the sample mean $\overline{\mathbf{x}}$ and the sample standard deviation $s_{\mathbf{x}}$.
• Student’s $t$-distribution is introduced as probability model in Section 2.6.
• Sampling distributions and the central limit theorem are first introduced in Section 2.8 as a probability concept.
• Sampling distributions are then covered again in Section 3.1 in the context of statistical inference.
• The standard error is also explained in Section 3.1 as well as the estimate for the standard error computed from the sample standard deviation using the plug-in principle
• Student’s $t$-distribution then appears again in Section 3.1 as an approximate model for the sampling distribution of the mean when using estimated standard error $\widehat{\mathbf{se}}_{\overline{\mathbf{x}}}$ instead of the true standard error $\mathbf{se}_{\overline{\mathbf{x}}}$.
• The logic of hypothesis testing is presented in Section 3.3 using simulation methods. This allows me to explain the key concepts like:
  • Statistical hypotheses $H_0$ (no effect) and $H_A$ (some effect exists).
  • Test statistic like the sample mean (presented as alternative uses of the descriptive statistics readers learned in Section 3.1).
  • Obtaining the sampling distribution of the test statistic under the null hypothesis using simulation.
  • Computing the $p$-value of the observed test statistic (or a more extreme value) by computing the proportion of the simulated test statistics under $H_0$ that are equal to or more extreme than the observed test statistic.

The overall complexity readers are exposed to is the same in my book as in other books, but by introducing the “moving parts” step by step, the complexity becomes manageable. In practice, this means 400 pages of prerequisites (Part 1 of the book), and a long Section 3.1 where sampling distributions are discussed at length. It’s a lot of work to get though all these prerequisites, but as samurai saying goes, when you cry during training, you can laugh on the battlefield. My aim was to provide a similar experience: build up the reader’s skills, so that by the time they get to the “battlefield” in Section 3.4, they feel totally at ease.

Did I succeed with my aim? I guess it’s not for me to say. I’ll have to wait until next week to get the feedback from readers, and to see if they managed to solve the exercises!