# Ling 250/450: Homework 3

**Due: Monday April 6, 11pm**

For this homework, you will read the short article titled [The Perils of Post-Hockery](https://ruscio.pages.tcnj.edu/files/2016/08/Ruscio-1998-SI-Post-Hockery.pdf) (linked here) and answer some questions about it. The article is about 8 pages of actual content — short enough to read in one sitting.

This document is written in Markdown format (.md), which is encoded as plain text but can be displayed with nice formatting. To view the Markdown file side-by-side with the rendered version, open it in VS Code and use the shortcut ⌘+K then V on Mac or Ctrl+K then V on Windows.

You should submit your completed work **on Blackboard, as a PDF**. To convert your Markdown file to a PDF, install the "Markdown PDF" extension in VS Code (Settings > Extensions, search "Markdown PDF"). Then use shift+⌘+P (Mac) or shift+Ctrl+P (Windows) to open the Command Palette, search for "Markdown PDF: Export (pdf)", and run it.

Here is a guide for writing in Markdown: https://www.markdownguide.org/basic-syntax/

---

## Q1

In your own words, what is "post-hockery" as Ruscio describes it in this paper? What is the key distinction he draws between the **context of discovery** and the **context of verification**, and why does conflating them lead to bad science?

## A1

---

## Q2

Ruscio describes the "hot hand" belief in basketball — the idea that a player who has made several consecutive shots is "on a streak" and more likely to keep making shots. What does the research evidence actually show about the hot hand?

## A2

---

## Q3

What is the difference between the **strong (predictive)** version of the hot hand belief and a weaker, more **descriptive** version of it? What piece of evidence does Ruscio say is missing when players, coaches, and fans talk about someone having a hot hand?

## A3

---

## Q4

Ruscio raises concerns about "computerized data analysis" in scientific publishing. Why does he describe this as potentially problematic? What habits does it promote that interfere with rigorously testing hypotheses? *(Keep in mind this was written in 1998, when automated statistical testing was newer than it is today.)*

## A4

---

## Q5

Describe **confirmation bias** in your own terms. What two mechanisms does Ruscio identify that allow confirmation bias to go uncorrected over time?

## A5

---

## Q6

Do you hold any low-stakes beliefs — like astrology, personality tests (Myers-Briggs, Enneagram), or folk explanations like the "blackout baby boom" — that you know the evidence probably doesn't support, but are more fun or satisfying to believe in anyway? Describe one, and reflect briefly on why confirmation bias makes these beliefs sticky even when we know better. (There is no wrong answer here — this is a reflection question.)

## A6

---

## Q7

The section "Hindsight Bias and Overconfidence" includes a calibration exercise from Plous (1993). Try it: for each of the 10 questions in that section, write down a **90% confidence interval** — a range you are 90% sure contains the correct answer. Do *not* look up the answers first.

After you have written down all 10 intervals, look up the answers and count how many of your intervals contained the true value. Report that count here, and briefly reflect: were you overconfident, underconfident, or well-calibrated? What does this imply about how we estimate uncertainty in general?

*(If the specific questions feel too dated, you may instead come up with three similar questions of your own — things you don't know the exact answer to — and report your calibration on those.)*

## A7

---

## Q8

Ruscio raises concerns about running many statistical tests on the same dataset. Suppose a researcher measures 20 linguistic variables and computes every pairwise correlation. How many statistical tests are being run? If the researcher uses a significance threshold of p < .05, how many of those tests would you expect to yield a "statistically significant" result purely by chance, even if none of the variables are actually related?

What does this imply about how we should read studies that report significant correlations without disclosing the total number of tests run?

## A8

---

## Q9

Why is this paper assigned in a Data Science class, during our unit on hypothesis testing? What is at least one concrete takeaway from this paper that you can keep in mind as you develop a hypothesis to test for your term project?

## A9

---