Adding Friction. How Can I Teach Students to Think of Numbers as Evidence Rather than Answers?

by Debbie Abilock, March 2016

Students usually include population figures in state or country reports but operate on the naïve belief that such statistical data is accurate. Since they believe that an authoritative answer exists, they see no reason to seek and compare multiple sources of data (Bråten, Strømsø, and Samuelstuen). Instead they make arbitrary choices about what statistics to use and subjective decisions about how to display them. They often miss the subtle distinction between precision and accuracy and fail to ask larger questions about context, motivation, and bias.

Let's look at a short-term research project designed by a social studies teacher, a math teacher, and a librarian. Positioned just before the longer country report, it provides just enough friction to help students recognize that their choice of statistics and visualization of the data is a reasoning process rather than a representation of indisputable truth. The assignment requires students to justify a statistics choice on the basis of several criteria and then create a meaningful visual display of the data. The teachers choose to focus on the Koreas, a single country until after World War II, which present remarkable contrasts.

Students are divided into teams and asked: "What is the difference in population between North and South Korea?" Each team is to produce three products:

1. A visual comparison of total populations for both countries. Each team can choose what population data to use and how to display it.
2. An explanatory caption below the visual.
3. An annotated bibliography which describes how the numbers are derived, and how confident the team is about the accuracy of their data.

Finding Data

Click to Enlarge +Adding Friction. How Can I Teach Students to Think of Numbers as Evidence Rather than Answers?

The period is divided into three, repeating segments: 10 minutes of work; 5 minutes of whole-group debriefing, and 15 minutes for peer reviewing. Initially some students choose familiar sources such as the CIA World Factbook (#1) and the open web (#2). They find slightly different population figures — the first quite precise, the second clearly rounded off.

Having been told that they need to justify the data they use, students broaden their searches, look for corroboration, and paste results into a common team matrix to make comparisons easier. At first, most matrices contain only three columns: a URL and two numbers. Some teams notice that other information (date of publication, data sources) provides useful points of comparison and gather additional information from their sources.

Evaluating Data

Some students propose using the CIA World Factbook, the source with the most recent publication date, as their single selection criteria. Other groups, discussing ways to account for the numerical discrepancies, notice that some numbers are labeled as estimates and decide to provide a range. As groups look more closely at how the data had been compiled, other criteria emerge:

1. Two sources (#2, 3) draw only from World Bank data (#4, 5) but only Google Public Data (#3) numbers match the World Bank's. Since they also have population data directly from the World Bank, they discard both derivative sources.


2. Two sources collect their own data: the CIA World Factbook (#1) and the UN (#5), so students burrow down into the background FAQs to examine their collection process:
   
   
   • CIA World Factbook uses the U.S. Census which is gathering data in the process of assisting countries to improve "their national statistical systems by engaging in statistical capacity building activities" (U.S. Census Bureau).
   
   
   • UN data is self-reported "official statistics produced by countries," compiled by the UN statistics division and other UN agencies ("UNdata: Frequently Asked Questions").
     
     
   • Students question whether self-reported country data might be colored by the purpose of collection. For example, a country might not report information accurately if it had something to hide, if it was suspicious of the organization's motivation, or if inflating the numbers would result in increased foreign aid


3. Two sources compile data from multiple, independent sources (#4, #6).
   
   
   • The World Bank (#4) transparently lists its sources which include both the Factbook and UN data, as well as country-reported and regional data ("Population, Total").
   
   
   • Wolfram|Alpha (#6) uses proprietary computations to calculate answers using both the UNdata (#5) as well as other unnamed sources which it stores in an "internal knowledge base" ("Frequently Asked Questions"). It was unique in providing suggestions for visual comparisons of general population data and suggesting related data to explore.

Contextualizing Data

All data exist within a context. As exploration continues, other questions emerge. How could two sources (#1, #4) provide such precise numbers? How certain can students be about population data from a country controlled by a hereditary dictator ("North Korea Profile - Overview") that "treats even the smallest piece of information as a state secret" (Frank)? By the end of three rounds, the teams agree that they can advance competing arguments about the data's validity.

During the final 15 minutes, student teams exchange their products and give feedback using this rubric. (See "Use This Page" for a copy of this rubric.)

Reasoning with Data

This short research project adds friction to the process of selecting and using data. Consider these information literacy essential questions: Who publishes data? For what purpose and under what circumstances? Discussions of profit, power, and purpose emerge as students reason with data. When students initiate their own country research, they will think more deeply about statistical data in light of government transparency (e.g., Denmark, China), data collection practices during periods of social and political instability (e.g., Sudan, Somalia), and how disruption caused by armed conflicts (e.g., Afghanistan, Iraq) might influence their confidence in something as "simple" as the population of a country.

Common Core Standards: Research to Build and Present Knowledge:

CCSS.ELA-Literacy.CCRA.W.7
Conduct short as well as more sustained research projects based on focused questions, demonstrating understanding of the subject under investigation.

CCSS.ELA-Literacy.CCRA.W.8
Gather relevant information from multiple print and digital sources, assess the credibility and accuracy of each source, and integrate the information while avoidingplagiarism.

CCSS.ELA-Literacy.CCRA.W.9
Draw evidence from literary or informational texts to support analysis, reflection, and research.

Works Cited:

Bråten, Ivar, Helge I. Strømsø, and Marit S. Samuelstuen. "The Relationship between Internet-Specific Epistemological Beliefs and Learning within Internet Technologies." Journal of Educational Computing 33.2 (2005): 141-71. Sage Journals. Web. http://jec.sagepub.com/content/33/2/141 (accessed November 16, 2015).

Frank, Ruediger. "A Question of Interpretation: Economic Statistics from and about North Korea." 38 North. U.S.-Korea Institute at SAIS, 16 July 2012. http://38north.org/2012/07/rfrank071612/ (accessed November 16, 2015).

"Frequently Asked Questions." Wolfram|Alpha. Wolfram|Alpha, n.d. https://www.wolframalpha.com/faqs.html (accessed November 16, 2015).

"North Korea Profile - Overview." BBC. BBC, 15 Sept. 2015 http://www.bbc.com/news/world-asia-pacific-15258878 (accessed November 16, 2015).

"UNdata: Frequently Asked Questions." UNdata. United Nations Statistics Division, n.d.. http://data.un.org/Host.aspx?Content=FAQ (accessed November 16, 2015).

U.S. Census Bureau. "Fact Sheets." United States Census Bureau. U.S. Dept. of Commerce, n.d. https://www.census.gov/population/international/about/factsheets.html (accessed November 16, 2015).

"Population, Total." World Bank. World Bank, 2015. http://data.worldbank.org/indicator/SP.POP.TOTL/countries/ (accessed November 16, 2015).