Research Project for Statistics/Computer Science/Political Science C79

[Description] [Details] [Roles and Responsibilities] [Grading Criteria] [Due Dates] [Resources]

Description.

Broadly, the project is to unearth the science behind a current scientific news story that involves risk, to determine whether the news report was accurate, and to assess the credibility of the underlying science.

35% of your grade in this course is based on this project. We expect the project will take you 40 to 60 hours to complete—so start early.

The research project has an individual component and a group component. (Your grade does too.) For the group component, you need to work with three other students. One of you will be the team leader. The responsibilities of the team leader differ from those of the other team members; see Roles and Responsibilities, below.

 

Details.

  1. Be sure you have read Huff, D., 1954. How to Lie with Statistics, W.W. Norton & Co., NY. Read Chapter 28 of SticiGui.
  2. Pick your team. Teams should have four students. (Advice for forming study groups.)
  3. Pick a team leader.

  4. Read the New York Times, the Wall Street Journal, the Los Angeles Times, or another premier newspaper daily until you find an article that reports a scientific finding involving risk that interests you. “Scientific” is meant broadly: astronomy, biology, chemistry, earth science, economics, epidemiology, medicine, public health, political science, psychology, sociology, etc., are all fine.

    Scientific news items tend to fall into a few categories. One category is a research announcement, tantamount to a press release. The journalist covered a conference where a new result was announced, or had a preview of an article about to appear in a scientific journal. Such items usually cite the scientific literature, which gives you a toehold to start your research project. Another category is more like a review. Such articles tend to quote individual scientists, but often do not cite the scientific literature. It will be a little harder to do the background work for such an article, but by searching the academic literature for the names of the scientists who were quoted, you should be able to make progress. If an article does not cite the literature and does not quote experts by name, it will probably be hard to do the rest of the project. Note that your news article needs to be a current article (not a historical article from the past).

  5. Find the scientific source on which the popular article is based, or relevant articles by people cited in the news story, if the news story does not itself cite the scientific literature. The scientific source might be a book, a journal article, or a newly released study, for example. There might be more than one source cited by the news story; if so, pick one you find central to the claims. Make sure you can read the source.
  6. DUE 5 April 2013: The names, student ID numbers and email addresses of the people in your team, and the name of the student who will be the team leader, by email to Wayne Lee (lwtai@stat). Also, include a copy of the news item you propose to use, and a copy of the principal scientific source. We will inform you within a week whether your proposed topic and source are approved. If your item is not approved, we will be available to meet with you during office hours to help you select a topic and source.

  7. Each team member—other than the team leader—should find at least one scientific article he or she thinks is particularly relevant to the claims made in the source article. (Start with the bibliography in the main source article.) At least one of the articles should contradict the primary scientific source you are studying. The team leader is responsible for making sure at least one article is in this category.

    (For the article that contradicts the primary source, we prefer a scientific article—e.g., a book, journal article, or published study—if one is available, but if not, you can use another reasoned article that gives an opposing view and explains or summarizes the science and reasoning behind the opposing view. For example, a survey or opinion piece by a knowledgeable expert from that field would qualify, if it presents an opposing view and presents the reasoning and basis behind the opposing view.)

  8. The team as a whole should write several paragraphs summarizing the primary source article. Each team member (other than the leader) should also write several paragraphs summarizing the supporting article he or she chose. The summaries should include:
    1. the scientific question the article addresses
    2. the names of the authors, and their institutional affiliations
    3. the source of funding for the research, if known, and the nature of that entity (government? industry? private foundation?)
    4. a description of the data
    5. a brief description of how the data were collected (survey? experiment? randomized or not?)
    6. an overview of how the data were analyzed
    7. the main conclusions
    8. a synopsis of the argument that connects the data to the conclusions
    9. A bibliography item in APA style (www.apastyle.org/) for this article.

    The team leader should collect the paragraphs from the team members and circulate them within the team for peer editing. The team leader should assemble these paragraphs into a well formatted document. The names and SIDs of the students on the team should be on the cover page. Each set of paragraphs should start on a new page, labeled with the name and SID of the student who drafted them.

  9. DUE 16 April 2013, at the beginning of class: A written, well formatted document containing the summary paragraphs described above. Turn this in, on hard copy, at the beginning of class.
  10. DUE 18 April 2013: The team leader will give a 3-minute oral status report in class on April 18, stating the main point of the news item and your progress investigating it.
  11. Now the real fun begins. Read the source article more carefully. Determine whether the news item correctly reported the findings of the source article, including any disclaimers or qualifications. Note any gaps, overstatements or distortions. Read the other four articles carefully. Assess whether their data and methodology are trustworthy, and examine their arguments. Decide whether you believe the claims in the source article.
  12. DUE 3 MAY 2013 at noon: An 8–12 page group paper based on your team's work. The paper should include citations in APA style and a bibliography in APA style listing all materials you relied on (including the news item, the source article, the four other scientific articles, and all other materials you relied on). The paper should be submitted by email to David Wagner, in Adobe pdf format. We will not accept Microsoft Word files.
  13. DUE DURING READING WEEK (6 MAY 2013): A 10-minute oral presentation by each team, in which all four team members participate. The presentations should have 10 minutes of material presented by the team and 5 minutes open for questions from the instructors and the other students. The time for your presentations will be scheduled in late April. We will schedule two 2- or 2.5-hour blocks for presentations. Everyone is expected to attend both blocks. The presentations will be in 20 Barrows Hall, from 10am-noon and 1-3:30pm.

Roles and Responsibilities.

The team as a whole is responsible for:

The team leader is responsible for:

Other team members are responsible for:

Grading Criteria.

All writing will be graded for content, accuracy, logic, clarity, brevity, spelling, grammar and style. Individual grades will be based 50% on the quality of the paragraphs that student drafted, and 50% on the quality of the team report as a whole.

An A+ team paper would address the following—thoughtfully, not perfunctorily:

  1. Did the news report present the science accurately? Were there errors, distortions, or important omissions? Did the journalist see the big picture? Or did he or she cherry-pick, perhaps to make the results seem more sensational?
  2. Who says? Is the source reliable on its face? Is the research funded by industry or government or private foundation or ...? For whom do the authors work? Is an axe being ground here?
  3. Does the result pass the "sniff test?" Is it prima facie plausible? What experience or knowledge do you have have that could be used to check whether the conclusion is reasonable?
  4. What evidence do the authors have? What are the data? Was the data collection fair, or possibly biased? Was sampling used? What was the population? The frame? Might frame bias be an issue? What were the sampling units? The sampling design? Was it a sample of convenience? A quota sample? A systematic sample? Was the sample self-selected? Was the sample drawn at random? How big was the sample? What was the response rate? Is nonresponse bias an issue? Are the authors extrapolating beyond their sample? Beyond the sampling frame?
  5. Is this an experiment or an observational study? Is it a controlled experiment? If there were controls, were they historical? Is this a longitudinal study? A cross-sectional study? If there were human subjects, was the experiment blind? Double-blind? Were subjects assigned at random to treatment and control? What was the actual wording of any survey questions? Do the questions seem to be worded to elicit particular responses?
  6. What methods were used to analyze the data? Were the methods appropriate? Are the methods generally reliable? Are there situations in which those methods tend to distort the picture? What are the assumptions of the methods? Were the assumptions justified in this case? Would violations of the assumptions matter much? If so, can you figure out how plausible violations of the assumptions would affect the conclusions?
  7. Is confounding likely to be an issue? What variables might be confounders? Did the authors control for the confounders? If so, how? Might aggregation cause confounding here? Is Simpson's paradox an issue? Is the ecological fallacy an issue?
  8. What is the logical structure of the argument? Is it sound? Are there obvious gaps or errors in the logic? Is causation confused with association? Did somebody change the subject? Are there realistic possibilities not contemplated by the argument?
  9. Would it be possible to reproduce the research from the description in the article? If not, what's missing? Were ad hoc choices made that might influence the results?
  10. Do the authors confuse statistical significance with practical significance? Do they report uncertainties? Are their uncertainty calculations appropriate? Do they take into account sampling error? Bias? Specification error (error from using the wrong variables or the wrong model)? Do the uncertainty calculations depend on questionable assumptions about the origin of the data (e.g., independence)?
  11. Are there other explanations of the data just as plausible as those advanced by the authors? Did the authors consider those explanations? Did they have good arguments against those competing hypotheses?
  12. Were the results presented in a neutral way, or were technical devices used to make things look more impressive (gee-whiz graphs, one-dimensional pictures, ...)?
  13. Did the authors cite papers that disagree with their conclusions? Do the authors seem circumspect, or do they seem to be stacking the deck?
  14. What study or experiment might be done to confirm or dispute the research? What would be involved in conducting that research? Is it feasible?

Oral presentations will be graded for clarity, style, and appropriate use of graphics.

Due Dates.

Resources.

Newspapers:

Article Databases:

Other resources:


Copyright 2006–2013. P.B. Stark. All rights reserved.

Last modified 21 March 2013