Home Subject: Experimental Research In Education: The Most Exciting Talk at the 2005 Joint Statistical Meetings To: EdStat E-Mail List ApStat E-Mail List Teaching Statistics E-Mail List sci.stat.edu Usenet Newsgroup From: Donald B. Macnaughton < firstname.lastname@example.org > Date: Thursday August 25, 2005 Cc: Mack Shelley, Candace Schau ----------------------------------------------------------------- For me, the most exciting talk at the 2005 Joint Statistical Meetings in Minneapolis (August 7-11) was an invited talk given by Mack Shelley, II of Iowa State University. The talk was enti- tled "Education Research Meets the Gold Standard: Statistics, Education, and Research Methods after 'No Child Left Behind'". Here's the abstract: This talk will inform the national debate over the role of scientific standards for research in education, par- ticularly as those standards are influenced by statisti- cal methods and theory. It will bring together research- ers in statistics and education to discuss the dramati- cally changing context of contemporary education re- search. Standards for acceptable research in this key area are affected greatly by creation of the Institute of Education Sciences in the U.S. Department of Education and passage of the No Child Left Behind Act of 2001 and the Education Sciences Reform Act (H.R. 3801). These re- constituted federal support for research and dissemina- tion of information in education, are meant to foster "scientifically valid research," and established the "gold standard" for education research. Greater emphasis in education research now is placed on quantification, as well as the use of randomized trials and the selection of valid control groups. This talk should sustain and ex- pand the dialogue between the statistical community and those who implement the education research agenda. The PowerPoint slides for the talk are at http://www.matstat.com/teach/Shelley.ppt Consider a definition: An experiment (or randomized trial) is a "proper" experi- ment if it has been performed according to generally ac- cepted principles of scientific practice and experimental design, as described by Bailar and Mosteller (1992), Box, Hunter, and Hunter (2005), Fleiss (1986), Kirk (1995), Winer, Brown, and Michels (1991), and many others. In the past, proper experiments in education research have gener- ally not been done, mostly because such experiments are somewhat complicated. Instead, educators have relied on observational re- search (and anecdotes) to support education policy decisions. However, this approach is unreliable, as illustrated by the New Math fiasco in the 1960's and early 1970's (Fang 1968; Kline 1973; Miller 1990; Stein 1996, chap. 12). Proper experiments in education research have two important ad- vantages over observational research: - Proper experiments are unequivocal, but observational research is invariably equivocal. Thus proper experiments in education research (for the most part) reliably increase our knowledge of how best to conduct an education program. (This is fully analogous to the way that proper experiments in medicine have greatly increased our knowledge of how to promote wellness and fight disease.) - Proper experiments focus attention on the important question of what we would like education to do for us. (This focus is through the choice of the response [outcome, dependent] vari- able[s] for the experiment.) Perhaps due to lack of knowledge, some education researchers con- tinue to perform less rigorous education research, which leads to wastage of time, wastage of opportunities, and wastage of money. In view of this wastage, and as noted by Shelley in his talk, the "What Works" arm of the United States Department of Education is beginning to rate education research projects on how well they satisfy the requirements of proper research. Each evaluated re- search project is given a rating on a three-point scale, with the levels being (1) Meets evidence standards, (2) Meets evidence standards with reservations, and (3) Does not meet evidence screens. Researchers planning to perform education research may find it helpful to read about the evidence standards to review or learn what's needed for education research to be "proper". The What Works program is described at http://www.whatworks.ed.gov An effective way for a less experienced researcher to ensure that their research is proper is to collaborate with another re- searcher who is familiar with experimental design and the pit- falls of education research. I discuss some issues about proper research in the field of sta- tistics education in appendices A and B and I discuss a way to substantially increase the power of statistical tests in educa- tion experiments in appendix C. Don Macnaughton ------------------------------------------------------- Donald B. Macnaughton MatStat Research Consulting Inc email@example.com Toronto, Canada ------------------------------------------------------- APPENDIX A: PROPER RESEARCH IN STATISTICS EDUCATION The discussion in the body of this essay applies to all areas of education. Of special interest to me is the area of STATISTICS education and in particular the introductory statistics course for students who aren't majoring in statistics. This course is important because statistics is a cornerstone of science, and thus proper understanding of the basic use of statistics in sci- entific research will give students a better understanding of science. Unfortunately, many students fail to understand the introductory statistics course. We know this by the experience that most sta- tistics teachers have with non-statisticians they meet, perhaps at a party -- many non-statisticians report that they took an in- troductory statistics course, but were totally lost. It seems clear that we can improve the introductory statistics course through proper designed experiments aimed at determining which selection of topics and which teaching approaches give stu- dents the greatest benefits. I discuss a useful response variable for experiments in statistics education in appendix B. Some researchers in statistics education do not perform experi- ments and they state that proper experimental research in statis- tics education is premature. If asked why experimental research is premature, these researchers give vague answers, such as say- ing that "preliminary work" must be done. I hope that research- ers who believe that proper experimental research in statistics education is premature will clearly spell out the steps they feel are necessary before this very important research can begin. It is wasteful to delay when so many students fail to understand our field. APPENDIX B: A USEFUL RESPONSE VARIABLE FOR EXPERIMENTS IN STATISTICS EDUCATION Candace Schau has developed the Survey of Attitudes Toward Sta- tistics (SATS). This survey, which can be administered to stu- dents in less than ten minutes, consists of 36 statements that students rate on a seven-point scale ranging from "strongly dis- agree" to "strongly agree". For example, the fifth statement is "Statistics is worthless." The SATS provides six reliable scores for a student that reflect the student's attitudes toward statis- tics on scales that are named Value, Affect, Cognitive Compe- tence, Difficulty, Interest, and Effort. The Value scale is particularly important because it reflects how highly students value the field of statistics. We might say the "best" introductory statistics course for a group of students is the course that most improves the students' scores on the SATS Value scale. This is reasonable because a student's sense of the value of the field of statistics (as instilled by a course) is arguably more important than any statistical knowledge instilled by the course. This is because statistical knowledge (e.g., how to do a t-test) is generally forgotten shortly after the student completes the final exam. But the student's valuation of our field usually lasts a lifetime and drives his or her decisions and remarks about the field. Administering and scoring the SATS is easy. Therefore, I encour- age every introductory statistics teacher to administer it to their students before and after each statistics course they teach. If you compute the average difference between the stu- dents' "before" and "after" scores, you can determine whether the course tends to make students' attitudes better or worse, and by roughly how much. The SATS is available at http://www.unm.edu/~cschau/satshomepage.htm (The results of the SATS can be disappointing because SATS scores in many courses are worse at the end than at the beginning. How- ever, for a seriously committed teacher it is useful to be aware of this problem and its extent as a stimulus to search for im- provements.) (On a technical matter, the differences between the students' "before" and "after" scores on a SATS scale are useful as a rudi- mentary measure. However, if SATS scores are used in analysis of variance, for complete information the raw "before" and "after" scores for the analyzed scale should be included in the analysis instead of merely using the differences between them.) APPENDIX C: USING REPEATED MEASUREMENTS TO INCREASE POWER Measuring the response variable in each student before the course begins and again at the end of the course (as discussed in appen- dix B) can substantially increase the power of the statistical tests in an experiment. This is because the repeated measurement of the response variable enables us (when other standard condi- tions are met) to use the statistical procedure of repeated meas- urements analysis of variance. This results in certain key sta- tistical tests being based on "within-student" comparisons, which generally provide substantially more powerful tests than the be- tween-class comparisons that may otherwise be necessary. (Despite the point in the preceding paragraph, it's usually nec- essary to study more than two classes of students in experimental research comparing education programs. We need more than two classes to eliminate the possibility of reasonable alternative explanations muddying the interpretation of the results. For ex- ample, we must eliminate the possibility of teacher differences accounting for significant differences in the mean values of the response variable for the different programs being compared. We can eliminate this possibility with multiple classes with multi- ple teachers. And [if proper random assignment of students to classes can't be done] we also need multiple classes to reduce the chance of a student-class selection bias.) REFERENCES Bailar, J. C., III, and Mosteller, F., eds. 1992. _Medical uses of Statistics._ 2d ed. Boston: NEJM (New England Journal of Medicine) Books. Box, G. E. P., Hunter, J. S., and Hunter, W. G. 2005. _Statistics for Experimenters._ New York: John Wiley. Fang, J. 1968. _Numbers Racket; The Aftermath of "New Math"._ Port Washington, NY: Kennikat Press. Fleiss, J. L. 1986. _The Design and Analysis of Clinical Experi- ments._ New York: John Wiley. Kirk, R. E. 1995. _Experimental Design: Procedures for Behavioral Sciences,_ (3d ed.). Pacific Grove, CA: Brooks/Cole. Kline, M. 1973. _Why Johnny Can't Add: The Failure of the New Math._ New York: St. Martin's Press. Miller, J. W. 1990. Whatever Happened to New Math? _American Heritage,_ 41(8) (Dec), 76-83. Stein, S. K. 1996. _Strength in Numbers: Discovering the Joy and Power of Mathematics in Everyday Life._ New York: John Wiley. Winer, B. J., Brown, D. R., and Michels, K. M. 1991. _Statistical Principles in Experimental Design_ (3d ed.). New York: McGraw- Hill.
Return to top
Home page for the Entity-Property-Relationship Approach to Introductory Statistics