Whose Horse in the Race?

Among the many pitfalls of high stakes testing, the one that troubled me the most when I was teaching high school (in Kentucky) was misplaced accountability. My students’ performance on annual Commonwealth Accountability Tests (CATS) had real consequences for me, but not for my students. Teachers and schools were assessed, and actions taken,  based on student scores. What difference did the score make for the student? Scores weren’t used to determine advancement or placement. We couldn’t have used the scores to determine grades, the test scores weren’t released in time. Teachers and schools may have created ways to incentivize performance, but there weren’t any natural consequences to students built into the testing program. Begun in 1999, the explicit purpose of the CATS testing was to hold all schools accountable to achieve at least a pre-designated proficiency target score by 2014.

Are they going to make it? Well, if you set a goal and see you aren’t going to make it, it’s a lot simpler to change the goal than to change what you are doing. The CATS tests were completely overhauled in 2007, making it impossible to compare scores before and after that change. CATS was dropped completely for 2011. And so goes the trajectory of every major reform that moves accountability away from students. I’m reminded of a cartoon I saw when I was teaching that stuck with me. An angry father is looking at a low test score his son has brought home and says: “Son, I am very disappointed... in your teacher.” I’m sure I chuckled when I first saw it, but the effect of removing responsibility is serious -- moral -- stuff. When the community in which a child is developing says, in effect, someone else is responsible, then he will feel relieved of that obligation and his efforts will reflect that.

Is that true? Do student scores on tests correlate with the consequences of the test for the student? In a previous post I cited a recent study by ETS called Motivation Matters in which a proficiency test was given to three groups of college students each with a different consent form. The group given a consent form that said, in effect, your score on this test will affect your future employment did significantly better than the group given a consent form that said scores would be kept confidential. It makes sense: if the score matters to the student she will work harder. Of course I felt this teaching school. My job became about motivating students to do well on the test -- because there were no intrinsic reasons for students to do well. How I would have liked to merely help students see the direct relationship between their performance and their own future goal -- if only there had been such a relationship!

Most frustrating of all, the more I pushed as a teacher, the less reason my students had for taking responsibility themselves. If it was my horse in the race, why should my students worry over it? This brings me to another very recently published study. This large, well conducted study at representative colleges found what, at first blush, is a surprising relationship. Students whose parents paid more toward college had lower GPAs than those whose parents paid less. The careful and thorough analysis came to this conclusion: students who had to pay more had a horse in the race. OK, those are my words, but the conclusion boiled down to the student having a greater or lesser stake in the outcome of each class she took. Now to be fair, the study revealed a seemingly opposite relationship between graduation rates and parental investment. You can sort through the fine print in the article itself. Suffice to say I don’t think this finding undermines the conclusion that students who have a greater stake in the outcome will perform better. In my previous post I went so far as to suggest that we might more accurately think of tests as measuring motivation than as measuring skills or knowledge. The same, I think, goes for GPAs. If we look closely I think we’ll find that student GPA tracks more closely with motivation than with skills and knowledge. If only I was sufficiently motivated to do the study!

I Need an Intervention

First semester freshman year of college was a real wake-up call. It wasn’t obvious how to handle the sudden and new found freedoms. The open blocks of time were an invitation to feel at ease -- there would be plenty of time later. Presently we could play games -- frisbee in daylight and ping pong at night. I certainly attended class -- though not everyone did. No one was making us do anything. It turned out that attendance wasn’t enough. I did not do well on the calculus and chemistry exams. About two-thirds of the way through the semester it hit me: I needed an intervention. I sought help. I dropped the calculus class. I enrolled in small group chemistry tutoring that would allow me to take an incomplete in chemistry. I ended that first semester having only completed two courses. When my coach, having been alerted to my academic struggles, expressed concern, I reassured him that I had a plan to get back on track. He didn’t mention it again. He didn’t have to. He had been around, after all, and had seen this plenty of times before.

Successful college students who struggled at first must have changed something. Otherwise they would not have gone on to be successful college students. Many do like I did -- they seek help to make a plan and get back on track. Colleges offer help in many forms including tutors, learning centers, support groups, mentors, etc. We don’t doubt that these interventions help. Students who seek them out do improve. But why are these programs effective?

I just returned from a Lilly conference with the theme “evidenced-based learning and teaching.” The onus was on presenters (including me) to show, with data, the effect of our efforts. I attended many rich and engaging sessions, including one by Saundra McQuire from LSU. Her career has consisted primarily of teaching chemistry though in recent years she has worked with students at LSU’s Center for Academic Success (CAS). I thoroughly enjoyed the presentation which was about teaching metacognitive strategies to students. Dr. McGuire radiates positivity. So of course I didn’t raise my hand to object to the major flaw in her presentation. Folks around me were enjoying her as much as I was. I have no doubt that she is well loved by the students, staff and faculty around her. I am sure that the CAS is an important component of student success at LSU. Still. It isn’t fair to use data showing student improvement after an intervention as evidence that the intervention caused the improvement. The evidence for why we should teach students particular metacognitive strategies, Dr. McGuire suggested, was because students who were taught these did better afterward.

To which I wanted to say: of course they did. Your sample of students was self-selected to be those who, faced with an eye-opening problem, sought help to address the problem and do better. Nearly any intervention, so long as it is reasonably well intentioned, is likely to show a similar kind of success -- because people chose it who were poised to improve. This doesn’t defeat the purpose of having learning centers (or tutors, or study groups, etc), but it does suggest we ought to be careful about what we claim when we measure the effectiveness of such interventions -- and to whom we attribute responsibility for student success.

"Thatta Boy" Better Than Cash?

I had to wait for it, but my twelve-year old finally started a real conversation with me about school work. He had written a persuasive essay about whether or not students should get paid for good grades. He asked me to read his draft. Naturally I was quite interested. He argued the con, citing the incentive to cheat. We talked about that. Cheating is bad, he thought, an unintended consequence of paying for grades that detours kids around the purpose of school. Getting paid for grades puts the emphasis on the wrong thing, I thought, namely getting a high grade instead of learning something new. If grades or for that matter test scores are what we care about, then why not utilize any means at our disposal for getting good grades or high test scores? In his notes he held the trump card that effectively combined our two points of view. It was a quote he found from a 2008 editorial by Diane Ravitch on Forbes.com. In it she contrasts the intrinsic motivation of those in India to the extrinsic motivation of Americans, and concludes: "Does the future belong to those who struggle to better themselves, make sacrifices to do so and work hard? Or to those who must be cajoled and bribed to learn anything at all?" 

If you scour the internet (as I have just done) to try and decide the issue you will find ample support that paying kids to get good grades is a bad idea. The arguments boil down to this: young people need to develop the intrinsic motivation to learn. While I agree, I also sense a kind of moral undercurrent. I think people have a gut reaction about paying kids for grades -- that it is wrong because kids ought to be pursuing education for loftier reasons.

But think about this more deeply. What else (and I'm talking to everyone EXCEPT the student) have you got to offer? If you are trying to get a kid to do something he wouldn't do on his own (and what else would be the point of education?), what kind of leverage do you have? Isn't anything you offer a reward of some kind -- and not so different from offering money? Most all schools assign grades. How are grades different from monetary rewards? Many families probably do the math anyway, estimating the added (monetary) value to future careers of good grades now. 

Perhaps the most innocuous reward is attention of some kind from the teacher. Need a chart of rewarding phrases? Here's a link to 275 ways to say "good job!" If kids do it so that you'll say "good job" or some variant, how is that different from doing it so you'll give them 10 cents? What do we really mean by "intrinsic motivation?" The idea seems easy: wanting to do something without the need for any external reward, but what does that mean? Don't we ultimately do everything for something? You might like to read for its own sake, but doesn't that just mean that you get a little boost of some pleasure chemical in your brain when you read? Meaning that you still do the activity for a reward -- it just happens to be a chemical reward. Cash rewards come from "outside," but isn't cash ultimately convertible to deeply intrinsic interests -- meaning kids can spend cash on things of value to them?

What do you think?

Training Effects

One of the bedrock principles of training is that it is specific. A demand is imposed on the system and the system, to the extent it is ready and able, adapts to that demand. To train for an endurance event, for example, I can perform a series of tasks that demand higher-than-normal energy production. The human body is very well equipped to respond to such a demand by ramping up the production of hormones and enzymes that encourage the production and storage of energy. After that series of tasks, therefore, I will have adapted to the specific demand for sustained energy. An endurance event will pose a whole variety of challenges to the athlete, however, that are considerably more complex than any single demand to which the body can adapt. In order to run a fast marathon an athlete would need additional adaptations in bony and connective tissues, cardio vascular components, neurological components, and of course muscular tissue.

Muscle tissue adapts quickly – on the order of weeks. Connective tissue adapts slowly – on the order of months. Training gains and losses therefore depend on the nature of the adaptation. They also depend on the readiness of the system to respond and adapt. Sports provide the framework for contriving and testing all manner of adaptations and for comparing the differences between people who compete. The whole premise of sports is that differences between people will demonstrably emerge.

Schooling, at its best, is training. It places demands on students who then adapt to those demands. A highly trained student is just like a highly trained athlete: they are both primed for competition.
When demonstrable differences emerge between competitors who have similar access to resources, whether athletic or academic, we attribute that to differences in talent, motivation, or a combination of the two. If we were interested in developing an Olympic training program for athletes, we might begin by trying to screen young people for talent. We would hope to reveal the people most likely to respond to very specific kinds of training for specific sports. We could administer general fitness tests to kids and get a rough measure that is likely to correlate with future responsiveness. A more refined screening instrument would probably need to include a direct measure of responsiveness, though, to specific kinds of training. For example, kids are pre-tested, given a training regimen, and then post-tested to measure the gains made through training.

Though we don’t tend to, we could look at schooling the same way. If we are interested in developing competitive thinkers in a variety of fields, we might begin by trying to screen young people for the likelihood that they will respond to specific kinds of academic training. One such screen is the IQ test. From its inception the IQ test has been designed to screen for those most likely to adapt to school generally. [Technically Alfred Binet was commissioned by the French government to screen for the converse – students who would struggle in school.] Perhaps it wasn’t worth the added expense to get the more refined result we might expect if Binet (and his successors) had directly measured how students respond to specific kinds of training.

The cost of testing is likely less of a road block than issues surrounding the concept of intelligence. Problems arise because of the inference that intelligence is fixed and that it determines academic success. Although I am unaware of any actual pronouncements that intelligence works this way, I am well aware of the constant barrage of pushback against the possibility that intelligence may be conceived of in this way. There appears to be a raft of studies seeking confirmation that intelligence is not fixed and that academic success can be achieved by all people regardless of any perceived biological constraint. And the results of studies that do not directly pertain to intelligence and academic success are combed through for any hint that people are indeed freed from the shackles of biological determinism to pursue whatever may interest them.

So I wasn’t surprised by the blog that showed up on my Scientific American News Feed titled: “Virtues of Cognitive Workout: New study reveals neurological underpinnings of intelligence.” In it, Samuel McNerney sets up his interpretation of a recent study with this narrative: “For decades researchers believed that fluid intelligence was… largely determined by genetics. The implication of [this 2008 study] suggested otherwise: with some cognitive training people could improve fluid intelligence and, therefore, become smarter.” In his review of the recent study, 17 participants trained over 3 sessions to perform a mental task that requires working memory. Not too surprisingly, these participants improved on a test of fluid intelligence that requires working memory. This finding resonated with the chorus of those building a bulwark against an imagined enemy -- biological determinism.

To me, this finding is analogous to any training effect we’d expect from tissue that adapts quickly to imposed demand. It is pretty silly to generalize from this result that we can, in effect, train anyone to accomplish whatever academic goal we happen to value, especially if the goal is to be competitive with others. Here’s the problem: among the 17 who were trained to use working memory some improved more than others. Further, working memory adapts quickly and specifically to the task imposed, which means any gains will be quickly lost once the demand is removed. The results of this study do not mean anything about the relative performance of different people on complex tasks over the course of a lifetime.

The only thing that can be said to determine success in life is motivation, and we don’t even know how to talk about what that really is, much less where comes from. I’m working on a framework to at least have an illuminating conversation about it. Stay tuned.