Statistically Snookered….

Here in my hometown, a conservative think tank called The John Locke Foundation often challenges my thinking. The foundation is one of our state’s most outspoken critics of public education, and I find myself drawn to their work because it often forces me to refine and revise my own positions on education policy topics ranging from charter schools to standardized testing. 

One of their recent research briefs—written by Terry Stoops and titled Learning About Teacher Pay: N.C. teachers are favorably compensated; what they need is merit-based paycaught my attention today because it tackled a topic I’m trying to learn more about: How districts can create innovative ways to pay teachers differently. 

It didn’t take long, however, to realize that Stoops wasn’t interested in abiding by the Locke Foundation’s mission of working “for truth, for freedom, and for the future of North Carolina." Instead, he seemed committed to spin and to pushing politics.

What gave it away? His careless use of statistics to "craft" a position and to draw tenuous conclusions. 

Take this assertion, for example, found on page 4: "In addition, states in the region that compete with North Carolina for teachers rank significantly lower in adjusted average compensation. South Carolina is tied for 21st with Alabama, Maryland ranks 30th, Virginia ranks 33rd, and Florida ranks 36th in the final ranking."

Now, a tip I learned long ago from Gerald Bracey’s book on educational statistics was to be openly skeptical of rankings.  All too often, the difference between spots on ranking lists can be insignificant and yet seem overwhelming. 

In this case, the difference between North Carolina’s rank (19) and South Carolina’s rank (21) seems to imply that teachers in North Carolina are compensated far better than teachers in South Carolina. In reality, the difference between the adjusted average compensation of teachers in both states is slightly over $600. That could hardly be considered "significantly lower."

Even more interesting is the complete lack of information in this piece on how merit pay will benefit teachers. Despite its prominent position in the subtitle, merit pay is mentioned first in the closing sentence of the research brief when Stoops writes, "A system of merit-based pay, along with streamlined certification requirements, would provide generous incentives for highly qualified individuals to enter and stay in the teaching profession."

This is another statistical trick: Use seemingly convincing numbers to draw completely unrelated conclusions. 

While the figures that Stoops has collected may show that teacher compensation in North Carolina is comparable to compensation in surrounding states, he fails to prove that this compensation is "favorable" (heck…even uber-conservative, anti-public school critic Jay Greene cited Raleigh-Durham teachers as underpaid compared to other professionals in a recent policy report) or that merit pay is an effective solution to the staffing problems facing our state’s classrooms. 

There is also no data included to support his claim that merit pay will "provide generous incentives" that keep teachers in the classroom—outside of one unconvincing quote from another research report referenced in his footnotes: “The fact that the [merit pay] program appears to have reduced departure rates of teachers from the schools serving disadvantaged and low-performing students means that the program could potentially have raised student achievement had it remained in operation for a longer period of time.”

My concern is that conversations in education are increasingly driven by statistical tricks such as these. Advocates on both sides of every issue paint incomplete pictures based on limited evidence that is often biased and unreliable. Few partisans can be trusted, yet the general public is incredibly trusting. 

Is there any hope for unbiased school reform carried out in a spirit of genuine cooperation?