By far, the most interesting discussion of the week for me started with a simple tweet:
My goal was pretty direct: I wanted to point people to a tool to help learning teams measure their comfort levels with data as a tool to drive instruction.
I created the survey with my learning team in mind because we’ve always struggled to figure out just what to do with data, and I’ll bet that other teams are in the same boat. The way I see it, if my experiences and resources can help other teams, life is grand.
Jon Becker—an assistant professor for Ed Leadership at VCU, a provocative, well respected scholar, and one of the people following my posts in Twitter—replied with a question that got me thinking:
The conversation that followed was pretty fascinating. First, I tried to explain to Jon that teachers aren’t hung up on validated tools. Instead, we’re looking for quick and easy resources that we can use immediately to start conversations with each other.
Here’s a piece of that conversation. For those who aren’t Tweeting yet, My comments to Jon are colored red. His comments to me are colored blue:
Over time, I started to wonder out loud whether the concept of validation is changing in the Web 2.0 world. To traditionally trained scholars, school leaders and experts, validation means putting your work through a semi-scientific review process that—as you can imagine—is time consuming and intimidating, covering questions like:
- Does this test accurately measure the surveyed domain?
- Does this test measure the range of behaviors possible within the surveyed domain?
- How close does this test correlate with other measures that it should theoretically correlate with?
- Can the creator of this survey demonstrate that it is valid by relating it to another measurement known to be valid?
- Has the survey creator controlled for any extraneous variables that may interfere with validity?
- To what degree can causal inference be generalized from the sample studied to the target population of the survey?
As a result, teachers and other “lay people” have traditionally been locked out of the community of creators.
Because we didn’t have the sophisticated understanding of the steps involved in making our work “valid” as defined by authorities, we were left to sit on the sidelines taking direction from those “in the know.” And because our society has always been driven by hierarchies, we had no chance of becoming recognized experts with worthwhile ideas to share.
Times are changing, though:
Interesting stuff, huh? This is a conversation that still has me thinking. Here’s what I’m wondering:
- Are there any real risks associated with using surveys and tools that haven’t been psychometrically validated? I’m pretty confident that teams and teachers are going to use any tool that they find to be valuable, whether it has been tested or not. Does that carry consequences that I haven’t considered?
- Has “empowering the masses” actually harmed education? I can’t even begin to explain how much I learn every day from people who have no formal credentials. Social media—blogs, Twitter, Skype, Facebook—give me instant access to minds that I wouldn’t have ever found before. But in a sense, social media cheapens the formal knowledge earned through systematic study of classical practices and texts. Will that carry any consequences that I haven’t considered?
- Do teachers stand a chance of ever being seen as the intellectual equals of those who have “advanced” within the existing system? Even as digital tools make it possible for teachers to raise their voices, our P-20 school systems still function as hierarchies—and in those hierarchies, teachers remain at the bottom with little organizational power. Does that mean our contributions will always be questioned by those who have “moved up?” If so, what consequences does that carry for our profession?
Looking forward to your responses….