Last Monday in Liberation Math, the class, we had a great conversation about Logan LaPlante’s “hackschooling” TED talk. It’s a great talk, and I highly recommend checking it out. LaPlante is a winner and is doing great things with his education. He makes it all look easy, but extraordinary resources go into making his hackschooling education possible. His family skis a lot, which tells us that they have an income that makes that possible. He has an extensive network of opportunities, which means that his family is aware of the opportunities, that they are available to people like LaPlante, and they have the time and funding to allow him to access all of those opportunities.
LaPlante is a kid who is likely to be successful no matter what system he finds himself in — he’s charming and well connected. We can’t use him as a test case if we want to figure out if hackschooling is useful for a broad range of students because he is not a typical student, and he’s certainly not one of the millions of students struggling to successfully complete their education. Of course, it could be argued that if LaPlante was forced into a soul-crushing educational system that he might have begun to have difficulties. A bad education might have hurt LaPlante and decreased his chances at success in education and in life.
The fact is that educational innovations and interventions work — nearly all of them work, which doesn’t mean that we should implement every educational fad. We spend a lot of money on educational “fixes” that don’t really change things very much, often at a high cost. And the impact of the intervention depends a great dean on the population you start with. Suppose we take 100 well-resourced students like LaPlante, 80% of whom were going to be successful without intervention. If we do an intervention with these students that shows a 25% rate of improving student outcomes, then we see the following:
You are a student in this group who is doing fine or better. Was it the intervention that made the difference for you? To calculate the probability that the intervention moved you from struggling to fine, given that you are a student doing fine or better, we have to take the number of students who are fine in the end but would have struggled (5 students), and divide by the total number of fine or better students (85 students). This gives a 6% chance that the intervention is what made the difference here. We have to provide this intervention to 20 students in order to move a single student from struggling to fine. If this is an expensive intervention, that may be impractical for the results that we get, but will be entirely worth it if you are that one students and you possess the resources for change
Now imagine a population of students that has far fewer resources and experiences greater struggles and a greater likelihood of failure. Let’s say this new population of students has only 20% who are going to be successful with no intervention. We’ll imagine the same intervention that helps 25% of students:
In this new situation, if you are a student doing fine or better, what is the likelihood that your performance is a result of the intervention? Here again, we take the number who were struggling and are now fine (20 students) and divide by the total number who are doing fine or better (40 students). This gives a 50% chance that your positive outcome could be credited to the intervention, a big difference from the previous population of students! Here we need to provide the intervention to just 5 students to move a single student from struggling to fine, giving a much greater efficiency. Perhaps this hypothetical intervention now looks great, but if it is an expensive intervention, requiring a lot of human capital, we still may not be able to provide the intervention to a broad range of students who need it.
How effective does this intervention appear to be? If it is implemented with the first population, then 85 students are doing fine or better, so it appears to have an 85% success rate. But if the intervention is implemented with the second population, then just 40 students are doing fine or better by the end, so it appears to have a 40% success rate.
But of course I made up the numbers about the population and the effectiveness of the intervention out of my head! We can model the situation in general with a population in which F out of 100 students are fine and we implement an intervention which is I percent effective. In this general situation the success rate will be F+(100-F)*I/100, which we can graph F on the horizontal axis and I on the vertical axis, coloring each point in the plane with the success rate as below, where lighter colors means a higher success rate. You can see that both the base success rate in the population and the effectiveness of the intervention constrain the overall outcome. For very successful populations, nearly any intervention will appear to be successful, but not all interventions that “work” with a naturally successful population will work with a struggling population. And for an intervention that is nearly 100% effective, you can achieve amazing things with nearly any population. The trouble, to my mind, is that there aren’t any interventions that are 90% effective or better. Even the best interventions are going to be unlikely to get much over the 25% mark. Of course you have to keep in mind that this entire scenario is simply a “toy model” — it’s not reasonable to measure effectiveness simply by reporting a percent or lump students into two categories of “struggling” or “fine.”
Educational success is not always easy to achieve. Interventions, even when successful, aren’t going to solve everyone’s problems. When we see amazing educational success, we need to ask ourselves who is successful and why they are successful. Our educational system holds out the idea of advancement for all, but the reality is often that the greatest advances are made by those who were already set up for success. I’m really trying to wrap my head around these things and connect them to other ideas, so I’d love to have your thoughts!