How can teachers judge the efficacy of edtech?

One of the projected trends for 2023 is evidence-driven educational technologies (edtech): the development and use of these tools based on an independent, scientific evaluation of impact. For almost a decade, teachers had to parse through marketing claims versus evidence of actual impact. As the studies summarised in the Department of Education’s 2022 edtech report showed, with poorly designed edtech, well-intentioned teachers might inadvertently promote ineffective practices or even practices with negative impact on children’s learning.

Since the 1960s, evidence-based practice in medicine has had a clear mandate: to ensure that only the most stringently evaluated, up-to-date scientific advice, informs products and services. Applying the same standards to education has been met with a considerable amount of resistance in educational circles — relived in the form of edtech’s evidence debates.

In the US, the top quality criteria — the ESSA efficacy standards — mirror that of medical trials: randomised controlled trials (RCTs). RCTs are the highest form of evidence because the trials randomly assign participants to an intervention and control group. While there are good reasons for seeing evidence in terms of RCTs, there are also a number of problems. The 2019 analysis of 141 educational RCTs commissioned by governmental agencies found that 40 per cent of them produced no effect at all, or effect comparable to natural maturation. Since then, the educational field has made a great progress to collectively identify the reasons for RCTs’ null findings and improve its implementation protocols. 

Nevertheless, the high demand of resources required for RCTs leave little room for financing and executing other types of studies. Edtech creators and schools need to know not only whether a tool or approach works, but also the context in which it might work. Here, observations and usability studies coordinated within edtech testbeds work best. The Swedish EdTest model supports teacher communities to give updates on an edtech’s successes, but also sound the alarm when they experience unintended impacts.

Efficacy, testbeds and other evaluation approaches can be combined in a compromise model: the edtech evidence portfolio. An edtech evidence portfolio includes independent, scientific evaluation of efficacy as well as teachers’ and students’ views.

Edtech is at various levels of funding and possibility to pay for an efficacy or effect evaluation study. However, teachers should not shy away from demanding a variety of evidence of both small and large edtech providers. This includes a proof of compliance with national demands for evidence (in some countries, like USA, these are well-established while in others they are in the process of development) and activities or plans for, engaging with research. Various evidence-as-a-service models make it possible for all edtech providers to connect with research and aspire for the most rigorous form of impact assessment.

Edtech companies can engage with university researchers in various ways. London, as a leading edtech hub, offers various programs: UCL EdTech Labs is an edtech research accelerator programme; Goldstar EdTech Diagnostics provides a certification service and Educate Ventures offers research guidance to EdTech companies.

In addition, kitemarks and awards judged by teachers, combined with internationally recognised seals of approval, such as the ISTE Standards, are natural opportunities to leverage on an edtech provider’s scaling-up journey.

Research-aware edtech providers have a variety of evidence indicators in their evidence portfolio. Different procurement teams are likely to weigh up the individual elements within the portfolio differently — and that is fine. In the end, education is a form of social interaction and diversity is its bloodline.

Everyone agrees that if an edtech solution is implemented in the classroom as intended, based on tested procedures and recommended guidelines, it is more likely to positively impact students’ learning.

But if edtech evidence is to be more than a trend – if it is to deliver on the vision of a long-term commitment to solving educational inequity – then teachers’ voices are crucial to driving that transformation.

That starts by demanding richer evidence from providers, who now exist in a system equipped to provide it and validate it.