Evaluating EdTech: A Strategy for Selecting Digital Tools
March 01, 2017

Bewildering. Confusing. Overwhelming. Those are some of the milder terms Iāve heard educators use to describe the landscape for educational technologies. The good news: If you have a learning requirement in your school district, thereās probably a mobile app that promises to solve that for you. The bad news: There are likely to be hundreds, if not thousands, to choose from. On Appleās App Store alone, more than 80,000 learning apps are available.
Navigating through EdTech options can feel like a tightrope walk for school administrators hoping to balance the desire for innovation with the need to make fiscally sound choices. Fortunately, it does not need to be a binary choice. Over many conversations Iāve had in recent months with digitally savvy educational leaders, Iāve learned that itās possible to do both.
Iād like to share with you what Iāve learned.
Where to Start?
Most educators Iāve spoken with agree that defining your need is a good way to begin.
āLetās say you want kids to learn more math,ā says Steven Hodas, former executive director of the New York City Department of Educationās Office of Innovation and now practitioner in residence at the Center on Reinventing Public Education at the University of Washington. āAn administrator might see the problem as kids doing poorly in math. But a teacher might say that the problem is theyāre unable to give students sufficient individual attention. Both point to kids learning more math, but they define the nature of their objectives differently.ā
Hodas suggests administrators engage people who work with the problem on a daily basis to help refine the need before jumping to possible solutions. That also includes parents, school staff, classroom aides and even the community at large. Once the need is better defined, it becomes easier to narrow down the universe of digital tools to the ones that address your need.
Next, Barton Dassinger, principal of the CesĆ”r Chavez Multicultural Academic Center in Chicago, likes to proceed with a āresearch questionā ā turning the need into a testable hypothesis. This forces people to think of ways to quantify their objectives. āAn example of a research question is: āDo students using product X show higher gains in Y than similar students not using the product?āā where Y is the metric used to gauge progress, whether thatās behavioral referrals, attendance, college admissions or test scores, Dassinger explains.
āBefore you make a purchase, you can work with the company to provide the data that will help you answer your research question,ā adds Dassinger, who recommends pressing vendors for weekly data reports. āThat way, you can evaluate their claims on the metrics that matter to you.ā
Improving Discovery
With tens of thousands of choices, finding the right tool can seem like searching for a needle in a haystack. Many educators have devised their own systems for filtering in the most relevant tools while remaining open to changing possibilities.
The Omaha Public Schools in Nebraska developed an app approval process that allows any educator in the school district to submit recommendations for digital products throughout the year. Since the process launched last fall, Omahaās teachers have submitted more than 200 apps. Most receive a decision within five to seven days. If the app is rejected, the teacher receives feedback on why the product didnāt pass muster.
āBefore this system, it was a manual process that would take monthsā to process a recommendation, says Rob Dickson, Omahaās executive director for information management services.
In Chicago, Dassinger prefers to interact directly with teachers within the classroom environment and have informal conversations to surface new products, keeping a list of prospects on a spreadsheet.
Shared Characteristics

These examples suggest thereās room for a wide range of approaches. But the most effective systems tend to have several characteristics in common ā they are ongoing, transparent and adaptable.
- Ongoing. Technological advancements occur nonstop, and teachersā needs donāt always coincide with adoption cycles. The best systems continually evaluate whatās new in the market.
- Transparent. Danny Wagner, manager of Common Sense Educationās STEM Education Content and a former science teacher in Boone County, Ky., believes teachers need feedback so they can better understand why their submissions did or didnāt make the grade. This is critical. Feedback shows teachers their voices are valued, and it will help them stay involved, providing a fresh flow of ideas.
- Adaptable. Dickson calls his system in Omaha āa workflow for changeā because he can update it with new curriculum requirements and learning standards. āIf we create a sound system for evaluating apps, it shouldnāt matter what new technologies are thrown at us,ā he says. āWe can still do whatās best for our kids.ā
If your districtās discovery process involves issuing a request for proposals, Hodas recommends focusing on the learning goal, not the solution. āIf your RFP contains a list of specifications, people try to check the boxes and tell you what you want to hear. Thatās not likely to be productive,ā he advises. āYou need to reach out in a way that gives people the freedom to give you creative solutions.ā
Filtering Options
Letās assume your discovery phase leads to a flood of options. An efficient way to narrow the field, Hodas says, is to ask teachers to watch a five-minute demo of each product (most tech products have video demos posted on their websites). After viewing, ask the teachers to answer this question: Would you use it in your classroom? If the answer is no (or mostly no), toss it out of consideration.
Once you have a manageable list of candidates, you can put them through a more rigorous evaluation. In Omaha, reviewers use a wide range of criteria, from compliance with the federal Childrenās Online Privacy Protection Act and best practices to pedagogy and alignment to P21ās Framework for 21st Century Learning.
In the Charlotte-Mecklenburg School District in North Carolina, Valerie Truesdale, the chief technology officer, says she first determines whether the products are interactive and adaptive. āStudents have to be interacting with the software and not just be given digitized worksheets,ā she says. āAnd the software has to adjust its pace to individual progress so every student can have a personalized learning experience.ā Products then are evaluated using North Carolinaās Standard Course of Study.
At Common Sense Education, where weāve reviewed more than 2,500 educational technologies, our rubric includes engagement (how compelling the product is), pedagogy (how well it supports deep learning) and support (how accessible the experience is to students and teachers).
Recently, we added a fourth pillar: privacy. Over the past two years, weāve worked with more than 100 school districts to create two free resources for schools: (1) a program that evaluates educational software products on their safety, security and privacy policies, along with how well they comply with government regulations; and (2) an open source toolkit for districts that want to do their own evaluations. By the end of the school year, our director of privacy review, Bill Fitzgerald, plans to have completed evaluations for 300 educational technology products on our site .
Start Modestly
Finally, once youāve selected a technology to try out, start small and iterate.
āThink of the first implementation as a pilot,ā Hodas advises. āDonāt think of pilots as free software to try out. Think of the vendor as someone who is collaborating with you to create a product that will result in the changes you both want to see.ā
This can be a cultural leap for school districts used to being given fully developed materials from established publishers. With technology companies, particularly the younger and more innovative startups, products are in constant evolution, and their developers are continually adding features, fixing bugs and improving the overall experience. While it can be frustrating to deal with a moving target, schools can take advantage of that flux by jumping in to influence which features get added and how the product is improved.
At the end of the day, I am reminded of some sage advice for maintaining perspective, provided by Truesdale, a veteran school leader in the Charlotte-Mecklenburg Schools. āThe key is to remember that these are all digital tools,ā she says. āJust like building a house, you have many tools in your toolbox. What tools we use depends on the job we want to accomplish. But thereās no substitute for quality teaching.ā
Author
Additional Resources
Common Sense Education, a nonprofit organization, provides educators with free, research-based classroom tools to help schools harness technology for learning. The website includes a primer for evaluating educational software with particular attention to child privacy, teaching strategies and practical ideas on digital citizenship.
Also, Common Sense Education suggests these information sites:
- ConnectEd,
- Digital Promise,
- EdSearch/Lumos Learning,
- EdSurge,
- Federal Trade Commissionās explanation of the Childrenās Online Privacy Protection Act for educators,
- LEAP Innovations,
- Learning Innovation Hub,
- P21ās Framework for 21st Century Learning,
- Technology for Education Consortium,
- U.S. Department of Education, Office of Educational Technology,
Questions To Cut Through the Marketing Spiel
Whether youāre putting out a formal request for proposal or you just want a conversation with the vendor, here are some questions to help educators move past the marketing script and on to how the proprietorās products can improve student learning. Iāve collected some of these from Danny Wagner, head of Common Sense Educationās STEM Education Content, to spark your thinking.
Customization: Can the company customize the product to fit the needs of a wide range of schools and populations in your district? Can teachers create custom content? How well does the tool integrate with other tools teachers are using in your classrooms?
Connectedness: How does the technology help connect stakeholders, including parents, students, teachers, administrators and community members? The stronger these connections, the more students will take ownership of their learning, teachers will become engaged facilitators, parents will become decision makers and empower learning at home, and businesses will give their time, sponsorship and donations to educational initiatives.
Equity: How can a variety of student populations access the product? Is it available in multiple languages? Can resources be printed or read out loud? Are there options for library access or home access?
Technology Requirements: What are the platform and systems requirements? Does it work with existing school devices? Does it require the school to upgrade its infrastructure? What support does the company offer schools to help integrate the product? Can the product or service scale to support thousands of students?
These will surely prompt a dozen other questions for school district leaders to ask. If so, you are well on your way to being an expert educational technology evaluator.
Advertisement
Advertisement
Advertisement
Advertisement