In the Weeds: How To&Through approached the responsible inclusion of EL and IEP data
In 2022, the To&Through Data and Research team set out to add data on students who are English Learners and students with disabilities to our online tool for the first time. We knew from the start that we wanted to be especially thoughtful about the process. We have access to a lot of raw data, and believe that showing academic outcomes for students from these groups can provide important information to educators about how well, or not well, these students are being served (see our team Equity Stance for more of our thoughts on displaying data for different student groups).
Our team’s Core Values include believing that students, families, and communities are experts in their own experience; taking responsibility for the impact and consequences of our data; and prioritizing the collection and incorporation of feedback. Making careful and intentional choices about the labels we use for our data, and how we display it, is one way that we try to live these values in our work. This means slowing down our process to allow ourselves time to consider all the options, to discuss the implications of making certain choices, and to engage others for feedback.
We know that we are not alone in working through the complexities of how to show sensitive data, and hope that providing some insight into our process will contribute to a larger discussion about how to approach this work.
Data Organization and Labels
The first question we faced was how to organize and label our data, both for students who are English Learners and for students with disabilities.
Usually when data on English Learners is displayed, it’s shown in two categories: students currently learning English, and students not currently learning English. However, a 2019 report from the UChicago Consortium on Schools Research (our sister organization) demonstrated that grouping students this way obscurs the academic achievements of students who were former English Learners but had transitioned out of support services, who in some cases actually outperform students who were never English Learners. We therefore decided to display our data in three categories: current English Learners, former English Learners, and students who were never English Learners.
We also decided to only use the abbreviation “EL” when not referring directly to students; for example, we use “EL services” but not “EL students”. We made this decision to center the students and their abilities, rather than labeling them using an abbreviation that describes just one of their characteristics.
We also considered our data on students with disabilities, and we struggled here too with what language to use. While CPS uses the term “diverse learner”, we worried this was vague and potentially not commonplace outside the Chicago context. Further, disability activists have pushed for use of the word “disability” rather than a euphemism like “special needs”.[1] However, we were concerned that using the term “students with disabilities” would not be entirely accurate because not all students with a disability are diagnosed and given an IEP, and some students with a disability have a 504 plan rather than an IEP.[2] Therefore, we decided to use the term “students with an IEP”, which most accurately describes our data.
Visualization and Framing
Our next set of questions centered around how and what data we should display. From the start, we knew we did not want to group all students with IEPs together, because students’ disabilities vary widely in type and extent, and so their experiences in school and their academic outcomes also vary widely. Disaggregating this data is complicated, though, because the ways that CPS defines and labels different disabilities has changed significantly over time. Another complication is that, in order to protect student privacy, we do not show data on student groups below a certain size on our tools, and so displaying data on every individual disability group would lead to a lot of suppression.
In the end, we compromised by grouping the data into two categories: students with an IEP related to a specific learning disability (the largest disability category in the CPS data), and students with an IEP related to any other disability. Although this unfairly groups students with extremely different disabilities into one catch-all group, we feel it is worth the tradeoff in order to disaggregate the data to some extent, and allow us to at least better understand the experiences of students who have IEPs related to learning disabilities.
Lastly, we struggled with the question of whether to display data on students with IEPs at the individual school level. While this is important and actionable information for schools, we worried about small group sizes (especially at the grade level) and the overall sensitivity of IEP status. Ultimately, we felt this potential risk to student privacy around something as sensitive as IEP status was not worth the tradeoff, since schools have access to similar information on internal CPS data systems, and we decided to show IEP data only at the Network and District levels.
Gathering Feedback
Once we had ideas about grouping, displaying, and labeling data about English Learners and students with IEPs, we engaged others in feedback on those choices.
For advice on English Learners, we reached out to the authors of the Consortium report mentioned earlier, as well as some of their colleagues at the Latino Policy Forum, and the Office of Language and Cultural Education at CPS. To gather feedback on our decisions around students with IEPs, we reached out to the Office of Diverse Learners Supports and Services at CPS. The feedback we received led to some important tweaks to our language. For example, we changed the name of the category from “students with an IEP for another disability” to “students with an IEP for any other disability” to try to make the distinction clearer. Our conversations also pushed us to more clearly define our categories and make those definitions more accessible to our audience. For example, we added a pop-up definition to the graph that shows rates for students with IEPs to clarify that these students have a unique educational plan for accommodations and supports but must meet the same graduation requirements as other students in the district; it also clarifies that students in the “IEP for any other disability” category have a wide range of disabilities, including physical, cognitive, and behavioral disabilities.
The most important question we posed to our external partners was whether we should display comparison data for students in these two groups. Specifically, when showing attainment metrics for current English Learners and former English Learners, should we also show the attainment rate for students who were never English Learners as a comparison point? Similarly, when showing rates for students with IEPs, should we also show the rate for students without IEPs? This was a difficult decision and we weighed many pros and cons, ultimately deciding to treat each case differently.
The practitioners and experts we spoke to about English Learners felt that providing a comparison between students who were never English Learners and students who were former English Learners was an important way to shine a light on the achievements of this group. In many cases, students who were former English Learners actually outperformed students who were never English Learners, which felt like an important opportunity to both highlight the abilities of these students and the success of EL supports.
When it came to students with IEPs, however, our stakeholders were torn; on the one hand, showing the comparison between students with IEPs and students without IEPs could reveal gaps in outcomes that point to a lack of support that needs to be addressed. On the other hand, it could perpetuate the harmful narrative that students with IEPs can’t achieve at the same level as students without IEPs. Further, because students can transition into and out of having IEPs, or transition from an IEP to a 504 plan, the data is not clear-cut. We ultimately decided that, in this case, a more appropriate comparison was to show data on students with IEPs related to a learning disability next to data on students with IEPs related to any other disability, but not to show data on students without IEPs. While this may imply an unfair comparison between students with an IEP for a learning disability and students with an IEP for any other disability, it seemed important to have some comparison point, and this felt more appropriate than comparing to students without IEPs.
Our aim to display and talk about data in an equitable and responsible way is an ongoing process. We don’t consider this work done, and continue to grapple with decisions around the language and groupings we use with our data. The tradeoffs we chose felt worthwhile at this time, but that may change in the future. We also hope to continue receiving feedback from a wide range of sources as this data goes live on our online tool. If you would like to share thoughts or feedback, please feel free to email us at toandthrough@uchicago.edu. We are learning more every day, most importantly from those who are represented in and those who use our data.
Alexandra Usher is the Associate Director of Data & Research at the To&Through Project.
[1] For example, see the National Center on Disability and Journalism’s Disability Style Guide at https://ncdj.org/wp-content/uploads/2012/08/NCDJStyleGuide2015.pdf
[2] An IEP, or Individualized Education Plan, is a legal document that outlines the services and supports that a student with a disability is entitled to receive. For more information see: https://www.isbe.net/Pages/Special-Education-Individualized-Education-Program.aspx
