Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###

00:00:01.62  >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.45  >> Hey. It's Amy, and I am so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.60  >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.85  >> Hello. On this episode of "A Date with Data," I am joined by the directors of special education from two neighboring states. We have Leah Voorhies, who is the Assistant Superintendent of Student Support with the Utah State Board of Education, and Alissa Trollinger, who is the Deputy Associate Superintendent for Exceptional Student Services with the Arizona Department of Education. And they will be reflecting on their states' IDEA data quality and really looking forward to hearing what each of you are focusing on and excited about and maybe if there are similarities between kind of what you're doing but also really curious about your unique journeys as well. So welcome to both of you, and ...

00:01:07.32  >> Thank you for having us.

00:01:08.42  >> Thank you.

00:01:08.67  >> Yes. So can you each tell us a little bit about yourselves? And, Alissa, if you want to start off.

00:01:14.99  >> Sure. Thank you. I am in my eighth year as State Director of Special Education in Arizona. But this is my 18th year with the state in our Special Education Department. And it's been quite an adventure in my role, but I feel like we are really privileged to do a lot of important work.

00:01:42.05  >> Thanks. Looking forward to hearing more about your adventure, and Leah?

00:01:47.11  >> I've been the State Director for this going on 7 years. Prior to that, I was a coordinator, which is like the assistant director ...

00:01:58.46  >> Mm-hmm.

00:01:58.73  >> ... for another 6. So it's been 13 years. We have a very robust team in Utah, and we have several important priorities, and data play a huge role in each of those priorities, and we're working on improving meaningful, making inclusion more meaningful for students with disabilities and improving postsecondary outcomes and improving the number and the ... So the preparation and the recruitment and retention ...

00:02:35.83  >> Mm-hmm.

00:02:36.25  >> ... of both paraprofessionals and professionals in special education, and all of those initiatives require an enormous amount of data analysis.

00:02:46.51  >> Yes, I would imagine so. Okay. Well, that was a little taste of some of your priorities. Leah, do you want to talk a little bit more in detail about some of your IDEA data quality priorities in Utah?

00:03:01.44  >> Sure, so for the past decade, we have been having what we call data literacy deep-dive conversations with our LEAs about all of the IDEA indicator, SPP/APR indicator data. So we spend a week traveling around the state and meeting with leadership teams from all the districts and charter schools in Utah, which right now is about 160-ish, looking at all of their indicators and asking them questions about their data, about the quality and about what it actually means and what they might want to do about it. And then we generally help them create program improvement plans. And as they've gotten better and better at that, they've asked more in-depth questions and the past 2 years, our special education advisory panel has also asked more in-depth questions because we reviewed the APR/SPP data with them every year as well.

00:04:12.21  >> Mm-hmm.

00:04:12.52  >> And they've asked specifically about the relationship between the amount of time students with disabilities spend in general education and their proficiency and their graduation and their drop-out rates and the post-school outcomes. So one of our major data initiatives is, we have an external statistician who is doing regression analyses for us, just dumping in all of the data that we have about outcomes and about least-restrictive environment and doing it by both LEA and by school. And then we are working with our school improvement team, which I also supervise, the Federal School Improvement, but we also have a State School Improvement team to compare their data with special education data, and we're working with schools across the state to specifically look at school improvement efforts for schools that are identified for targeted support and improvement for students with disabilities, so that's a major initiative that we have going ...

00:05:14.77  >> Uh-huh.

00:05:15.32  >> ... that has come out of, as a result of LEAs and the Special Education Advisory Panel asking questions about our SPP/APR data.

00:05:25.55  >> That is very cool. And have you gotten results yet from the statistician and started to use those data and analyses?

00:05:34.81  >> We have. So I've been meeting with the principals and special ed directors across our state who have both better-than-expected results, not average, but expected results ...

00:05:51.03  >> Mm-hmm.

00:05:51.44  >> ... and not-quite-as-good-as-expected results. I haven't been meeting with principals about that. I've been meeting with special ed directors. But I've been meeting with principals and leadership teams across our state, talking about their, having them tell me their story. What is it that you are doing that you believe is contributing to your better-than-expected results? And it's so fun, and then we're reporting those and aggregating the information about those stories, and we're reporting them to all of the special ed directors in the state and anyone else who will listen.

00:06:31.75  >> So, Alissa, what about you? What's happening in Arizona in terms of your data priorities?

00:06:38.34  >> Well, I think that Leah and Utah are a lot further along than we are. But I'm pretty proud of the progress that Arizona has made to structure our data systems and begin to build our data literacy culture. I'll kind of give a beginning so that it's more understandable about why we're in this space and why we were kind of on an adventure. In the 18 years that I've been here, before I was the State Director, we had 12 State Directors ... excuse me, five State Directors in a 12-year period, and each of those State Directors had varying degrees of understanding and experience and comfort with data. And I feel like when I started in this position, and my leadership team started, we were really beginning from scratch to build our entire, really our entire system of general supervision, putting it back together. Some pieces had existed previously and dismantled, and some pieces didn't exist at all. And so we've spent really the first part of my time in this position putting systems back together, making sure that we have programmatic monitoring system that functions correctly, making sure that we are able to collect data accurately and with validity. And that's kind of a challenge in Arizona because we have almost 700 LEAs.

00:08:10.11  >> Wow.

00:08:10.39  >> And about two-thirds of them are in charter schools ...

00:08:13.16  >> Oh, my god.

00:08:13.49  >> ... charter schools. Mm-hmm. Charter schools in Arizona stand as their own LEAs, and so they're each responsible for providing special education and related services but also submitting their own data.

00:08:26.78  >> Mm-hmm.

00:08:27.20  >> And so if you think about our data-collection system, not only did we need to have a structure that was built to collect all of that data from all of those differing LEAs, but we also needed to put a system or put together a system where we could continually train on how to do this because we have a high turnover ...

00:08:49.85  >> Mm-hmm.

00:08:50.16  >> ... in data staff at LEAs, particularly in charter schools. So that's really what we've invested a lot of time in. We've also worked really heavily in data governance in our state to make sure that the decisions that are made at the leadership level within the agency reflect accurate practices and that other parts of the agency understand special education data reporting requirements. I don't know, Leah, if you experienced this, but we often, I'm going to say with regularity, have to explain to the rest of the agency that there aren't any do-overs in the IDEA. Our data has to be collected and submitted timely and accurately and it, to do other than that incurs a penalty. And that's been something that the rest of our agency has really had to kind of get on board with. We need to all be on the same page. Our data terms need to be consistent. Our timelines need to be consistent. We all need to be working together to help support IDEA data collection. I think that's what everybody's goal is, but it puts an extra level of urgency in when we don't get any do-overs.

00:10:00.72  >> Yeah, especially now ...

00:10:01.79  >> So ...

00:10:01.91  >> ... with EDPass and modernization ...

00:10:04.20  >> Exactly.

00:10:05.10  >> ... really, yeah. The first time, it's got to be perfect, pretty much.

00:10:09.75  >> I think that Alissa is being very humble because I've been the state director the whole time that she has, so we're good friends.

00:10:17.94  >> Mm-hmm.

00:10:19.20  >> And the amount of systems work that Alissa has done, because we talk at least monthly. The amount of systems work that Alissa has done, it is really amazing, especially when you really think about 700 ...

00:10:37.66  >> Right.

00:10:38.04  >> ... LEAs.

00:10:39.62  >> Yeah, and the majority of them are single ...

00:10:43.11  >> Single buildings ...

00:10:44.28  >> ... schools.

00:10:44.62  >> ... absolutely.

00:10:45.25  >> Yes ...

00:10:45.48  >> Mm-hmm.

00:10:45.90  >> ... small LEAs, yeah.

00:10:47.51  >> Yeah.

00:10:48.79  >> And I completely agree with Alissa. I have been super frustrated with some of the processes in our agency to submit data, EDFacts data.

00:11:03.65  >> Mm-hmm.

00:11:04.04  >> And the lack of understanding, and sometimes it seems like the lack of caring, but it isn't. It's a lack of understanding ...

00:11:12.57  >> Yes.

00:11:13.29  >> ... about the high stakes ...

00:11:15.15  >> Mm-hmm.

00:11:15.46  >> ... of IDEA data. Two years in a row now, I've had to sit the Deputy Superintendent down with the Director of Data and Statistics and say, "Look. Your staff did not do their jobs in an accurate and timely manner, and this is the consequence for my staff and for the state." I don't think it had ever registered that IDEA data can, is really high-stakes, and the difference between getting a determination of meets requirements and a determination of needs assistance is really upsetting to people.

00:11:54.96  >> Yep.

00:11:55.40  >> Yeah.

00:11:56.62  >> And my internal team didn't get that, and I even invited them. I sat down, made them come to a discussion with OSEP data people so that they could hear it from someone besides me. And Alissa and I have talked about this a lot, so I know how many times she has done that, as well.

00:12:15.53  >> Absolutely. That has been really helpful, to have OSEP's support in articulating the rigid data reporting requirements to help the rest of our agency understand. And it's funny when I reflect on it. Years ago, when I supported data submission, we were having the very same conversations that we have now ...

00:12:35.32  >> Mm-hmm.

00:12:35.65  >> ... in terms of ensuring that all of the agencies onboard that were working on accuracy, but it's not only training our external stakeholders but ensuring our internal stakeholders are on the same page.

00:12:51.12  >> Agreed.

00:12:51.60  >> So in terms of our priorities, our priority, I think, at this point in Arizona is to make sure that we are not only increasing our data literacy but using all of the data that we believe is accurate and valid at this point to make good decisions. And so really we've been engaged in restructuring our priorities based on data, putting together data or, actually, we have some data dashboards, but enhancing those data dashboards and getting information out to LEAs so that they can make timely and meaningful decisions with that data and just really focusing our efforts on improving outcomes. It was really hard to get a grip on that when we didn't have confidence in our data.

00:13:40.15  >> Yeah.

00:13:40.27  >> But now that we do, and we have system that are working well, that's our challenge is, to make sure that when our special education advisory panel or our parent stakeholders ask questions, that we are able to dig into the root cause and get some answers.

00:13:58.31  >> Wow, great. Well, yeah, it sounds like for both of you, feeling at this point pretty good about the quality of your data, and that goal is always to get there and then be able to use it for improvement and decision-making, and it sounds like you're well on your way. So that is very exciting. And also, you kind of touched already a bit on building that data culture within your state, whether that's within your special education department but also really beyond and making sure that everyone throughout the SAA really understands the importance and what this all means and why having high-quality data is so critical. Leah, do you want to say anything else about kind of how you've been building that data culture in your state?

00:14:44.32  >> I think it has made a huge difference for us to take the time to sit with LEA special education leadership teams and talk through, and we provide thought-provoking questions that they can ask, that we can ask them, that they can ask us about their data and talk through, like Alissa was saying, their system for collecting and for verifying and for submitting their data because so many LEAs have such different systems and so not necessarily just hearing from us. We'll get them the information that they need to feel confident in their processes because they need it to be very LEA-specific.

00:15:36.36  >> Mm-hmm.

00:15:37.02  >> So going to sit with them or doing it online, doing it virtually. And we make it a point to let them know that we are willing to come to them. We don't make them come to us. They can meet with us virtually, or I'll send a team. And hopefully that helps them understand how important it is. And I do think it's made the difference. We have some LEAs that are getting pretty savvy about their data now and can speak very confidently about their data. And like Alissa, we have over 120 charter schools, so we have more than two-thirds of our LEAs that are charter schools, and some have multiple buildings, but most of them are one-building charter schools. And there's an enormous amount of turnover. So even though we have LEA leadership teams and special ed directors that are really savvy, we also constantly have new leadership teams and new directors that have no idea where the data comes from. They don't even understand that they give it to us before we analyze it.

00:16:49.49  >> Yeah.

00:16:49.58  >> Right.

00:16:52.76  >> So we have really had to think critically and intentionally about having a tiered system of supports for LEAs because we have really savvy directors, and we have brand-new, 22-year-old, right-out-of- bachelor's-degree-program directors.

00:17:14.74  >> But it sounds like you have a robust system in place to really support them and realize that challenge of the turnover but being prepared and having those resources and technical support in place, I'm sure, is really making a big difference.

00:17:31.24  >> Yes, and the system constantly has to be tweaked because every year, we'll learn that there's a gap or that there's a great new question that we want to address. And then the processes change, too. Systems change. LEAs' student information systems change. And our system changes, and OSEP changes the calculations on us.

00:18:01.14  >> Mm-hmm.

00:18:01.35  >> Yeah.

00:18:01.69  >> Yep.

00:18:02.26  >> So it's a good thing we believe in continuous improvement because it's not an option.

00:18:08.34  >> Right. You don't have a choice.

00:18:09.98  >> It's definitely never static, and I would agree with Leah, I think, as I think about that question. In Arizona, we have really solidified our data reporting internal structures so that now that we have built accurate systems internally, we can continue to maintain them, and that has meant that we expanded our internal staff to support data collection, to centralize data collection efforts, at least to push out data collection and building strategies out to different parts of exceptional student services in the agency. And now we centralize it with a data management team, a data operation team, and we think that, that will help us maintain our structures moving forward.

00:18:57.66  >> Mm-hmm.

00:18:58.20  >> It will also be working with the IDC to document all of our data-collection procedures, so we're really excited to get moving on that work and put that documentation in place so that it will outlive us and our positions. I think we have established a data culture for a long time primarily through our programmatic monitoring system, and that is not to say that all the data comes from monitoring. But we have used our specialists who work within the programmatic monitoring system to be kind of our data ambassadors. We use a risk analysis tool that is really just a giant data dashboard that contains not only indicator data from the SPP/APR but also pieces of data from all of the data that we collect from LEAs. And every year, we are updating that information and presenting it back to LEAs using specialists who case manage those schools in our program monitoring system. So our programmatic monitoring specialists are kind of jacks of all trades. They understand what each of the data pieces are. They report that back out, and they talk about data quality, data trends. We're really trying to keep data literacy from superficial to deep understanding on the tip of LEA special education leaders' tongues, and that doesn't necessarily get into the deep root-cause analysis or trending, but it does at least help them be refreshed about what the data that they're submitting means. As Leah said, a lot of times, they don't even realize that they are submitting the data, so this helps to keep them fresh. And then it also means that when we do want to work with them on more meaningful root-cause analysis work that it's not novel. They're not surprised. So I think that, that has been a real key to our data culture and working with our LEAs, and we'll keep expanding on that as we continue to hone in our data priorities.

00:21:06.91  >> So that data that you mentioned, it sounds like a really robust collection that you, yeah, could do a lot more with even as districts get further along and want to do more of that root-cause analysis and go further, so ...

00:21:21.93  >> Sure.

00:21:22.57  >> You've really set yourself ...

00:21:23.86  >> I would say just as a kind of step feature, the risk analysis tool that we use is really designed initially to help make decisions about the type of monitoring that was going to be conducted with an LEA. Really, how at-risk is their system of special education?

00:21:41.93  >> Mm-hmm.

00:21:42.27  >> Are they going to be able to implement special education in the way in which the IDEA intends so that students receive the [Indistinct]? But we have built on that, and so that was kind of a step to get us to a broader data culture.

00:21:57.52  >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at IDEAdata.org.