Q&A
Survey methodology and potential biases in Summer Rising program evaluation
1:38:23
·
136 sec
Council Member Stevens raises concerns about the survey methodology used to evaluate the Summer Rising program, questioning potential biases and the types of questions asked. DOE officials respond to these concerns and explain their approach to gathering feedback.
- Council Member Stevens expresses concern about leading questions and the potential for biased results
- DOE officials assert that they use legitimate survey methods and statistical analyses
- The surveys include both targeted questions about specific aspects of the program and open-ended questions
- DOE uses the survey data for continuous improvement of the program
- The discussion highlights the importance of gathering diverse perspectives and avoiding potential biases in program evaluation
Althea V. Stevens
1:38:23
Is it still more yes or no questions?
1:38:26
I remember that was a big because I actually have not been able to look at the survey this year.
1:38:31
But is it still more yes or no questions?
1:38:32
Or is there room did you guys allow more room for anecdotes and and more expression?
1:38:36
Because I think sometimes, like, if you have a leading questions, You won't get the answer as you want because if you're good, if you're asking the right questions, you're going to lead them down the road.
1:38:47
And so I'm just So, of course, your survey is gonna say it's overwhelmingly positive, right, where I'm looking at the survey that was done by UNH that doesn't look as plastic.
1:38:57
It doesn't look negative.
1:38:58
Right?
1:38:58
And I'm not saying that, but the if there is some there are some discrepancies because I don't want us to think just because people are some people don't have options.
1:39:08
And so if you don't have an option, what you're gonna say is trash, you're not gonna think when you your child and a crash program.
1:39:14
So
Emma Vadehra
1:39:15
I will I will say so so 2 quick things that we can talk about the specifics of this survey, did not mean to imply.
1:39:21
There's not plenty of room to improve.
1:39:22
Just didn't
Althea V. Stevens
1:39:23
mean to I thought we had to be betting.
1:39:25
Have a You guys have actually been listening to the feedback.
1:39:28
Like, when we bring it back to you, you guys are coming back with answers.
1:39:31
I'm not that's not what I'm applying at all.
1:39:34
So we
Emma Vadehra
1:39:35
do see improvements year over year, I would say.
1:39:37
We do we have a survey person back there.
1:39:39
We do work pretty hard to make sure we use legitimate surveys, statistical analyses, and all those sorts of things across this.
1:39:45
But point taken, there are different versions.
1:39:48
And and yes, as a family, if you send your kids somewhere, you want to feel good about where you're sending them.
1:39:53
We do ask specific questions about this part of the experience about transportation, about academic portioning.
1:40:00
We ask both students and families and staff and providers those questions, so it's not just how do you feel.
1:40:06
Right?
1:40:06
We do try and get more targeted.
Althea V. Stevens
1:40:08
Even with the providers, you know, always go back and forth with the providers.
1:40:11
Action providers, you're the funder and action and provider how you fill out the program.
1:40:14
They can't be that's on there, so I keep telling you that.
1:40:16
We we gotta find a different one because that don't work for me.
Emma Vadehra
1:40:19
They're not.
Althea V. Stevens
1:40:20
Is it anonymous?
1:40:20
Exactly.
1:40:21
It's still they're still they're still they're funded.
1:40:24
Exactly.
Emma Vadehra
1:40:25
Susan, did you hear that?
Althea V. Stevens
1:40:26
They asked you.
1:40:27
She's heard this before.
Emma Vadehra
1:40:30
But I was just gonna say we do use that a lot for continuous improvement because it is more targeted versus questions like this is where we wanna make improvements.
1:40:37
This is where we want more field trips, all that kind of stuff.
Althea V. Stevens
1:40:39
Is this information public, and especially the academic piece around the the evaluations and surveys and things like that?
