Your guide to NYC's public proceedings.
Q&A
Development process and upgrades of MyCity chatbot
0:34:44
·
3 min
CTO Matthew Fraser explains the development process of the MyCity chatbot and recent upgrades to improve its performance. The discussion covers the transition from GPT 3.5 to GPT 4.0 and measures taken to address inaccuracies.
- The chatbot was developed using a combination of internal and external resources.
- An upgrade from GPT 3.5 to GPT 4.0 was implemented to improve the chatbot's capabilities and accuracy.
- Safeguards and controls were put in place to manage the chatbot's responses and protect constituent data.
Matthew Fraser
0:34:44
Yeah.
0:34:44
So both the MyCity and the chatbot tool were built with a amalgamation of internal and external resources.
0:34:50
Mhmm.
0:34:51
The actual concept of the my city chatbot was led and overseen by a number of people within OTI and actually helped built by the folks within OTI as well.
Jennifer Gutiérrez
0:35:01
And so I think it was a little bit more than just, like, a little little bump in the road because I I think still as of last week, there were there were some instances of folks using the chatbot for basic small business information, had to get started, where to find a permit, and the information was not necessarily accurate.
0:35:20
So how are you all looking to kind of assess that?
0:35:25
Like, how how are you improving it?
0:35:26
What does that what does that look like?
0:35:28
How can you improve this chatbot feature?
0:35:30
If still through last week, was immacuating for Yep.
Matthew Fraser
0:35:32
So the the initial version of the chatbot that was released was using a legacy version of GPT, GPT 3.5, and then we subsequently upgraded to GPT 4.0.
0:35:44
Right?
0:35:45
Now what that means is the engines that make the the determinations in terms of what information it pushes forward.
0:35:51
As you go up in versions, the capability significantly increases, and it gives us the ability it gives the algorithm the ability to make better determinations on what it puts out.
0:36:01
And the cases where we got public sentiment around hallucinations and things along landlines, even in the reported cases, we ran an assessment of every time the chatbot had been asked any one of those questions.
0:36:14
And outside of the cases that were reported, there were very few and few and far in between.
0:36:19
I think for us, one of the things that we consistently do, and it's part of the development process, especially when you you leveraging emerging technologies, as you continuously assess, collect feedback, and and refine.
0:36:31
And when we were looking at something like a a chat instance that gives information back in real time and a closed instance, one of the things that we did as part of the New York City development process, we made a determination that anything that we built, we had to ensure that we protected our constituent data first.
0:36:48
So a lot of the people that leverage public models, that model is continuously refined by public feedback.
0:36:53
Someone's using it or do you have multiple customers using the same model they can update in real time.
0:36:58
For us, we wanted to ensure that as we built that those models were only updated with with content that we wanted it to learn.
0:37:07
And we have a team that's dedicated ensuring that we continuously refine that to make it better.
0:37:11
I think after the upgrade, I should say, I think.
0:37:13
I know after the upgrade, when we went from GPT 3 to GPT 34, we updated some of the safeguards in place.
0:37:19
We've now put a lot of control in place.
0:37:22
When someone searches something, which is outside of the chatbots capability, we've been very clear proactively serving up saying this is outside of the use case.
0:37:30
Please refer to the disclaimer.
Jennifer Gutiérrez
0:37:33
Okay.
0:37:33
And can you just remind us when when when was the upgrade?
0:37:37
If said it?
0:37:38
I'm sorry.
Matthew Fraser
0:37:38
From there?
0:37:39
So we upgraded the the back end engine that that's used to provide information out.
0:37:45
The there was chat EBT version 3.5.
0:37:49
We upgraded from 3.5 to version 4.0.
Jennifer Gutiérrez
0:37:52
But when was that?
0:37:53
I'm sorry.
Matthew Fraser
0:37:54
Oh, I defer to our associate commissioner Amit Singh.
Amrit Singh
0:37:58
I also defer to somebody.
Jennifer Gutiérrez
0:38:00
This okay.
0:38:00
Because it was launched in was it announced in where do I have it?
0:38:05
I don't know.
0:38:05
October, September of last year?
Matthew Fraser
0:38:08
Yeah.
0:38:08
Yeah.
Jennifer Gutiérrez
0:38:08
Okay.
0:38:10
Okay.
0:38:10
So just this summer, it was it was updated.
Matthew Fraser
0:38:12
Yep.
0:38:12
That's correct.