QUESTION
Why are nearly 50% of datasets not automated in the NYC open data portal?
0:44:10
·
140 sec
The NYC Office of Technology and Innovation (OTI) addresses automation delays in datasets, attributing them to practical resource limitations and the frequency of dataset updates.
- Datasets flagged for potential automation undergo an assessment of practicality and update frequency, deterring automation of infrequently updated datasets.
- Certain datasets, like those updated with census data, are not automated due to the infrequency of updates and the high effort required.
- OTI aims for more automated datasets to enhance user experience but faces challenges due to limited engineering resources and the complexity of automating data processes.
- Work is ongoing to automate as many datasets as possible with the current resources, aiming for constant improvement in data automation.
Robert Holden
0:44:10
I just I wanna bring up automation of datasets, and I think it's lagging at this point.
0:44:19
I think it's been 874 datasets currently the New York City open data portal have been flagged for automation.
0:44:27
Yet, however, 425 of them, nearly 50% have not been automated.
0:44:33
Can you tell us what to hold up?
0:44:35
Or
Zachary Feder
0:44:37
So the there there's a flag that every agency when they submit a dataset to us, and they indicate whether that dataset could potentially be automated.
0:44:49
It it's important that there's within that potential, there's some assessment between their team and ours.
0:44:56
As to what actually makes sense.
0:44:57
So we have some datasets, for example, that are updated once every couple of years, and they absolutely can be automated if we wanted them to.
0:45:06
But given the amount of work it would take to make that automation and versus the work to send that file for once every couple of years, that would be something that we would would not prioritize.
0:45:18
There are certainly other datasets that can potentially be
Robert Holden
0:45:23
automated.
0:45:24
Example or 1?
Zachary Feder
0:45:26
A lot of the data, let's say, that that city planning is responsible for that's updated, like, when the census comes out.
0:45:33
We we we could have automations for that, but it is so infrequent that the again, the amount of work that would go into it, we prioritize things exactly where the data actually coming in frequently every day multiple times a day.
Martha Norrick
0:45:46
Our ideal is that as many data sets that can be automated, you know, will be automated.
0:45:50
And, you know, as Zachary said, we're prioritizing and we have limited resources to build these automations, there's it's not sort of, like, you know, always push button move data.
0:46:01
It requires, you know, some some engineering work, and so we are and but we know that that's the best user experience for people who are interacting with with open datas for those automations to just be there and and happen in that data be refreshed without sort of the drama of, like, who's on vacation.
0:46:18
So we're we are it's it is our it is our goal to get there, and we are working as quickly as we can with the resources that we have, and it's something we hope to, you know, to be constantly improving on.
Robert Holden
0:46:28
Great.
0:46:29
Alright.
0:46:29
Thank you, Cher.
