QUESTION
How did the DOE communicate with IBM about technical issues and did IBM meet its contractual obligations?
0:25:29
·
4 min
The DOE communicated a technical issue to IBM at 1 PM the day before it became critical, using email and Slack, and IBM is asked to file a proactive service ticket in the future. IBM’s performance on the day of the snowstorm is reviewed, with specifics on service levels and adjustments made.
- The Department of Education (DOE) notifies IBM of technical issues via email and Slack, with a recent emphasis on proactive service tickets.
- IBM is informed of an impending issue at 1 PM the day before the snowstorm, and DOE's communication methods are outlined for future incidents.
- The citywide contract with IBM Services is managed by the Office of Technology and Innovation, allowing city agencies to purchase from IBM.
- IBM provided above the contracted service level of 400 transactions per second, increasing it to 1400 and then 2000 transactions per second during the snowstorm.
- Despite the increased capacity, some students and staff experienced failures in logging in, prompting IBM to attempt raising the capacity to 3000 transactions per second.
Farah Louis
0:25:29
The first question is in regards to everything is based off the statement, but I wanted to know what is DOE's process to notify vendors of critical technology support needs, meaning Who do you call or email?
0:25:45
And what's the process?
0:25:46
And what kind of acknowledgment is required for that?
0:25:50
The second one is in regard in regards to your statement on page 5 when you reached out to IBM.
0:25:58
I wanted to know when did DOE actually communicate to IBM that there was a technical problem and what means of communication was used and what was said and communicated.
0:26:09
With IBM, how often does DOE have an executive level sync with IBM's executive team?
0:26:16
And what expectations are expressed at those meetings that you have with IBM.
0:26:21
And did IBM meet its minimal contractual obligation in terms of service on the morning of the snowstorm.
0:26:29
Meaning, did IBM perform or was their performance met with the contract requirements?
0:26:36
And if so, why did the contract not have a higher level of performance?
0:26:41
And last one, which agency owns the contract that IBM operates currently for the DOE?
0:26:48
Thank you.
Scott Strickland
0:26:51
I didn't answer as many as I can.
Farah Louis
0:26:53
I know that was a lot, but just to provide some context based off your statement, which I thought was fine.
Scott Strickland
0:26:57
Great.
0:26:58
Thank you.
0:27:00
So we have regular meetings with our major vendors.
0:27:04
We meet with IBM weekly, I think.
0:27:06
And they were aware that this was coming.
0:27:10
The exact communication went to them at 1 PM the day before.
0:27:15
We sent them an email customer success representative who we deal with on these kind of issues, acknowledge that.
0:27:22
I will say that IBM has asked us to also in the future file a service ticket, which then gets into their help desk scenario and gets the right support people lined up.
0:27:32
We've actually done that once or twice since We had threatening weather a couple of weeks ago, and what they've asked for is if we see something maybe gonna happen on a Thursday, put put up what they call proactive ticket in on a Monday, and let us know and then waive it down if it's not going to be a remote day, and we actually have done that.
0:27:51
We use Slack to communicate on the morning of They immediately looked at their systems and they were already monitoring them and realized that a problem did exist.
0:27:59
They were seeing load, all the allocated capacity was being used by our students and staff trying to log in, and they were seeing failures of students and staff not getting accurate responses.
0:28:12
The contract is a citywide contract with IBM Services that is held by OTI, the Office of Technology And Innovation.
0:28:22
What city agencies are allowed to do is purchase from that contract, and they set up a mini bid for resellers to pick the items that we wanted to buy.
0:28:30
This was one of the services we buy from IBM.
0:28:34
The bid was won by a major reseller CDW.
0:28:37
We should purchase orders CDW.
0:28:39
They provide the service and use that contract and work with IBM to with the reseller agreement to move the money that we pay to the reseller to IBM.
Farah Louis
0:28:52
Just the last part, do you feel that their performance met the requirement?
Scott Strickland
0:28:55
Good good point.
0:28:56
This one will get into a little bit of the weeds here.
0:29:00
What we buy from IBM is a service called IBM Verify.
0:29:04
What the unit of service is is you need to buy it in a 100,000 user bundles.
0:29:13
So we buy 30 of those bundles.
0:29:15
So we can have 3000, excuse me, 3,000,000 accounts on their system, each with a unique username and password that supports the logging in.
0:29:25
In the description of that service, even though we buy a lot of it, again supporting 3,000,000 users, the service is indicated that it runs at 400 transactions per second.
0:29:37
And that is what it says.
0:29:41
Since the beginning of the school year this year, IBM has provided more than that.
0:29:47
When the school year started, they provided 1400 transactions per second.
0:29:52
That was the upper limit that we could use.
0:29:53
We consume much less than that on a normal day.
0:29:56
But on the day that on February 13th, we had used up all 1400 transactions per second, still not getting everybody serviced during the period where IBM was doing corrective action, they moved that up to 2000 and in fact even tried to go to 3000 and they've left it at 2000 and that's where it's sits today.