<jor technology services outage caused a Wifi outage throughout campus on the morning of September 11. Earlier that morning, a critical data center network switch rebooted due to a software bug. This led to a major loss of network connectivity, leaving students, faculty and staff without access to Moodle, the Hub and other academic sites that were critical on the first day of Tuesday classes. Information Technology Services (ITS) took time to isolate the problem and had a solution that had most of the core services restored by noon.
However, Director of Systems and Infrastructure Dave Flynn explained that the original bug is still present in one of two clustered switches and will be patched at a later date. ITS has stopped monitoring traffic that triggered the bug and is working with a vendor that will schedule to bring professional services to campus, who will then perform longer maintenance operations during midterm or winter break.
“Where we are is safe enough, that it wasn’t worth risking another downtime window,” said Flynn.
After the incident, ITS sent out an email to debrief students, faculty, and staff on what occurred. They also had a debriefing within ITS to identify technical and communication successes and areas for further improvement.
During the outage, ITS called the individual faculty members who taught during the outage, as well as their administrative assistants. ITS also arranged for Printing and Mailing Services to handle printing of materials needed for that day’s classes. Printing and Mailing Services was able to assist at almost no notice. Approximately $335 worth of printing occurred as a result of the outage, which is likely over 5,000 pages.
In the future, ITS would like to have more information added to their emergency incident management notebook for situations where communication is compromised. Improvements could also be made to more easily access and authenticate cloud services such as Google and Dropbox without a fully functioning data center.
Technology services issues in the past were often smaller and lasted longer, but a disruption this large to the entire data center is very uncommon.
Professors who taught at 8:15 a.m. and 10:10 a.m. that morning were affected the most, although ITS was able to contact them with help from Senior Curriculum and Scheduling Associate Ann May from the Registrar’s Office and Senior Assistant to the Dean of the College Peggy Pfister. Unfortunately, language professors who taught at different times were not immediately notified of the status of the outage, but ITS hopes to ensure that they will be contacted in the event of future disruptions. Students who could not find their classrooms also went to the Registrar’s Office, where May assisted them with a handwritten chart of all classroom assignments.
ITS advises that if anything similar happens in the future, staff and students are should call the ITS Help Desk. They should also subscribe to ITS Carl Alerts in case of future disruptions.
“We appreciate the community’s patience and support,” said Chief Technology Officer Janet Scannell. “Everyone seems to appreciate that technology is unpredictable and it’s impossible to avoid bugs. We recognize the level of inconvenience and scrambling that faculty and students had to go through and are working on a set of steps to limit future disruptions.”