The final phase of an electrical infrastructure upgrade in the Industrial Agriculture and Communications Center that required a 12-hour campuswide information technology outage June 1 has been successfully completed.
The upgrade adds significant backup power capabilities for core campus technologies and marks a major milestone in the NDSU Information Technology Division’s plan to deploy next-generation network equipment this fall. The team behind the project, including staff from Facilities Management, Information Technology Division and Police and Safety, summarized it as an overall success during a wrap-up meeting a few days after the event.
“From our perspective, the process went very well,“ said Terry Wieland, director of network engineering and operations at NDSU. “Especially given this is the first time we’ve conducted a complete shutdown of all IT systems, including our network systems, sever operations, telecommunications and emergency systems, and the storage network.”
For more than four years before this month’s outage, which occurred from midnight to noon on Saturday, June 1, NDSU’s network core had remained operational without interruption.
The IT outage was required to transfer centralized data, voice and video systems to a new electrical architecture in the Industrial Agriculture and Communications Center’s data centers. The new infrastructure includes updated uninterruptible power supplies that provide standby battery backup in case of power failure, two static transfer switches capable of transferring electrical loads from one power source to another in a fraction of a single electrical cycle, and four power distribution units that provide power diversity and enhanced equipment protection.
In preparation for the electrical transfer, IT staff from Network Engineering and Operations, Enterprise Systems and Telecommunications and Emergency Support Technologies carried out a strategic, sequential shutdown of all servers and telecommunications systems at the onset of the outage window, followed by a complete shutdown of the network core. “Given the complexity of these enterprise systems, the overall orchestration of the project was impressive,” said Joan Chapek, assistant vice president for Telecommunications and Emergency Support Technologies. “It was amazing to see the confidence and knowledge of our staff throughout the planning and execution of this event.”
The project provided an opportunity for the telecommunications department in partnership with university police in realizing their long-term goal of creating a “plug-and-play emergency survivable police communications call center on campus,” Chapek said.
In preparation for the June 1 event, backup telecommunications systems were put in place in NDSU’s Research Park, which enabled University Police and Safety to continue its 24/7 emergency call center operations. The Research Park location was strategically chosen because of its high elevation, as determined by NDSU’s 2000 flood mitigation plan, its adjacency to diverse network and telecommunications infrastructure and direct access to local network providers.
“The value of the work completed during the outage will go a long way to provide the university with backup capabilities, should our communications systems be affected in the future,” said Ray Boyer, director of University Police and Safety. “This will further enable the University Police and Safety Office to provide timely communications during emergency situations.”
After electrical work in the Industrial Agriculture and Communications Center data centers was completed, the Network Engineering and Operations team brought the network core back online in approximately 45 minutes with minimal issues. According to Wieland, one critical piece of network equipment failed – a primary supervisor engine that provides intelligence for the wireless network core. A secondary, redundant supervisor engine took over without interruption at the time of failure, and replacement equipment was brought on site within three hours. “As part of our strategy to ensure we are able to keep our network operational 24/7, we identify key system components and negotiate a maintenance contract with our system provider to have spare components available to us within a four-hour window,” Wieland said.
The entire process – including shutting down all systems, transferring electrical power and restarting all systems – took place within the estimated 12-hour outage window. “This was an outstanding cooperative team effort on behalf of the campus by all participants and helps NDSU to better understand its resources, and the abilities of those resources, for events that may impact our communications systems,” Boyer said.
Work on the electrical infrastructure project began in October 2012. Since that time, the project team – including NDSU departments and units, state agencies, and contractors – put in an estimated 240 hours of labor in preparation for the June 1 event.
Wieland said the improvement significantly increases electrical redundancy supporting NDSU’s data center and telecommunications systems, and is a step in the direction of “future-proofing the network substructure to accommodate NDSU’s ever expanding, mission critical IT infrastructure.”
Marc Wallman, interim vice president for information technology, expressed his appreciation for all staff involved in the project. “Their time spent planning and preparing made this project extremely successful and minimally disruptive for our campus community and constituencies.”
NDSU is recognized as one of the nation's top 108 public and private universities by the Carnegie Commission on Higher Education.
