FSU’s university-wide resiliency program focuses on doing the basics better

Florida State University CISO Bill Hunkapiller wouldn’t let Covid derail his plans to improve the university’s resiliency capabilities.

Hunkapiller started devising Seminole Secure, a four-part program designed to boost FSU’s disaster preparedness and response, just before the pandemic hit. He refined his plans through 2020 and then, this year, implemented its wide-reaching recommendations to ensure his institution could handle even better whatever emergency came next.

“Not the best time to roll this out,” he says. On the other hand, he admits that Covid demonstrated why a robust plan is so critical to have in place. “We had always tied disaster recovery to hurricane season, but a pandemic is also one of those risks and threats we need to be ready for.”

He adds: “The real issue, though, is how well can you respond in a time of disruption?”

Seminole Secure, which won a 2021 CSO50 award for outstanding business value and thought leadership, answers that question by delivering a program for managing risk assessment, vulnerability management, continuity of operations and disaster recovery planning on an ongoing basis with an emphasis on continuous improvement.

“At a strategic level, Seminole Secure is focused on centrally identifying the major risks to mission-related services that must be managed; developing university policies and a system and structure for maintaining needed plans and capabilities; and building cybersecurity capabilities in FSU units through concerted training activities,” Hunkapiller says.

Complexities and challenges

Hunkapiller started developing Seminole Secure in late 2019, after an FSU Office of Inspector General Services audit showed that some of the university’s 307 administrative units did not have effective disaster recovery plans.

“The audit was the stick needed to get change, and we used that audit as leverage to get traction for what we needed to do,” Hunkapiller says.

Even with high-level support for improvement, revising the system was no small feat.

The university has about 15,000 faculty and staff serving some 45,000 students. Its main campus is in Tallahassee, but it has additional locations in Panama City, Fla., as well as in England, Italy, Panama and Spain. It also supports a medical school and its multiple medical facilities around Florida.

In addition to its far-reaching geographical footprint, FSU has a broad range of operational needs to support the diversity of work typical of a university. It also has distributed IT. All those factors make for additional levels of complexity within disaster recovery and business continuity plans.

Furthermore, at the time of the audit, the university had 307 different units expected to devise their own disaster and recovery plans as well as complete an annual 140-question risk assessment. (Compliance with that requirement had historically been low, with less than 10% of units responding to the assessment in March 2017.)

Hunkapiller sought to overcome those complexities by using a multipronged approach to first tackle the inadequacies in the university’s business continuity, disaster preparedness and response capabilities and then encourage continuous improvement.

“The idea was to better identify risks, improve our vulnerability management and resiliency plans, ensure continuity of operations and bring risk down to a level that was tolerable,” says Hunkapiller, who worked with FSU’s Department of Emergency Management to devise Seminole Secure.

He adds: “It’s an elegant solution for the problem we had.”

The plan consolidated the 307 units (a setup Hunkapiller deemed unmanageable) into 54.

It then divided those 54 units into three groups, with 18 units in each group. The three groups cycle through a three-year rotation of tasks.

In the first year, the first 18 units each conduct a business impact analysis and then develop and document their own individual unit disaster recovery plan followed by a lessons-learned exercise.

The second 18 units would each develop their own unit-level risk assessments.

And then the third group of 18 units takes a break.

Then each group of 18 units moves to the second task and then the third before starting the three-year cycle again in the fourth year.

In addition to the tasks in that three-year rotation, each unit must undergo each year a disaster recovery exercise to identify and fix any areas that need improvement.

And each unit must perform a monthly vulnerability scan.

Furthermore, each of the 54 units has to identify someone to serve as an information security manager, who is  tasked with partnering with Hunkapiller’s centralized 20-member security team. And each unit must use the security department’s templates.

Hunkapiller’s security team works with the units throughout all the Seminole Secure processes, helping them with risk assessments, guiding them through the work, and training them to use the templates.

To help incentivize participation and robust plans, Hunkapiller devised security score cards to rank each unit.

“It’s just a way to hold these units accountable, to see if they’re doing what they should be doing. That was the whole idea around this program,” he says.

Key innovations

Seminole Secure also delivered several key innovations:

First, it required the central IT organization to develop a custom Continuity of Operations Planning (COOP) application to streamline the continuity planning process and provide a central system of record for continuity and emergency management information, emergency contacts, and response plans.

Seminole Secure also called for the development of a best practice toolkit for completing business impact analyses and disaster recovery plans. The toolkit, based on National Institute of Standards and Technology (NIST) Special Publication 800-34, provides scalable and automated business impact analysis workbooks and disaster recovery planning templates, with boilerplate language and step-by-step guidance to assist unit participants.

Additionally, Seminole Secure incorporated tools that helped FSU work through the pandemic—tools such as Microsoft Teams and Zoom to support collaboration and virtual engagements. And it established partnerships among FSU stakeholders, enabling them to share feedback so Hunkapiller and his team can refine and streamline processes and improve the overall program.

Building resiliency into the future

FSU rolled out Seminole Secure in early 2021.

Hunkapiller’s team developed a communication strategy and campaign to kick off and workshop program requirements with all participants, leveraging a one-stop-shop website, Microsoft Teams sites, and Zoom to enable communication, collaboration, and support.

Hunkapiller credits his team for the successful rollout of Seminole Secure, noting that he has a high-trust, high-performing team that includes former CISOs and CIOs who are skilled at both listening to their FSU colleagues’ concerns and conveying the importance of getting the resiliency work done.

Hunkapiller further credits his team’s skills in keeping the Seminole Secure project on track, despite the demands on FSU when Covid hit. He says that the security department faced pushback on the program’s timing, with colleagues insisting that they didn’t have the resources to tackle the required tasks; he heard some call it an “unfunded mandate.” While acknowledging those concerns as valid, he countered them by laying out the consequences of not taking action. “We’d go back to the fact that the president wants this done,” Hunkapiller adds, stressing the value of having top-level support for the security program.

Another reason for the successful launch of Seminole Secure: attention to the fundamentals, Hunkapiller says.

“None of this is cutting-edge technology. It’s the basics: Do you have a backup? Where is it? When do you need systems to come back on? It wasn’t like we had to build a rocket. We had the rocket. It was an organizational problem. We knew what we had to do, we just had to execute.”

But Hunkapiller stresses that Seminole Secure’s programmatic approach is one of its most valuable elements; it’s not a project or a one-time initiative but rather an approach designed to continuously improve and mature FSU’s resiliency capabilities.

“Already we’ve made the university more resilient, but we’ll always have additional threats, natural and manmade, and there are new vulnerabilities and new risks every day,” Hunkapiller says. “Now with every year we can get units even more secure and we can get them to respond more quickly and recover even faster.”