On Friday, July 25, 2025, the City of St. Paul, Minnesota suffered such a large cyberattack that St. Paul Mayor Melvin Carter asked Minnesota Governor, Tim Walz, to bring in the National Guard to help recover. I am writing this on Saturday night, August 2, to organize my thoughts before talking about it on AM1280 The Patriot Sunday, August 3, 2025 at 2:30 p.m. US Central time, with Brad Carlson. I’ve been stewing about this for a while and, believe me, I have plenty of thoughts.
I’ll update this paragraph with a frame containing the recording after my appearance. Or listen live right here.
Before offering St. Paul cyberattack thoughts, here is the sequence of events, with help from the Google AI assistant, https://gemini.google.com.
- Friday, July 25, 2025 (early morning): The City of St. Paul’s Endpoint Detection and Response (EDR) systems detected “suspicious activity” on its internal network. EDR is the latest evolution of what we used to call antivirus software. This triggered an immediate response from the city’s IT teams. Here is a video from St. Paul Mayor, Melvin Carter.
- Monday, July 28, 2025: As a defensive measure to contain the threat and prevent further damage, the city initiated a full shutdown of its information systems. This included taking down wireless internet in city buildings, as well as systems for libraries and internal applications.
- Tuesday, July 29, 2025: Mayor Carter held a press conference to provide details on the “deliberate, coordinated digital attack.” He declared a state of emergency, which allowed the city to mobilize support from local, state, and federal partners. In response, Governor Tim Walz activated the Minnesota National Guard’s Cyber Protection Team to assist the city.
- City Chief Information Security Officer (CISO), Stephanie Horvath, and Office of Technology and Communication Director and Chief Information Officer (CIO), Jaime Wascalus, also participated in the press conference. At around the 12:30 mark in the press conference, CIO Wascalus offered these comments.
“We take our employee and our city data very very seriously. [Italics added.] We do use multifactor authentication in the majority of our services and we follow the federal guidance when it comes to our security posture. We have a very secure system, but unfortunately, these attacks can happen anywhere. And that’s something where we’ve been briefing leadership here about all the threats against us in the city, and so, unfortunately, this was our time and we are taking a look at all of our security policies and we do have very strong security policies. I’ll say very briefly, I’ve been here for three years and sometimes it’s so secure I can’t do my job as quickly as I want to. So it’s been wonderful to see the work the team has been putting in for decades to secure our city services.”
CISO Horvath followed up with damage control.
“Lots of security controls, but obviously, something didn’t work well here because we have this threat actor in our environment. But we have this dynamic and multilayered security controls. And that’s why we are in a containment and took aggressive action alright, but we are going to isolate and ensure this threat actor cannot do any further disruption in our network. And so, to go through all those layers right now, I don’t think would be most prudent becasue we are really involved in the investigation, the restoration, but at some point, I am primarily responsible for the security of the city’s networks and systems, so I’ll answer ongoing questions about our security controls.” - Ongoing (as of late July/early August 2025): The investigation continues with the help of the FBI, the Minnesota National Guard, and two private cybersecurity firms. While some services, like the city’s phone system, have been restored, many others remain down. The city has said there is no specific timeline for when all systems will be fully operational, as they are prioritizing a safe and secure recovery.
St. Paul Cyberattack Thoughts
From the July 29 press conference, we know that EDR (formerly called antivirus) software on city computer systems alarmed, and city staff eventually shut down many services to isolate compromised systems and limit further compromises. This seems like a prudent course of action.
CIO Jaime Wascalus played a starring role in the typical sensational data breach scenario I wrote back in 2015, when she said they take security seriously. I’m glad CISO, Stephanie Horvath, followed up with damage control. I met Stephanie several years ago when I helped with military family videos with Tee it up for the Troops. She also helps coordinate the annual Minnesota Cybersecurity Summit. I hope she keeps her word on letting the public know what happened with the St. Paul cyberattack. How did the attackers get inside? What systems did they compromise? What lessons did they learn?
And a biggie – what’s the city’s disaster recovery plan? They will need to capture all relevant forensic data and then wipe and rebuild everything relevant. This includes desktops, servers, routers, switches, and other potentially compromised devices. This will take time. It will be tedious.
They may need to temporarily stay silent because disclosure might somehow mess up an investigation. Or maybe they need to be sure of their facts before going public. But too many cyberattack victims use that as an excuse to never share what went wrong, which means everyone else misses another opportunity to learn, and somebody will eventually repeat the same mistake.
That leads to accountability. As the city CISO and a public figure in the cybersecurity community, Stephanie Horvath is on a hotseat.
Which leads to – what special expertise does the National Guard possess beyond what private-sector companies who respond to cyberattacks all the time offer? From Gemini again. “The Minnesota National Guard has a dedicated Cyber Protection Team, a unit trained specifically for this kind of crisis. These “citizen-soldiers” have extensive cybersecurity experience from their civilian careers and military training. Their role is not related to law enforcement but rather to provide technical assistance in the recovery and restoration of city systems. Their mission is to help get rid of the “bad actor” in the network, close security holes, and rebuild critical systems that were impacted.”
Which is less than useful, because St. Paul already had private-sector companies with lots of expertise onsite. But as I compose this, an idea bubbled into my head. Those private-sector companies bring IT architects who cost lots of money. But rebuilding hundreds of PCs and servers does not need elite skill. The National Guard probably saves money vs. private sector cybersecurity first-responder companies. It’s reasonable speculation.
Saving money leads to a fundamental business question that the public, and especially IT professionals, should ask more often. Is the City of St. Paul IT infrastructure an expense or an asset? If it’s an expense, then managers will focus on reducing its cost, which might have contributed to this attack. But if it’s an asset, then managers will work to maximize its value. So – does St. Paul operate its IT network cheaply to save money, or smartly to protect valuable assets? Hopefully somebody looks into that question during the inevitable St. Paul cyberattack postmortem.
Recent Comments