On Sept. 11, 2019, two security experts at a company that had been hired by the state of Iowa to test the physical and network security of its judicial system were arrested while probing the security of an Iowa county courthouse, jailed in orange jumpsuits, charged with burglary, and held on $100,000 bail. On Thursday Jan. 30, prosecutors in Iowa announced they had dropped the criminal charges. The news came while KrebsOnSecurity was conducting a video interview with the two accused (featured below).
Gary DeMercurio, 43 of Seattle, and Justin Wynn, 29 of Naples, Fla., are both professional penetration testers employed by Coalfire Labs, a security firm based in Westminster, Colo. Iowa’s State Court Administration had hired the company to test the security of its judicial buildings.
Under the terms of their contract (PDF), DeMercurio and Wynn were permitted to impersonate staff and contractors, provide false pretenses to gain physical access to facilities, “tailgate” employees into buildings, and access restricted areas of those facilities. The contract said the men could not attempt to subvert alarm systems, force-open doors, or access areas that require protective equipment.
When the duo’s early-morning Sept. 11 test of the security at the courthouse in Dallas County, Iowa set off an audible security alarm, they followed procedure and waited on-site for the police. DeMercurio and Wynn said when the county’s sheriff deputies arrived on the scene just a few minutes later, they told the officers who they were and why they were there, and that they’d obtained entry to the premises via an unlocked door.
“They said they found a courthouse door unlocked, so they closed it from the outside and let it lock,” Dan Goodin of Ars Technica wrote of the ordeal in November. “Then they slipped a plastic cutting board through a crack in the door and manipulated its locking mechanism. (Pentesters frequently use makeshift or self-created tools in their craft to flip latches, trigger motion-detected mechanisms, and test other security systems.) The deputies seemed impressed.”
To assuage concerns they might be burglars, DeMercurio and Wynn produced an authorization letter detailing the job they’d been hired to do and listing the names and mobile phone numbers of Iowa state employees who could verify their story.
After contacting some of the court officials listed in the letter, the deputies seemed satisfied that the men weren’t thieves. That is, until Dallas County Sheriff Chad Leonard showed up.
“The pentesters had already said they used a tool to open the front door,” Goodin recounted. “Leonard took that to mean the men had violated the restriction against forcing doors open. Leonard also said the men attempted to turn off the alarm—something Coalfire officials vehemently deny. In Leonard’s mind that was a second violation. Another reason for doubt: one of the people listed as a contact on the get-out-of-jail-free letter didn’t answer the deputies’ calls, while another said he didn’t believe the men had permission to conduct physical intrusions.”
DeMercurio and Wynn were arrested, jailed, and held for nearly 24 hours before being released on a $100,000 bail. Initially they were charged with felony third-degree burglary and possessing burglary tools, although those charges were later downgraded to misdemeanor trespass.
What initially seemed to Coalfire as a momentary lapse of judgment by Iowa authorities quickly morphed into the surreal when state lawmakers held hearings questioning why and how someone in the state’s employ could have so recklessly endangered the safety and security of its citizens.
Judicial Branch officials in Dallas County said in response to this grilling that they didn’t expect Coalfire’s physical penetration testing to be conducted outside of business hours. State Sen. Amy Sinclair was quoted as telling her colleagues that “the hiring of an outside company to break into the courthouses in September created ‘significant danger, not only to the contractors, but to local law enforcement, and members of the public.'”
“Essentially a branch of government has contracted with a company to commit crimes, and that’s very troubling,” lamented Iowa state Sen. Zach Whiting. “I want to find out who needs to be held accountable for this and how we can do that.”
Those strong words clashed with a joint statement released Thursday by Coalfire and Dallas County Attorney Charles Sinnard:
“Ultimately, the long-term interests of justice and protection of the public are not best served by continued prosecution of the trespass charges,” the statement reads. “Those interests are best served by all the parties working together to ensure that there is clear communication on the actions to be taken to secure the sensitive information maintained by the judicial branch, without endangering the life or property of the citizens of Iowa, law enforcement or the persons carrying out the testing.
Matthew Linholm, an attorney representing DeMercurio and Wynn in the case, said the justice system ceases to serve its crucial function and loses credibility when criminal accusations are used to advance personal or political agendas.
“Such a practice endangers the effective administration of justice and our confidence in the criminal justice system,” Linholm told The Des Moines Register, which broke the news of the dropped charges.
While the case against Coalfire’s employees has rallied many in the cybersecurity community around the accused, not everyone sees this dispute in black-and-white. Chris Nickerson, a digital intrusion specialist and founder of LARES Consulting, said in a Twitter post Thursday that “when a company puts us in harm’s way due to their poor planning, failed sales education, inadequate project management and deplorable contract management…We shouldn’t celebrate them. We should hold them accountable.”
Asked to elaborate, Nickerson referred to a recent podcast which touched on the arrests.
“The things that concern me about this situation are more of the pieces of safety that exist across how the industry instruments doing these types of engagements,” Nickerson said. “They seem very, very reasonable and obvious once they become obvious but until then they’re completely foreign to people.”
“It’s really on the owners of the organization to educate the customer of those potential pitfalls,” Nickerson continued. “Because there isn’t a good standard. We haven’t all gotten together and institutionalized the knowledge that we have in our heads and dump it down to paper so that someone who is new to the field being tasked with this can go through and say, ‘Hey, did you ask them if the city versus the state versus the building owner and the real estate people…are all of these people in lock step?'”
Coalfire CEO Tom McAndrew seemed to address this point in our interview Thursday, saying there were two unique aspects of this particular engagement. First, although the client in this case said they did not want Coalfire to make local law enforcement aware of the ongoing engagement prior to testing the physical security of the site, it was clear after the fact that state officials never did that on their own.
More importantly, McAndrew said, there was ambiguity around who actually owned the buildings that they were hired to test.
“If you’re doing a test for the state and you walk into the building and it’s the courthouse and you’re doing a test for the court system, you’d think that they would have jurisdiction or own it, and that turned out not to be the case in this scenario because there’s some things the state owns and some things the county owns, and that was something we weren’t aware of as we did some of this work,” he said. “We didn’t understand the nuances.”
Asked what Coalfire has learned from this ordeal, McAndrew said his company is likely to insist that local, state and even federal law enforcement be informed in advance of any penetration tests, at least as far as those engagements relate to public entities.
“When we look at the contracts and we look at who’s authorized to do what…typically, if a [chief security officer] says test these IP addresses, we would say okay that’s enough,” he said. “But we’re questioning from a legal perspective at what point does that need to have legal counsel review.”
McAndrew said it’s probably time for experts from various corners of the pen testing community to collaborate in documenting best practices that might help others avoid a repeat of the scenario in Dallas County.
“There’s no standard in the industry,” he said. “When it comes to these sorts of issues in red teaming — the legal challenges and the contracts — there’s really nothing out there. There are some things that can’t be undone. There’s the mugshots that are out there forever, but even as we get the charges dropped, these are permanently going to be in the federal database. This is a permanent thing that will reside with them and there’s no legal way we’re aware of to get these charges removed from the federal database.”
McAndrew said while he remains frustrated that it took so long to resolve this dispute, he doesn’t believe anyone involved acted with malicious intent.
“I don’t think there were any bad people,” he said. “Everyone was trying to do the right things — from law enforcement to the sheriff to the judges to the county — they all had the right intentions. But they didn’t necessarily all have the right information, and possibly people made decisions at levels they weren’t really authorized to do. Normally that’s not really our call, but I think people need to be thinking about that.”
Agreed that there needs to be more collaboration among the Pen Testing community and that engagements need to be clearly defined and signed off by all parties involved. Let’s not lose site that the purpose of the test was to legally identify and resolve weaknesses to enable a secure environment. Evidently the senators and sheriff involved don’t realize that someone with actual criminal intent could care less if was two in the morning or two in the afternoon, or if a door was locked or unlocked. They have done a disservice to the citizens of Iowa.
After conducting over 200 vulnerability assessments and pen tests, my take is that this was primarily a failure on Coalfire’s part. Why? Whenever one is dealing with Government clients, e.g. Federal, State, Local, etc., a level of ineptness has to be assumed on the client’s side. So before operating in these environments, the pentest team has to ensure everything is in place and all key personnel are in the loop… i.e., due diligence is critical to avoid hitches.
The link to the contract terms is broken suddenly this evening. It worked fine this afternoon:
This is why so many go to the Dark Side…good guys get slammed and they say “Why me” n “Why bother” ?
So are they locking the doors at the court house now?