Bit9, a company that provides software and network security services to the U.S. government and at least 30 Fortune 100 firms, has suffered an electronic compromise that cuts to the core of its business: helping clients distinguish known “safe” files from computer viruses and other malicious software.
Waltham, Massachusetts-based Bit9 is a leading provider of “application whitelisting” services, a security technology that turns the traditional approach to fighting malware on its head. Antivirus software, for example, seeks to identify and quarantine files that are known bad or strongly suspected of being malicious. In contrast, Bit9 specializes in helping companies develop custom lists of software that they want to allow employees to run, and to treat all other applications as potentially unknown and dangerous.
But earlier today, Bit9 told a source for KrebsOnSecurity that their corporate networks had been breached by a cyberattack. According to the source, Bit9 said they’d received reports that some customers had discovered malware inside of their own Bit9-protected networks, malware that was digitally signed by Bit9’s own encryption keys.
That last bit is extremely important, because Bit9 is a default trusted publisher in their software, which runs on customer PCs and networks as an “agent” that tries to intercept and block applications that are not on the approved whitelist. The upshot of the intrusion is that with a whitelist policy applied to a machine, that machine will blindly trust and run anything signed by Bit9.
An hour after being contacted by KrebsOnSecurity, Bit9 published a blog post acknowledging a break-in. The company said attackers managed to compromise some of Bit9’s systems that were not protected by the company’s own software. Once inside, the firm said, attackers were able to steal Bit9’s secret code-signing certificates.
“Due to an operational oversight within Bit9, we failed to install our own product on a handful of computers within our network,” Bit9’s Patrick Morley wrote. “As a result, a malicious third party was able to illegally gain temporary access to one of our digital code-signing certificates that they then used to illegitimately sign malware. There is no indication that this was the result of an issue with our product. Our investigation also shows that our product was not compromised.”
The company said it is still investigating the source of the breach, but said that it appears that at least three of its customers were sent malware that was digitally signed with Bit9’s certificate.
There may be deep irony in this attack: While Bit9 has made a name for itself based on the reality that antivirus software cannot keep up with the tens of thousands of new malware variants being unleashed on the Internet each day [the company brags that Bit9 is the only security firm to stop both the Flame malware and the RSA breach attack, even before they were identified by traditional/legacy antivirus companies], there is a better than even chance that the malware signed with Bit9’s keys was first detected with traditional antivirus products. But only time will tell how the initial discovery really played out.
Jeremiah Grossman, chief technology officer for security testing firm Whitehat Security, said the attackers who broke into Bit9 almost certainly were doing so as a means to an end.
“I guess if you’re a bad guy trying to get malware installed on a computer at a hardened target that is using Bit9, what choice do you have except going through Bit9 first?” Grossman said. “This is not the result of some mass malware blast. This is almost certainly highly targeted.”
Grossman and other experts say the attack on Bit9 is reminiscent of the 2011 intrusion at RSA Security, which was widely viewed as a precursor to attacking upstream targets protected by the company’s products. In that intrusion, the attackers targeted RSA’s proprietary algorithms that protected networks of thousands of companies.
“In that case, the attackers weren’t doing it for gain at RSA as far as anyone’s been able to tell, but there were reported attacks shortly after that against defense contractors that had characteristics of someone exploiting what was probably taken from RSA,” said Eugene Spafford, professor of computer science at Purdue University. “Those defense contractors were the real targets, but they were using a very strong security tool – RSA’s tokens. So, if you’re an attacker and faced with a strong defense, you can try to break straight through, or find ways around that defense. This is more than likely [the product of] very targeted, careful thinking by someone who understands a higher level of security strategy.”
In that sense, Spafford said, the Bit9 and RSA attacks can be thought of as “supply chain” hacks.
“Supply chain doesn’t necessarily mean the sale of finite items, but it’s all along the chain of where things might find their way into your enterprise that can be contaminated, and I suspect we’ll continue to see more of these types of attacks,” Spafford said.
The potential impact of this breach — both within Bit9 customer networks and on the company’s future — is quite broad. According to a recent press release, Bit9’s global customers come from a wide variety of industries, including e-commerce, financial services, government, healthcare, retail, technology and utilities. The company was founded on a U.S. federal research grant from the National Institute of Standards and Technology’s Advanced Technology Program to conduct the research that is now at the core of the company’s solutions.
“One of the things I’ve stressed to security companies I’ve done work for is that everything they do is based on trust in their brand and product, and that getting hacked is a fundamental attack on that trust structure,” Spafford said. “That’s an object lesson, but it may also say something if they aren’t eating their own dog food, so to speak.”
Grossman said the compromise at Bit9 demonstrates both the strengths and weaknesses of relying on an application whitelisting approach.
“It’s also interesting that they went after Bit9’s certs, and not by trying to exploit vulnerabilities in it. Instead of hacking the Bit9 application or network device, they went after Bit9 directly. That says a lot on its own.”
What is odd is that something so critical to their business and application, the signing keys, weren’t protected by their own application. How does that happen? Wow.
Or that their digital code-signing certificates weren’t on an isolated network. Isn’t this the very same mistake that some CA’s have made recently?
The keys to the kingdom. Yikes!
Waiting for R.S.H.: “There is no security. Deal.”
I’ve challenged RSH’s meme, or an interpretation, before. Blog readers thought it was one of the more entertaining and heated battles between regular contributers. Maybe Krebs readers will enjoy it so I pastebined it. I’m not calling a winner: it was just good fun between two debate rivals and both of us had our supporters.
Note that my views on specific things have evolved over time so don’t take these as current. I’ve edited these posts mostly to remove tangent discussions and a miswording or two on my part. Our original arguments (and rhetoric 8) are left in entirety.
pastebin link to nick p vs rsh: http://pastebin.com/w0Sjef0e
A modern summary of my view on what is required to develop secure software/systems.
https://www.schneier.com/blog/archives/2013/01/essay_on_fbi-ma.html#c1102869
Of course, Nick, you should acknowledge that your recommendations for “security”, as you have admitted before IIRC, don’t cover EVERYTHING that affects security.
Having “secure software” – which I still don’t admit necessarily exists in the real world unless it is “provably correct” AND large enough to be a real world useful product – is just a part of overall “security”.
“PEBCAC” will forever be a vulnerability until we have “end users” who are “provably correct” AI programs who have replaced all the humans in an organization. 🙂
“Of course, Nick, you should acknowledge that your recommendations for “security”, as you have admitted before IIRC, don’t cover EVERYTHING that affects security. ”
Security is an ongoing process that’s only as strong as its weakest link. We all know this. I work to make certain links as strong as possible. Others work on the other links. In my experience, the links I strengthen do well enough combined with careful (or restricted) users.
““PEBCAC” will forever be a vulnerability until we have “end users” who are “provably correct” AI programs who have replaced all the humans in an organization. :-)”
Lol. It’s very possible that it would take that. However, any useful Artificial General Intelligence software would be about as complicated in practice as the human brain. Then we’re back to square 1 with users: f***ed.
That’s why I said “provable correct”. We sure don’t have that with humans and never will! 🙂
Whether that will be possible with AIs (or Transhumans) is unknown at this point. I suppose it depends on processing power and whether the problem is computationally impossible or not. I’m not enough of a computer scientist or logician or mathematician to say.
Meanwhile, as long as humans are in the loop, my meme is unchallenged! 🙂
Thank you Bit9, for coming clean; with the breach info. Locally, the state must maintain both safety online and off!! Hiding info, hits the bottom line in sales, in the community you rely on, accountability is a must!!!
Let me just note that I just read a blog post elsewhere that cited this incident as a reason to consider white-listing “an absolute impossibility” (according to the blog post title anyway.)
I disagree completely. White-listing falls under the “you can haz better security” clause of my meme IF it’s applied intelligently and with a fine enough grain – which it isn’t in many white-listing products.
Products that white-list an entire PLATFORM – such as Java, .NET or PowerShell are stupid. For white-listing to work, you have to narrow it down to specific executables. Only THIS Java app should be able to run, or THAT PowerShell app – not just anything written in it.
The blog post also said that white-listing can’t be applied to Web programs, which seems obvious to me since you have no control over someone else’s Web site.
But black-listing clearly falls under the “you can haz worse security” clause, because there is no way the industry can keep up with 25 million pieces of malware, with another 75,000 a day being produced.
White-listing should never be considered a panacea. That falls under the “there is no security” clause.
I agree. Blacklisting has overall failed. Intelligent whitelisting (combined with other measures) has proven itself as a security improvement in both home and corporate settings.
I’ll add that a web site can be trusted [more] if you control the attack vectors. In recent years, technology has popped up for untrusted native code execution (Chrome NaCl), untrusted JavaScript w/out browser modification, robust middleware alternatives, etc.
However, I still contend what I did almost a decade ago when web was getting popular: insecurity and untrustworthiness is so baked into its design that any security tech for Web is just a bandaid. That will fall off.
I prefer old school TCP/IP, client-server apps written in safe languages and libraries. Actually, I ditched TCP for UDP-based protocols (i.e. UDT) in my desings years ago. Fewer problems there. Also, FTP haters should look up Tsunami or UDT-based Sector.
“Products that white-list an entire PLATFORM – such as Java, .NET or PowerShell are stupid. For white-listing to work, you have to narrow it down to specific executables. Only THIS Java app should be able to run, or THAT PowerShell app – not just anything written in it.
It’s not just platforms. On Windows, there are executables such as svchost.exe and rundll32.exe, as examples, which can execute malicious dll’s. This leads one to authorizing specific dll’s for such executables.
“The blog post also said that white-listing can’t be applied to Web programs, which seems obvious to me since you have no control over someone else’s Web site.
One’s control over web sites is very coarse-grained. It’s all or nothing for a web site or, perhaps, JavaScript embedded in iFrames. Modern browsers such as Firefox/NoScript, Opera and Chrome/Chromium allow an individual user to whitelist web sites for active content. Windows Group Policy for Internet Explorer also provides sysadmins with the capability to whitelist web sites for enterprise users.
Yeah, I hate svchost. On Windows XP, at least (not sure about 7 and 8 Task Managers), you need third party process analyzers to determine what’s being actually run by it which is a pain when checking for malware.
There’s a usability problem with white-listing things on the Web using NoScript. There isn’t enough granularity on the one hand, and on the other the site’s scripts are blocked completely until you enable it – and then things still don’t work until you enable a half dozen other sites – and you can’t always tell WHICH site needs to be enabled to make something work on the main site.
AdBlock enables you to block or allow any given element on a site’s page, which is nice granularity. But again you have to figure out which item you want to block which is not always obvious until you inspect the elements using browser tools.
None of this scales to be useful for white-listing executables on Web sites.
What’s needed is a tool that allows one to say “allow this script or executable on this/these site/sites to run and nothing else on this/these site/sites or any other site” – and provides some way to know which one to allow (or on the other hand assumes one already knows, for use with corporate intranet Web sites and executables) and this rule can be deployed globally.
This would help mightily with “drive-by malware” except of course when those white-listed executables are corrupted by hackers which in turn is a matter for the Web site to control.
One application at a time, shows the problems!!
” How does that happen? Wow.” “Due to an operational oversight within Bit9, we failed to install our own product on a handful of computers within our network,”. Simple, corporate policies and procedures were not followed plus there was not a test on the corporate network to verify that a box was compliant when the computer was attached to the network. I am going to assume that the situation was not the result of a malicious “inside job.” We can all learn a lesson from this unhappiness. First, review business procedures to prevent human error because, after all, most people do not awaken in the morning and say “Boy, today I am going to go to work and really mess up!” Create something as simple as a checklist that requires a sign-off. Airlines use checklists and hospitals have recently started using checklists in surgery. Then create an electronic audit of the box when it is first attached to the network followed by occasional subsequent audits.
I once worked for a company that experienced a monumental screw-up. It sank the company. Thousands of people had to “hit the bricks.”
“create an electronic audit of the box when it is first attached to the network
Shouldn’t Bit9’s digital code-signing certificates be on an isolated network?
People are human. They eff up.
Good ones know that, plan for it, and try to mitigate it. So their f* ups are in the planning and mitigation. Things like not running config compliance checks on their own network. Not having the level of network security that their enterprise customers would have (24x7x365 analyst monitoring, segmented internal subnets with locked down ACLs and full IDS coverage to give the analysts stuff to look at, etc.).
The cobbler’s children have no shoes…
yes indeed, you are right Cody. i think it is itself a big security flaw.
This is really bad for the company’s reputation
in french there’s an expression that sums it all:
“cordonnier mal chaussé.”
An expression that is completely meaningless to non-French speakers … unless you, I dunno, provide some sort of translation.
“The shoemaker’s son always goes barefoot” (cordonnier = shoemaker), only difference being that in French it is the shoemaker that goes barefoot 😉
I once heard this translated as “The cobbler’s children have no shoes.”
I now write software development tools, so as you might imagine I say this phrase quite often.
Bien dit. MDR
i’m rather surprised to see a whitelist vendor, of all things, using digital signatures in this way. you’d think they would know better. trusting the publisher is not the same as, and should not be mistaken for, authorizing the program.
this should be patently obvious, what with the digitally signed malware that has made the news in the past couple years (ie. stuxnet).
Chicken and egg problem. How do you whitelist an update to the whitelist software?
Yeah, I know, you can design it to solve the problem. But that’s harder than taking a shortcut.
Also have the problem of needing to update the whitelist every time any whitelisted app is patched, if you use hash sigs as your sole technology. Think about trying to keep up with just Adobe, Java and Microsoft. It doesn’t scale well at all.
So you trust certs. And what cert is more to be trusted than your own.
Oops.
“Chicken and egg problem. How do you whitelist an update to the whitelist software?”
well, since bit9 already uses the *cloud*, all they have to do is update their database to include a new trusted file.
“Also have the problem of needing to update the whitelist every time any whitelisted app is patched, if you use hash sigs as your sole technology. Think about trying to keep up with just Adobe, Java and Microsoft. It doesn’t scale well at all.”
i’ve already thought about it, and bit9 has too. they scale pretty well because they don’t do the heavy lifting. they scan files with multiple vendors’ products, generate hashes, and maintain a database. back in 2007 they reported they were processing 500,000 files a day from microsoft alone.
“So you trust certs. And what cert is more to be trusted than your own.”
no, you don’t trust certs – especially when you don’t have to, and they don’t have to. trusting digitally signed code is a poor substitute for a proper whitelist and i still maintain that it is surprising for the symantec of whitelisting to be caught doing that.
“Bit9, a company that provideD software and network security services”
There, fixed that for you.
But it really is a shame that one mistake will probably ruin this company’s reputation. Hope they bounce back from this.
Doubtful. This sort of failure often constitutes an “company extinction event”…
OUCH. Egg. Face. “Do as I say”, etc. etc. I don’t think this is a so called ‘EEC’ (Company Extinction Event). That happens when you have a false positive in your AV-engine and you brick thousands of PC’s. But this certainly is doing severe reputation damage. Heads will roll for sure!
Just about every AV vendor has bricked thousands of PC – I can think of Kaspersky, AVG, and Avast just recently IIRC – and none of them are out of business.
In fact, AV updates are the most problematic of all updates (with the possible exception of Microsoft .NET and Visual Studio.) I had incredible problems with Kaspersky with a former client constantly which led me to dump them and switch to Avast. KAV is constantly screwing up their updates – not so much causing false positives, but simply being unable to complete the update, constantly failing to update their license “black list”, taking ridiculously long times to update a single PC, etc.
Frankly, no one outside of Linux has even come close to solving the “update problem.” I’ve almost never had problems with Linux updates, but Windows has been a disaster for years.
I’ll never forget the time McAfee blacklisted Windows XP’s svchost.exe (April 2010). I know entire companies that went offline as a result of that debacle. A local hospital actually shut their systems down and left them down after it had already knocked out a sizable portion of their systems. Not only didn’t McAfee go out of business, Intel announced their acquisition intent in August 2010.
While I know many AV programs have had similar woes, I’ve kept an ear out for McAfee’s woes ever since the days when I did phone tech support; they flagged one of our hundred-ish MB video files as containing an executable virus (that was contained inside its own dedicated executable – a couple KB in size. ah, those were the days). The phone just wouldn’t stop ringing…
Is it just me…or are these types of hacks rapidly spiralling out of control?
If Bit9 and an ex-US president can get hacked, what’s next?
I have got a bad feeling that we’re heading inexorably towards something truly bad like a major infrastructure or hospital hack where lives will be lost.
I sincerely hope I’m wrong…..
If it happens, then it will be essentially by design. There have been nearly unbreakable setups for years. The government reserves the best of them for defense use. The stuff they push for the rest of us is easy to hack, backdoor, break, etc. They also try to push control/surveillance schemes disguised as security improvements. They also push insecure stuff with the label that it’s secure (*cough* nettop *cough*).
On the business side, the problems are incentives, ignorance, and apathy. Incentives are the biggest issues. Many SCADA, medical, and consumer devices are designed to provide features for a profit. Security hurting ROI won’t usually make the cut. Customers also don’t pay extra for it in many cases. Ignorance represents that many don’t understand security issues or know how to build resilient systems. Apathy is simple: many don’t care.
(I haven’t even mentioned legacy and convenience. They can be security killers.)
So, we can do better, but probably won’t. There will be niche markets that get the safety/security they demand. However, most stuff will remain insecure because people choose that at every level.
As I’ve said before, cybercrime is a “growth industry”. It won’t be stabilized until it has become ubiquitous.
That said, I don’t expect large scale fatalities or injuries any time soon. That generally isn’t the motivation for the attackers. Industrial and government espionage and profit are the main motivations. You start killing people, other people get cranky real fast.
However, given the US and Israel’s participation in Stuxnet – which had the possible potential of exploding a centrifuge which is not an insignificant event for nearby workers – the US is in a bad position to complain about injury-causing malware. Stuxnet was probably the greatest advance in precisely that area of malware.
Not to mention the assassination of scientists…
You mentioned that “the malware signed with Bit9′s keys was undoubtedly first detected with traditional antivirus products”. This is the part of the article that is hyperbole.
The (most likely) nation state responsible for the attack I’m sure is smart enough to test their malware on traditional anti-virus products before releasing in the wild. In practice I see only 20-30% of malware detected initially by different AV products.
Initially the malware not caught by AV is usually detected by dynamic analysis, Internet traffic analysis, or user reported based on anomalous behavior. The customers that reported the malware initially were probably fairly advanced and would be expected to have other security measures in place.
You’re right Rob, and I massaged that paragraph shortly after publishing the story to emphasize that it was pure speculation. There is a lot we don’t know at this point about the attack.
while it’s true that someone sophisticated enough to pull this off probably would have performed malware q/a as well to make sure AV products wouldn’t detect it right off the bat, it’s still possible that some cloud-based AV acquired a sample because it wasn’t something they had a hash for yet and determined it’s maliciousness subsequent to that.
Very unlikely. How, and why, would your “cloud based AV” acquire a sample? Just not having a signature for something is meaningless, there are gazillions of low grade malware packages that fit that description, and this was much higher grade stuff.
Odds are that this was caught by some network analysis, or custom proprietary or classified endpoint monitoring. Maybe some really good commercial package has heuristics good enough to catch this, but I doubt that. Really doubt any cloud based stuff could be smart enough to recognize that anything bad was even happening with this.
“Very unlikely. How, and why, would your “cloud based AV” acquire a sample?”
that’s how some of them operate. if they encounter a file they’ve never seen before (and if the customers settings allow it i suppose) the file gets uploaded to the cloud for automated analysis (and potentially manual analysis should certain conditions be met).
” Just not having a signature for something is meaningless, there are gazillions of low grade malware packages that fit that description, and this was much higher grade stuff.”
not a signature, a hash. signatures are not hashes and hashes are not signatures. some cloud-based products maintain databases of files both good and bad.
“Really doubt any cloud based stuff could be smart enough to recognize that anything bad was even happening with this.”
unless that ‘cloud based stuff’ happened to be designed to run the file in a sandbox looking for signs of badness. some do.
Cloud based security is still so unreliable. the security of all WIFI applications is in question. I warned this government in 2007, as soon as phones began too be capable of internet access. I do not know of a company that can safely guarantee any Wifi or any phone application safety~!!
I wonder who their security officer is (I mean…was)….
Not good….and actually worse than the RSA breach…Now they have to generate and get new keys to ALL customers. What a change management nightmare for an IT departement. Bye Bye Bit 9…..
The BIT9 site crows about:
“thousands of corporate, banks & Gov’t” customers, worldwide.
Q1:
Does anybody here
have even a partial list of these BIT9-compromised customers?
That would be useful, I think…
Q2:
Anything regular PC users of online banking, etc.,
can do to protect themselves when accessing the sites of these BIT9-“protected” Banks, etc?
(in plain English, please….).
Do not give up on Bit9, this unfortunate incident could safely secure more websites!!
Bit9 was reckless and got compromised. Plenty of companies have done the same over the past 10 years, many quite publicly. A company mishandling security virtually never leads to other companies improving security. A decade of the same kinds of attacks certainly didn’t help Bit9. And it’s a company specialized in beating malware. Ouch.
Unbelievable! When does this end????
The link to the Flame malware article doesn’t work:
https://www.bit9.com/company/news-release-details.php?id=254
-richard
Link to Flame article:
https://www.bit9.com/why-bit9/threats-stopped/flame/
So now that China has just put an American company out of business through malicious hacking, is this country finally going to be moved to act?
Source? Thought not.
And even if someone in China were the culprits, how exactly should the US “act”?
How should Iran “act” against the US given Stuxnet and Flame?
Bit9 wasn’t using a HSM to protect their keys?
Any alternatives to Bit9?
sure there are lots of whitelisting software providers out there. heck, you could even do whitelisting with kerio personal firewall.
now as far as vendors with comparable databases of binary file intelligence, that i’m not so sure about, but you don’t have to use bit9’s software (or even be a customer) to access that database – it’s just easier if you do.
CoreTrace is an alternative to Bit9. I assume that right at this very moment CoreTrace is checking their six.
We use SolidCore – now McAfee App Control. It’s solid and you can build multiple lists and apply them where needed.
For Windows, if your users are non-Admins, then there’s Software Restriction Policy. Properly used in disallowed-by-default mode, it has the effect of whitelisting what the Admin has installed, and blacklisting a list of potentially-malicious filetypes in any other location.
Or there’s AppLocker if your platform supports it. SRP is supported by Pro/Business/Ultimate/Enterprise versions of WinXP onwards.
As for the Bit9 situation, I think “facepalm” sums it up pretty well.
Check out bromium.com
And the hackers probably believe they have failed! They were after all, discovered!
Doubt that the hackers are anything less than successful. How many other networks did they target and still remain undetected? How long were they on the networks that discovered them before they were discovered? How many back doors did they manage to plant that remain undetected?
They are advanced, and they are persistent. This is just one arrow from their quiver.
Note an important distinction from the RSA case: Bit9states it did not follow adequate procedures on a few of its machines, but that its own product (Parity) was not breached.
The certificates were because of the process error, but nothing indicates the product itself was compromised.
So when you’re saying “malware that was digitally signed by Bit9′s own encryption keys”, it’s best to be clear that a certificate was compromised, not the Bit9 technology itself.
How well have they audited their source tree? How much of their tool chain could have been exposed to unauthorized modification by the same access that was able to steal or misuse their signing facility?
Note that Bit9 stated in their blog that their product was not compromised. Taking the statement at face value, does it cover both modification of their source code as well as theft of their source code?
If the miscreants acquired Bit9’s source code, they will surely analyze it for design and coding flaws. And perhaps attacks on Bit9’s whitelisting software will be forthcoming?
You’re kind of reading things into their statements. All we need to know is that the product’s root of trust is compromised, so it should be treated as a risk.
Regarding source compromise, it’s rare to see any of the things you suggested happen. Source is usually stolen by competitors for competitive advantage. The threat of subversion of source or binaries is the most damaging one of all, yet it’s rarely employed. It is possible that 0-days might be combed out of the source, but we don’t even know the source was compromised.
So, everything is idle speculation except to say the product can’t be trusted in its current state. I’d say that, at a minimum, they should lock their network down, restore from older backups, apply updates, check the signatures (manually), sign with a new key and push it to clients who’ve reinstalled their software. That’s a tough process to go through but should remove subversions that the attackers are most likely to use. It also recreates a root of trust [that’s not compromised.]
“nothing indicates the product itself was compromised. ”
Flip that on its head. Nothing indicates the product itself could have prevented the breach. Which is why saying the cause was that they didn’t install it on some machines is disingenuous.
Unless they tell us how the machine was breached and precisely how their software would have prevented it, it’s just PR for their product.
And even if their software would have prevented the breach, who’s to say the hacker wouldn’t have used another method to breach them?
By claiming the absence of their software was the sole reason they were breached, they are claiming their software is a “security panacea” which, by definition, it cannot be.
“But the real dumb statement – by them – is that the reason they were breached is because their white-listing product wasn’t installed on all their machines.
There’s more if you go to the Bit9 blog linked in Brian’s article:
“We eliminated the operational issue that led to the illegal access to the certificate and ensured Bit9 is installed on all of our physical and virtual machines.
Note that there are 2 items in this statement. Failure to install Bit9 on all of their systems isn’t the whole story given by Bit9. There’s also “the operational issue that led to the illegal access to the certificate”. Thus, my above comments on isolation of their digital code-signing certificates.
Exactly my point. Their statement quoted here is:
““Due to an operational oversight within Bit9, we failed to install our own product on a handful of computers within our network… “As a result,…”
“As a result” of failing to install their own product…
So whatever they claim on their Web site in terms of other issues, this is the one they’re hawking – that their product would have protected them from this breach.
Which is contradicted by the other statement.
In fact, the other statement refers only to the CA servers – as you correctly point out – and not to the overall breach.
And then there’s…
“There is no indication that this was the result of an issue with our product. Our investigation also shows that our product was not compromised.”
Which again is ninety percent concerned with touting their product. Which is understandable but still disingenuous in my view.
It would actually be easy to protect something like this using very robust methods. They control the network, signing system, client app, communications methods, etc. They can eliminate risk at each level with ease. Instead, they practically handed the signing keys to the attackers. Far from merely “effing up,” this “security firm” showed utter ignorance of how to protect software assets.
Clive and I have been talking about these kinds of issues on Schneier’s blog for years. There are numerous security schemes and services in widespread use that trust some central authority. Even a near perfect client is only as trustworthy as the central authority. The problem is that the center of trust is often ill-protected. Many are insecure enough that they can’t stop regular hacking attempts, much less sophisticated subversions.
Such systems should be developed to a higher standard of security. At the least, the low hanging fruit should be knocked out with every free, cheap and easy countermeasure we have. There’s quite a few of them. It’s obvious they weren’t in use. And another one bites the dust.
“They control the network, signing system, client app, communications methods, etc. They can eliminate risk at each level with ease.”
That’s really overstating the case, pardner! 🙂
If that were true, there would be no infosec industry.
Of course I understand what you mean – they could have done more.
But the real dumb statement – by them – is that the reason they were breached is because their white-listing product wasn’t installed on all their machines.
Who says their product could have prevented the breach when we don’t know how the breach was done in the first place? There are probably thousands of ways to breach a network that doesn’t rely on running an unauthorized executable.
Their statement is just an attempt to continue to promote their product as a “be-all” solution to “security” while admitting they screwed up at REAL “security”. They’re essentially claiming that their product would have prevented the breach and by extension can prevent any breach.
It’s just not true.
“Who says their product could have prevented the breach when we don’t know how the breach was done in the first place? There are probably thousands of ways to breach a network that doesn’t rely on running an unauthorized executable.”
I agree. It’s why I mentioned all those other things. 😉
“”They control the network, signing system, client app, communications methods, etc. They can eliminate risk at each level with ease.”
Let me justify the statement. Certainly they will try to use COTS solutions or libraries where they can. This especially makes sense when initially getting a product out the door. That said, I learned a trick from 90’s-era MLS systems: incremental assurance. You gradually increase the assurance of your setup over time, piece by piece, with ongoing sales providing the funding (and profit too!).
So, they might start with the basics. Then, the obfuscation and security improvements begin. Maybe the server is OpenBSD or a security-enhanced OS. Maybe ditch TCP for a simple UDP-based protocol. Furher isolate libraries, network segments, etc. Use very simple protocols over simple hardware to interface untrusted portions with code signing. Make actual code signer release data to untrusted net over data diode (cheap ethernet mods exist for this). And so on.
Roger Grimes’s list is a good place to start.
https://www.infoworld.com/d/security/10-crazy-it-security-tricks-actually-work-196864?source=fssr
Here’s a few I’ve done in the past that I posted on Schneier’s blog.
http://www.schneier.com/blog/archives/2013/02/new_york_times_3.html#c1138161
You see, they get to choose what to do about each thing on the list I gave. There’s many options that stop many attacks or levels of attacker. They don’t have to do it all at once. Hence, they “control” these aspects of their operational security.
And what did they do with that opportunity? See Krebs. 😉
I agree about the list of things that could be done.
What I was complaining about was the phrase “eliminate risk”… 🙂
“Security” should be about “handling threats” than “eliminating” them. The two concepts are not the same. Because the latter is not possible.
My current working definition of “security” is “the ability to handle threats so that one is not significantly harmed.” This subsumes “eliminating” threats but goes beyond that to assume that not all threats can be “eliminated” (or prevented from either existing or occurring) and therefore the rest must be “handled”.
For example, in current infosec theory, the goal is not to stop breaches from ever occurring, but to be able to detect and respond to a breach in such a manner as to defeat the attacker’s purpose. This concept fits in with my security theory fairly well. It doesn’t mean you don’t TRY to prevent breaches – you just assume you won’t be one hundred percent successful and therefore you have procedures in place to deal with the prevention failures.
This is what “layered security” was supposed to be about until it became more about installing stuff everywhere for prevention purposes rather than dealing with the situation when all of that stuff fails anyway.
As a sci-fi character once said, “One handled situations. It was that simple.” Except of course in practice it’s not simple. In fact, it’s almost impossibly hard – which is why my meme says, “There is no ‘security’.”
I see your point you’re making.
But you can’t market a security product as being something other than a solution which stops everything. At least not when every other product is being touted as something which stops everything. Not when you’re trying to sell a product to customers that have a choice.
sounds like an internal f-up, but i’ve seen their software be ruthlessly effective at stopping malware. hope they play their cards right and recover!
Another case for my claim that you should not trust software that is only distributed as binaries:
http://www.koch.ro/blog/index.php?/archives/153-On-distributing-binaries.html
If you really care about security, you should try to only run software of which you got the source code, that you can compile yourself and that is available in the repository of a big enough distribution. Of course this is only a start.
Although this is only really of value if you have the ability/resources/time to review the source code that you are downloading. Unfortunately this is true in a very small number of organizations.
Having source code (and time to review it) is only one piece of the pie. Can you trust your tool chain? How about your kernel? And don’t forget the hardware itself. Read Ken Thompson’s “Refections on Trusting Trust” if any of these ideas are new to you.
Finally, binaries from repositories can be (and have been) compromised.
I think Bit9 is not telling what happened. It is also possible they don’t know what happened.
The scenario is far more troubling.
Here is my guess.
First, there is no way the Bit9 CA servers were directly connected to the Web. Most likely behind a NATed firewall.
To reach those servers, the hackers probably targeted some Bit9 employee with a spear attack. They could have also used an exploit on Bit9 servers that are connected to the web. From the compromised machine, they scanned the network and found the servers that held the key.
Bottom line is that they got around the Bit9 product. To control damage and protect their product, Bit9 is feeding us horseshit. Enjoy! Next course is more horseshit.
Also, most attacks, be it RSA or Dupont or Bit9 will follow the mechanism I have outlined. Comparing how a real attack unfolds with the press release sheds good light on BS.
“First, there is no way the Bit9 CA servers were directly connected to the Web. Most likely behind a NATed firewall. ”
Like that matters – if the firewall itself has a vulnerability that wasn’t patched. A LOT of security appliances can be bypassed or even taken over by hackers according to a number of infosec conference talks I’ve seen.
The bottom line is that the Bit9 software is a “white list” product. It’s not designed to protect from the myriad OTHER ways a system can be breached and code can be run to bypass a white listing methodology.
White listing is not a panacea. It’s an ADJUNCT technology whose sole function is to minimize – not eliminate – the likelihood of unauthorized executables being executed.
If you have vulnerabilities elsewhere it will not protect you any more than any other technology.
My meme remains unchallenged in the real world! 🙂
Richard,
I agree NATed firewall is not a protection. I was just explaining how the attack may have propagated and the attack vector is more sophisticated than explained by Bit9. It is likely that Bit9 is either hiding what they know or they are just clueless. Not good either way.
My point is that the attackers have to have gotten around the Bit9 product exactly as you explained. In fact, if an exploit is executed, but not new file/process launched, Bit9 is useless. So there is a vast array of attacks that Bit9 will never detect.
I think we are in agreement.
Yup.
When a security company like this gets breached, in my view they have an industry obligation – as well as an obligation to their customers – to explicitly explain HOW they were breached and what they did to make sure it never happens again.
Of course, their legal departments generally disagree, which is understandable.
Perfect example of Talking the talk, whereas we would have been expecting such organizations to demonstrate examples of Walking the Walk
yes,but honestly cyber crime making lower street crime and violance,so if the wolfes not hungry then lambs can walk…if not cybercrime anymore,then this people who makes money of moneymules and botnets and all of those…they will do something else to stop cyber crime its dont stop nothing
need to work with problems not the results
90-s was more violance in eastern europe and eu and even in usa,and alot crime.couse tehconlogy wasnt so advanced in this times people made money more brutally now they have internet and computers if u take this away from them they will make something else and reason why? couse all eastern eusopean countries are f….d UP! so what they supposto do then? ….and nothing to loose couse if you dont have nothing then you dont have nothing to loose also people talk about always results…but why nobody dont talk about reasons? why something happening? and again what kind of crap is this like banks are in big trouble and stuff like that? they can make security measures for online banking better they can change if they want,,,its simple so this is just childsh what i see here.
and its interesting the olympic sponsor was lloyds tsb in uk so lloyds tsb got robbed most?? so were the hell lloyds got money?? lloyds should be broke as hell…now? but not? and bmw was also olympic sponsor…and we all know that carders and cybercriminals bought also bmw cars and they drive with bmw wich is not so important anyways all world and all banks and all finacial sytem is fraud anyways and its not holy,and what is not holy its means u can take s..t couse its all dirty and discusting stuff all banks and all law and everthign is just discusting and western countries all distcusng couse no moral and nothing its big mess and nothing else to talking about this make me feel sick in my stomack to see all this make me even more sick its all cartoon.its not real and banks are cartoon number 1 and all this world cyber security and virus protection its also cartoon illumination from western culture countries i but still western country people like all this…they dont wanna change nothing,they like high banking fees.they like unreasnble tax payments,and all that stuff,and soon they will be happy to take micro chips under skin,so this is the world were we live nothing pure and nothing real left its all scam,and fraud.and my question is why the crooks dont steal from russia or ukraine? those kind of countries?? why?? couse the people not can undesotod there waht going on,and thats why its not possible but why its possible in western countries?i think everyone knows the answer so this how the thing is
IF EVERYONE GIVE PROMESE in one day they will be honest and no fraud and no liang then we all will be honest and nobody dont steal and nothing,so until this time is not came jet lets all keep stealing liang and frauding each other until we dont have left to fraud noone,anymore and we start eat each other if the goverment will be with good moral then people will be too so each of us is just mirror of other person.thats it.MY ADVICE IS switch off all internet and banks and everthign and strt again couse all this is scrap and scam brothers we are point were we scam each other ether way…so is this is life??? and soon will be cyber securty so tight i cant even use google search anymore,i only will have options twitter and facebook,and only spefic viudeos in youtube.so this is the point were we are going to we are on the way to hell.
Concerned IT professionals [and maybe even Bit9 customers] might be interested to read about the “The Absolute Impossibility of White-listing”
http://blogs.bromium.com/2013/02/08/the-absolute-impossibility-of-white-listing/
I explicitly disagree with that post. The argument is overstated in order to tout their product over Bit9.
White-listing is a feasible approach if it’s done right. Many products don’t do it right. That doesn’t make it “impossible”.
White-listing is also not a panacea, merely one approach that can help.
Forgive me for not being super-technical, but Bit9’s explanation does not jibe with my understanding for how infections occur. Perhaps Mr. Krebs or someone here can help.
Most malware infections occur on the desktop which the attacker then uses as a launch point for reconnaissance and to move laterally (e.g., RSA breach, Google Breach, etc) so White-listing software on the server in question should not really be relevant to the discussion. Its very unlikely that the hacker group would know exactly which server to go after and then try to install an executable directly on it.
To me, this looks more like a longer-term resident infection (like RSA) where the attackers were able to identify the exact machine over time with the “crown jewels” and then exfiltrate/modify the data there.
If this is the case, then which desktop at Bit9 got popped and how did that happen if their software was installed on all their machines?
OR maybe I am missing something. Thoughts?
This story is so ridiculous, I can’t stop laughing. Really?? Bit9 didn’t follow its own protocols that it tells its customers to do? They deserved to get hacked! This should be a lesson for every company, federal agency, and business out there — make sure you use as many techniques as possible to keep your data as safe as possible. This snafu makes Bit9 look like a bunch of morons.
if this attack was targeted, how should their software help in any way?
i guess the hackers dont email their malware signatures to bit9 before they try to attack them 😉
That their software wasnt installed everywhere is just a very lame excuse. It would show how limited such software products are.
You cant tell your customers, that your snakeoil isnt working 😉
…..no matter how great your agent based security endpoint solutions are, they only protect what they can managed. If they had used an agent-less visibility and control solution like ForeScout’s CounterACT, they would have real-time detection of any non-compliant endpoints connecting to the network. They would have known that their whitelisting solution wasn’t deployed on the systems in question. Furthermore, they could then have auto-remediated the non-compliant endpoint at machine-speed….ohh well