Earlier this month I spoke at a cybersecurity conference in Albany, N.Y. alongside Tony Sager, senior vice president and chief evangelist at the Center for Internet Security and a former bug hunter at the U.S. National Security Agency. We talked at length about many issues, including supply chain security, and I asked Sager whether he’d heard anything about rumors that Supermicro — a high tech firm in San Jose, Calif. — had allegedly inserted hardware backdoors in technology sold to a number of American companies.
Tony Sager, senior vice president and chief evangelist at the Center for Internet Security.
The event Sager and I spoke at was prior to the publication of Bloomberg Businessweek‘s controversial story alleging that Supermicro had duped almost 30 companies into buying backdoored hardware. Sager said he hadn’t heard anything about Supermicro specifically, but we chatted at length about the challenges of policing the technology supply chain.
Below are some excerpts from our conversation. I learned quite bit, and I hope you will, too.
Brian Krebs (BK): Do you think Uncle Sam spends enough time focusing on the supply chain security problem? It seems like a pretty big threat, but also one that is really hard to counter.
Tony Sager (TS): The federal government has been worrying about this kind of problem for decades. In the 70s and 80s, the government was more dominant in the technology industry and didn’t have this massive internationalization of the technology supply chain.
But even then there were people who saw where this was all going, and there were some pretty big government programs to look into it.
BK: Right, the Trusted Foundry program I guess is a good example.
TS: Exactly. That was an attempt to help support a U.S.-based technology industry so that we had an indigenous place to work with, and where we have only cleared people and total control over the processes and parts.
BK: Why do you think more companies aren’t insisting on producing stuff through code and hardware foundries here in the U.S.?
TS: Like a lot of things in security, the economics always win. And eventually the cost differential for offshoring parts and labor overwhelmed attempts at managing that challenge.
BK: But certainly there are some areas of computer hardware and network design where you absolutely must have far greater integrity assurance?
TS: Right, and this is how they approach things at Sandia National Laboratories [one of three national nuclear security research and development laboratories]. One of the things they’ve looked at is this whole business of whether someone might sneak something into the design of a nuclear weapon.
The basic design principle has been to assume that one person in the process may have been subverted somehow, and the whole design philosophy is built around making sure that no one person gets to sign off on what goes into a particular process, and that there is never unobserved control over any one aspect of the system. So, there are a lot of technical and procedural controls there.
But the bottom line is that doing this is really much harder [for non-nuclear electronic components] because of all the offshoring now of electronic parts, as well as the software that runs on top of that hardware.
BK: So is the government basically only interested in supply chain security so long as it affects stuff they want to buy and use?
TS: The government still has regular meetings on supply chain risk management, but there are no easy answers to this problem. The technical ability to detect something wrong has been outpaced by the ability to do something about it.
BK: Wait…what?
TS: Suppose a nation state dominates a piece of technology and in theory could plant something inside of it. The attacker in this case has a risk model, too. Yes, he could put something in the circuitry or design, but his risk of exposure also goes up.
Could I as an attacker control components that go into certain designs or products? Sure, but it’s often not very clear what the target is for that product, or how you will guarantee it gets used by your target. And there are still a limited set of bad guys who can pull that stuff off. In the past, it’s been much more lucrative for the attacker to attack the supply chain on the distribution side, to go after targeted machines in targeted markets to lessen the exposure of this activity.
BK: So targeting your attack becomes problematic if you’re not really limiting the scope of targets that get hit with compromised hardware.
TS: Yes, you can put something into everything, but all of a sudden you have this massive big data collection problem on the back end where you as the attacker have created a different kind of analysis problem. Of course, some nations have more capability than others to sift through huge amounts of data they’re collecting.
BK: Can you talk about some of the things the government has typically done to figure out whether a given technology supplier might be trying to slip in a few compromised devices among an order of many?
TS: There’s this concept of the “blind buy,” where if you think the threat vector is someone gets into my supply chain and subverts the security of individual machines or groups of machines, the government figures out a way to purchase specific systems so that no one can target them. In other words, the seller doesn’t know it’s the government who’s buying it. This is a pretty standard technique to get past this, but it’s an ongoing cat and mouse game to be sure.
BK: I know you said before this interview that you weren’t prepared to comment on the specific claims in the recent Bloomberg article, but it does seem that supply chain attacks targeting cloud providers could be very attractive for an attacker. Can you talk about how the big cloud providers could mitigate the threat of incorporating factory-compromised hardware into their operations?
TS: It’s certainly a natural place to attack, but it’s also a complicated place to attack — particularly the very nature of the cloud, which is many tenants on one machine. If you’re attacking a target with on-premise technology, that’s pretty simple. But the purpose of the cloud is to abstract machines and make more efficient use of the same resources, so that there could be many users on a given machine. So how do you target that in a supply chain attack?
BK: Is there anything about the way these cloud-based companies operate….maybe just sheer scale…that makes them perhaps uniquely more resilient to supply chain attacks vis-a-vis companies in other industries?
TS: That’s a great question. The counter positive trend is that in order to get the kind of speed and scale that the Googles and Amazons and Microsofts of the world want and need, these companies are far less inclined now to just take off-the-shelf hardware and they’re actually now more inclined to build their own.
BK: Can you give some examples?
TS: There’s a fair amount of discussion among these cloud providers about commonalities — what parts of design could they cooperate on so there’s a marketplace for all of them to draw upon. And so we’re starting to see a real shift from off-the-shelf components to things that the service provider is either designing or pretty closely involved in the design, and so they can also build in security controls for that hardware. Now, if you’re counting on people to exactly implement designs, you have a different problem. But these are really complex technologies, so it’s non-trivial to insert backdoors. It gets harder and harder to hide those kinds of things.
BK: That’s interesting, given how much each of us have tied up in various cloud platforms. Are there other examples of how the cloud providers can make it harder for attackers who might seek to subvert their services through supply chain shenanigans?
TS: One factor is they’re rolling this technology out fairly regularly, and on top of that the shelf life of technology for these cloud providers is now a very small number of years. They all want faster, more efficient, powerful hardware, and a dynamic environment is much harder to attack. This actually turns out to be a very expensive problem for the attacker because it might have taken them a year to get that foothold, but in a lot of cases the short shelf life of this technology [with the cloud providers] is really raising the costs for the attackers.
When I looked at what Amazon and Google and Microsoft are pushing for it’s really a lot of horsepower going into the architecture and designs that support that service model, including the building in of more and more security right up front. Yes, they’re still making lots of use of non-U.S. made parts, but they’re really aware of that when they do. That doesn’t mean these kinds of supply chain attacks are impossible to pull off, but by the same token they don’t get easier with time.
BK: It seems to me that the majority of the government’s efforts to help secure the tech supply chain come in the form of looking for counterfeit products that might somehow wind up in tanks and ships and planes and cause problems there — as opposed to using that microscope to look at commercial technology. Do you think that’s accurate?
TS: I think that’s a fair characterization. It’s a logistical issue. This problem of counterfeits is a related problem. Transparency is one general design philosophy. Another is accountability and traceability back to a source. There’s this buzzphrase that if you can’t build in security then build in accountability. Basically the notion there was you often can’t build in the best or perfect security, but if you can build in accountability and traceability, that’s a pretty powerful deterrent as well as a necessary aid.
BK: For example….?
TS: Well, there’s this emphasis on high quality and unchangeable logging. If you can build strong accountability that if something goes wrong I can trace it back to who caused that, I can trace it back far enough to make the problem more technically difficult for the attacker. Once I know I can trace back the construction of a computer board to a certain place, you’ve built a different kind of security challenge for the attacker. So the notion there is while you may not be able to prevent every attack, this causes the attacker different kinds of difficulties, which is good news for the defense.
BK: So is supply chain security more of a physical security or cybersecurity problem?
TS: We like to think of this as we’re fighting in cyber all the time, but often that’s not true. If you can force attackers to subvert your supply chain, they you first off take away the mid-level criminal elements and you force the attackers to do things that are outside the cyber domain, such as set up front companies, bribe humans, etc. And in those domains — particularly the human dimension — we have other mechanisms that are detectors of activity there.
BK: What role does network monitoring play here? I’m hearing a lot right now from tech experts who say organizations should be able to detect supply chain compromises because at some point they should be able to see truckloads of data leaving their networks if they’re doing network monitoring right. What do you think about the role of effective network monitoring in fighting potential supply chain attacks.
TS: I’m not so optimistic about that. It’s too easy to hide. Monitoring is about finding anomalies, either in the volume or type of traffic you’d expect to see. It’s a hard problem category. For the US government, with perimeter monitoring there’s always a trade off in the ability to monitor traffic and the natural movement of the entire Internet towards encryption by default. So a lot of things we don’t get to touch because of tunneling and encryption, and the Department of Defense in particular has really struggled with this.
Now obviously what you can do is man-in-the-middle traffic with proxies and inspect everything there, and the perimeter of the network is ideally where you’d like to do that, but the speed and volume of the traffic is often just too great.
BK: Isn’t the government already doing this with the “trusted internet connections” or Einstein program, where they consolidate all this traffic at the gateways and try to inspect what’s going in and out?
TS: Yes, so they’re creating a highest volume, highest speed problem. To monitor that and to not interrupt traffic you have to have bleeding edge technology to do that, and then handle a ton of it which is already encrypted. If you’re going to try to proxy that, break it out, do the inspection and then re-encrypt the data, a lot of times that’s hard to keep up with technically and speed-wise.
BK: Does that mean it’s a waste of time to do this monitoring at the perimeter?
TS: No. The initial foothold by the attacker could have easily been via a legitimate tunnel and someone took over an account inside the enterprise. The real meaning of a particular stream of packets coming through the perimeter you may not know until that thing gets through and executes. So you can’t solve every problem at the perimeter. Some things only become obvious and make sense to catch them when they open up at the desktop.
BK: Do you see any parallels between the challenges of securing the supply chain and the challenges of getting companies to secure Internet of Things (IoT) devices so that they don’t continue to become a national security threat for just about any critical infrastructure, such as with DDoS attacks like we’ve seen over the past few years?
TS: Absolutely, and again the economics of security are so compelling. With IoT we have the cheapest possible parts, devices with a relatively short life span and it’s interesting to hear people talking about regulation around IoT. But a lot of the discussion I’ve heard recently does not revolve around top-down solutions but more like how do we learn from places like the Food and Drug Administration about certification of medical devices. In other words, are there known characteristics that we would like to see these devices put through before they become in some generic sense safe.
BK: How much of addressing the IoT and supply chain problems is about being able to look at the code that powers the hardware and finding the vulnerabilities there? Where does accountability come in?
TS: I used to look at other peoples’ software for a living and find zero-day bugs. What I realized was that our ability to find things as human beings with limited technology was never going to solve the problem. The deterrent effect that people believed someone was inspecting their software usually got more positive results than the actual looking. If they were going to make a mistake – deliberately or otherwise — they would have to work hard at it and if there was some method of transparency, us finding the one or two and making a big deal of it when we did was often enough of a deterrent.
BK: Sounds like an approach that would work well to help us feel better about the security and code inside of these election machines that have become the subject of so much intense scrutiny of late.
TS: We’re definitely going through this now in thinking about the election devices. We’re kind of going through this classic argument where hackers are carrying the noble flag of truth and vendors are hunkering down on liability. So some of the vendors seem willing to do something different, but at the same time they’re kind of trapped now by the good intentions of open vulnerability community.
The question is, how do we bring some level of transparency to the process, but probably short of vendors exposing their trade secrets and the code to the world? What is it that they can demonstrate in terms of cost effectiveness of development practices to scrub out some of the problems before they get out there. This is important, because elections need one outcome: Public confidence in the outcome. And of course, one way to do that is through greater transparency.
BK: What, if anything, are the takeaways for the average user here? With the proliferation of IoT devices in consumer homes, is there any hope that we’ll see more tools that help people gain more control over how these systems are behaving on the local network?
TS: Most of [the supply chain problem] is outside the individual’s ability to do anything about, and beyond ability of small businesses to grapple with this. It’s in fact outside of the autonomy of the average company to figure it out. We do need more national focus on the problem.
It’s now almost impossible to for consumers to buy electronics stuff that isn’t Internet-connected. The chipsets are so cheap and the ability for every device to have its own Wi-Fi chip built in means that [manufacturers] are adding them whether it makes sense to or not. I think we’ll see more security coming into the marketplace to manage devices. So for example you might define rules that say appliances can talk to the manufacturer only.
We’re going to see more easy-to-use tools available to consumers to help manage all these devices. We’re starting to see the fight for dominance in this space already at the home gateway and network management level. As these devices get more numerous and complicated, there will be more consumer oriented ways to manage them. Some of the broadband providers already offer services that will tell what devices are operating in your home and let users control when those various devices are allowed to talk to the Internet.
Since Bloomberg’s story broke, The U.S. Department of Homeland Security and the National Cyber Security Centre, a unit of Britain’s eavesdropping agency, GCHQ, both came out with statements saying they had no reason to doubt vehement denials by Amazon and Apple that they were affected by any incidents involving Supermicro’s supply chain security. Apple also penned a strongly-worded letter to lawmakers denying claims in the story.
Meanwhile, Bloomberg reporters published a follow-up story citing new, on-the-record evidence to back up claims made in their original story.
Very Interesting article…..
Fascinating – thank you for sharing.
Wonderful insights – Tony’s comments went in a bunch of directions I’d really not considered.
One thing I was (a little) concerned with around the denials from Amazon, Apple et al was the sheer speed with which they were made. I’m taken back to the old quote from Shakespeare (in Hamlet)… “methinks the lady doth protest too much.”
It shouldn’t be surprising that they denied buying flawed products from China, given their potential exposure to shareholder doubts and anger, market ridicule, and huge government contracts hanging in the balance.
Further, their corporate structures and shell games allow them to plausibly deny doing things with one hand, that are done with another hand.
They quickly lie with abandon, knowing there are no consequences.
Nice update on that point but I want to know CIA and apply Is denied for so I am not able to understand that is possible or not
Thanks for another excellent piece, Brian. It will be ‘interesting’ to see how this continues to unfold. As Han Solo might say, “I’ve got a bad feeling about this.”
Hope you had a pleasant trip to the Capitol Region — and missed the heavy weather (use to live in the North Country, about four hours further up).
“I find your lack of faith… disturbing” 🙂
It isn’t just a technical problem. If I had a target who had information I wanted, I could zoom in on all of that person’s communications and that of those around them. Then I could put pieces together.
Unless they are very careful in all spheres, it might be way more cost effective to look for ordinary human mistakes than try and sort out what is coming out of the internet fire hose.
You could intercept my communications for my whole life and not gain anything that might be of any use except maybe to steal money from me, but I have no information worth stealing.
You have to have the Sandia approach as well and audit employees so at least obvious blunders can be corrected.
KB: “How can we identify the attacker?”
Fox: “Anyone who wants to get inside the henhouse qualifies as an attacker.”
Invoke (“attacker”|”attacker such as NSA”).
“An attacker such as NSA has a risk model, too. Yes, he could put something in the circuitry or design, but his risk of exposure also goes up.”
“Could an attacker such as NSA control components that go into certain designs or products? Sure.”
So insight. Much enlightenment.
You wouldn’t be thinking of the possibly back-doored NSA elliptical curve algorithms that were being promoted by NIST, would you?
You started this publication stating that your interview was before the publishing of the Bloomberg article. But then you asked a question prefaced with, “I know you said before this interview that you weren’t prepared to comment on the specific claims in the recent Bloomberg article, but…” So you both knew about the Bloomberg article before its publishing?!
You’re right. The symposium was September 26.
The Bloomberg article was October 4.
Maybe they had access to an advanced copy of the article.
Or they’re time travelers.
They’re time travelers. That’s how you commented on October 14th on an article published October 18th, while it’s currently October 16th as I’m reading both. 😉
Lol, you almost had me.
The article was published on the twelfth of October, 2018. KOS formats that as 12 Oct 18.
You ask questions like someone who really knows what he’s doing. I’m impressed. 🙂
And when TS went into the IOT stuff, I recalled similar concerns raised by Bruce Schneier at a recent discussion I saw on C-Span from a book launch at the Aspen Institute.
https://www.aspeninstitute.org/events/book-launch-click-here-to-kill-everybody/
https://www.c-span.org/video/?451126-1/click-kill-everybody
Interesting article. Just as I’ve said. Except, for noticing iot. All known ports can and should be known to be comprised. And daily, new communications ports are discovered. And machines are made up of parts, creating a feature. That feature may not be doing what you want. How do you stop it from doing its job. Unless you find it.
“… prior to the publication of Bloomberg Businessweek‘s controversial story alleging that Supermicro had duped almost 30 companies into buying backdoored hardware.”
That accusation is stronger than what Bloomberg alleged in their article.
Has there been any evidence published indicating that Supermicro intentionally did what you allege?
Very enlightening interview, nevertheless.
There are state actors, who exploit the weakness in suply supply chain security; be that home grown or foreign:
https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-factory-show-cisco-router-getting-implant/
The interview vaguely mention software based implant, that easier to pull off, much more prevalent and more cost effective. This also the domain of the state actors and in addition, hardware/software companies utilize this venue as well. Most people probably recall Sony, Lenovo, Dell, etc., system came and they still probably do some type of monitoring/remote control “solutions” that seldom has any usefulness for the end users. Certainly, the name of these solutions changed to “telemetry”, but it does the same thing. It would be hard to find any software currently on the market that does not have “telemetry” functionality. Operating systems, apps and even drivers nowadays have this “feature” built-in an in some case, it is impossible to disable. I am looking at you Windows 10…
The question is, do the companies share the collected data with state actors? I leave it up to you to answer this question…
Great Article! Say, I wonder if new rules & regulation will be coming with the new popular “How to do it” on Instructables website: Article called “Sonic Screw Driver” where you remove you chip from your bank card and put it in a device called a Sonic Screw Driver!
While working in the mil-spec semiconductor & hybrid electronics industry in the 90’s I experienced first-hand the transition of military/government programs from dedicated design and manufacture of components & devices to more & more off-the-shelf components, primarily for program cost reductions. In most all cases the o-t-s components maintained the performance and reliability that had been required from the dedicated MIL-STD components. However, the reduction in military budgets during that period was certainly the catalyst for transition, as the manufacturers actually saw reduced incomes and profits as a result.
I find it interesting that Tony Sager is saying, “. . . we’re starting to see a real shift from off-the-shelf components to things that the service provider is either designing or pretty closely involved in the design, and so they can also build in security controls for that hardware.” This is exactly the approach the US Military & high tech government programs i.e. NASA used to operate under prior to the 1990s.
It appears the difference is the how the government & private enterprise operate, funding sources: taxes versus profit; and how decisions are made: political expediencies versus long term growth and profit.
Re: Bloomberg followup-
What ‘new, on-the-record evidence to back up claims made in their original story.’
I see nothing new .. well maybe metal sides instead of plastic [to get rid of excess heat] on the ethereal net connectors.. but it all seems so circumstantial, nothing specific – like what sites are contacted to deliver what data.
Sorry. ethereal was my little joke – a little levity in this time of great gravity.
As every device (at least every pci one, don’t know about usb) can come with its own drivers (it emulates an usb chipset connected to an memory stick containing signed drivers for this purpose) and can extend the uefi perhaps wouldn’t require additional chips for compromising systems. It might be sufficient to alter the device’s firmware.
If that isn’t possible: a fake a-few-megabytes-long driver I am usb controller connected to a spare pair of pci or usb lines that look as they were connected to one of several dozens of internal peripherals? Could be way smaller than a transflash card and might even be soldered to pads that were planned for debug purposes or things this specific server wouldn’t need but a future version might.
For less than the price of a Tank, Submarine, or Missile; we could make improvements. The key is Finding and Training talent who can work through these problems. I can see the odd traffic going out daily, at many levels.
Thanks – very frightening to think that in some very critical industries – energy, healthcare, finance, etc – we could already, or very soon, be compromised and not know it.
I have to wonder if those vehement denials regarding the Bloomberg article are the result of essentially, they were told by a 3 letter agency that this is classified, you can not talk about this or even admit to it publicly without federal charges, and furthermore must deny any public reports of such as well?
America is like the Death Star. fully protected from massive, big attacks but very susceptible to small, X-Wing attacks.
So any news on indicators of compromise yet? From the second Bloomberg article they seem to think that there is a third modification, in the ethernet port itself. Something about metal sides instead of plastic because of heat issues. Does this mean we can just take IR photos and see the port is warmer than it should be? There is a lot of heat on these systems because they are cluster nodes or servers so extra heat on an ethernet port might be masked by the general high temperatures at the back of the nodes.
So any word on IOC would be nice to see.
So Bloomberg says they’ve got proof and then offer the testimony of one man who provides nothing other than his credentials and experience in the way of evidence. They site the fact that a government has begun looking into the issue only to say that the reason they were is because of their article. Umm … I’m not even sure how much smoke there is here.
Sure, it’s possible. Sure, it’s potentially very damaging. Thing is, let’s see some proof. Not so-and-so says they saw a thing when some guy showed them a thing proof, let’s see someone actually identify the rogue component on an allegedly compromised system.
This is where I’m at. I don’t want to dismiss it out of hand because, if it’s true, it’s a vitally serious problem that absolutely demands attention. At the same time, where’s the corroborating evidence? Where are the network traffic indicators of compromise, for example? I’ve yet to see any IoCs released, and yes, that’s a big deal. Just because you can’t decrypt the payload doesn’t mean you can’t tell it’s out of the ordinary (with due nod to Sager here in regards to his accurate statement about the difficulty in identifying the traffic. Yeah, it’s hard. But it’s not impossible). Alternately, where are excerpts of the implant code that was on the chip? That’d help illuminate the issue too. But I’ve not seen any.
Sure, there are explanations for this that can encompass both that the supply chain was infiltrated and that there is still no other effects. It could be that there simply hasn’t yet been any attempt to use these capabilities, therefore there are no IoCs to be noticed to begin with. Or that this was simply a feasibility experiment to see what it takes to sneak such hardware into the chain. It’s possible for this to all be true and yet see no evidence in the wild.
But that doesn’t change the fact that there’s only so much that’s actionable in all this. Sure, companies can go to their datacenters and open their boxes up to check for such unaccounted hardware… if they had documentation telling them what to look for. And what happens to places that depend on co-hosting? You can’t quite walk into an AWS datacenter and demand that a machine be taken apart for you to look at. It’ll be hard to walk into even small and mid sized co-hosting companies and do that. Yes, threat hunters could go looking for the traffic. But what if they have, and haven’t found anything? Small numbers of orgs not finding certain malicious traffic is not a surprise; no one finding it is. And if anyone has, it’s not been published.
This whole issue is something to note, but there needs to be a LOT more substance before individual orgs can take action.
This is how the Death Star was compromised. That, and an unencrypted cartridge in a tape library with poor access controls.
That is a great point. You win the Internet today.
Scarif’s ISO 27001 audit noted poor compliance with at rest encryption requirements, but they thought it was more than compensated for with robust physical security. 😉
Thing is, Tarkin buried the report. What was the Senate going to do about that? Complain to the Emperor? Besides, the auditors got visited by Vader, and well, we know what happened afterwards. Let’s just say that things deteriorated to the point where a mob of mere Bothans with the Red Team ability of a Gamorrean were able to exfil stuff on the next project. The Empire’s data loss prevention unit couldn’t quite shoot all of them.
😀
You, I like you. Well played.
“What do you think about the role of effective network monitoring in fighting potential supply chain attacks.
TS: I’m not so optimistic about that. It’s too easy to hide.”
While I’m every bit in agreement with Sager about the technical difficulties, I fear he’s surrendering on this point too easily. Yes, network monitoring on a national scale is not feasible. But I’d argue that doing so per organization or company is actually necessary. And doable, although again, it sure as hell isn’t easy.
Yes, you won’t get much insight into the data itself – yes, as he noted, encrypting payloads is trivial and in fact expected for much normal traffic – but you still get information about where the external C&C traffic comes from (if it’s necessary for these types of attacks… I honestly don’t know the capabilities of these mysterious chips mentioned in the Bloomberg article), from where the bad traffic originates within your network, what it talks to and what else it tries to subvert, where it goes once it’s in the exfiltration stage… a lot of it is meta, and again, it’s very much the needle in a field of haystacks problem, but it’s **doable** if the scope is per organization. And not only does nothing stop different companies and orgs from talking to each other, but framesworks and groups are already in place for exactly that information sharing. That’s precisely what the ISACs are for, for example (https://www.nationalisacs.org/).
None of this is easy because network defense isn’t easy. But I’m a bit disappointed that Sager said what he did. He obviously knows what’s happening in that area; there’s no way he doesn’t given the position he’s in. It’s hard, and one of the bad things about it is that you’re always playing catch up to what the intruders are doing, but again, it’s doable. It definitely helps in the accountability stage that he discussed, but it can also help in stopping exfils in progress when they’re found in time. From my perspective, that area shouldn’t be written off so quickly.
Just to be clear: I agree with Sager on everything else. Plus, I’d bet we have 90% overlap on the particulars of network defense and threat hunting. The “highest volume, highest speed” problem definitely exists, even at levels below nation-state, government, and multinational corporations. There’s no question that it’s a difficult problem, as well as an expensive one. And all of that still presumes that you’re talking about a network infrastructure controlled centrally by a single company or organization. When you’re not dealing with that, when you have a distributed infrastructure, plans get shot all to hell. So I do agree that it’s really hard. My only point above was that there’s still good work being done there, there’s still evolution occuring with the tools to handle that aspect of the threat, and there’s no reason to just dismiss it as “too easy” to get around. That’s all I’m getting at.
TS was right. It is easy to hide.
Think of the Target breach, in which credit card data was stolen by thieves. Network monitoring didn’t catch this, because the thieves used an authorized account for an HVAC contractor.
Sony Pictures had 100 terabytes of data stolen over two months, including full length movies. The thieves were not caught during those two months, despite network monitoring, because the volume of data wasn’t unusual for a company like this and the thieves used a worm to hide their access point.
Network monitoring can be an effective analysis tool, but, by definition, it’s always focused on the past. (Watch a bullet leave a gun. By the time you’ve observed it, it’s already reached its target. It’s happened. It’s in the past.) Even the fastest analysis will always be a step behind.
Where I think TS faltered is in focusing on how to improve the security for IOT and internet-enabled machines. It can’t be improved; the Internet wasn’t built to be secure.
The internet was built on trust and cooperation. Bad people will always be able to use that against the rest of us.
TS should’ve instead suggested we unplug whatever doesn’t need to be connected.
I’ve been following this story very closely and it’s incredibly interesting. I’m very curious why Apple and Amazon denied the Bloomberg report so vehemently. I’m just not sure who to believe.
Aside from that it’s clear the US should have some sort of domestic hardware manufacturers just in case.
On this topic, how about when a company cannot give you documentation on what ports their products listen on or are using? How many folks are not asking these questions, not doing vulnerability assessments or security reviews, and just blindly using equipment they order? Does it concern anyone else that just about every new product has a “phone home” option that we’re expected to allow blind communications to just flow out of our perimeter firewalls? I’m not talking IoT, I’m talking enterprise-class networking devices and equipment.
The conversation has me looking at the problem in a way I had not put much thought to in the past. Reading posts like this let me learn new things, not every day but often.
At the very least, the ethernet port chip should enable quick methods to kill switch the connectivity. If you depend on clustered computing, this would be a good way of degrading the service.
Very informative article. Thank you.
The sheer scale of what/where/how (security) breaches can happen is mind boggling … the possibilities where these type of chips can be hidden is so enormous that it is nearly impossible to really find them or even ever to discover them.
I look after the IT of a smaller company. Sometimes I do some research of the traffic that leaves/comes into our border router. For example I have turned of every computer/device but the servers observing the traffic IN/OUT. Then I turn on 1 workstation (Windows 7/10) and check what happens. As soon as the workstation is running a barrage of traffic leaves/comes into the border router. Some IP addresses have no reverse look up but then a “jwhois” reveals it’s a well known software manufacturer (Adobe, Microsoft, Google, Mozilla etc) but the traffic is still encrypted – I have to TRUST them they are doing the correct thing. Hardware manufacturers are among them (video,network,raid,sound,etc) with traffic encrypted – same deal again no reverse lookup referring to some know entity but jwhois mostly reveals a know supplier – again I have to TRUST them they are doing the correct thing.
The above is without someone on the station, if it is an active station it is impossible to even discover whether there is some nasty traffic in there … just think of ads, libraries, images, webpages, software updates! Then think of when everyone is working … like 300 packets of different protocols/ports a second.
That is without the IoT’s, but I have already learned a lesson, no IoT traffic can leave/enter the DMZ – also I do updates manually.
I take my hat off and take a deep bow to the guys at AWS that they actually discovered some tiny data packets coming out of those servers that should not have been there – that is some forensics! Wow!
I am sure that people in many different places all over the world now have a different perspective moving forward …