Amid the hustle and bustle of the RSA Security Conference in San Francisco last week, researchers at RSA released a startling report that received very little press coverage relative to its overall importance. The report detailed a malware campaign that piggybacked on a popular piece of software used by system administrators at some of the nation’s largest companies. Incredibly, the report did not name the affected software, and the vendor in question has apparently chosen to bury its breach disclosure. This post is an attempt to remedy that.
The RSA report detailed the threat from a malware operation the company dubbed “Kingslayer.” According to RSA, the attackers compromised the Web site of a company that sells software to help Windows system administrators better parse and understand Windows event logs. RSA said the site hosting the event log management software was only compromised for two weeks — from April 9, 2015 to April 25, 2015 — but that the intrusion was likely far more severe than the short duration of the intrusion suggests.
That’s because in addition to compromising the download page for this software package, the attackers also hacked the company’s software update server, meaning any company that already had the software installed prior to the site compromise would likely have automatically downloaded the compromised version when the software regularly checked for available updates (as it was designed to do).
Image: RSA
RSA said that in April 2016 it “sinkholed” or took control over the Web site that the malware used as a control server — oraclesoft[dot]net — and from there they were able to see indicators of which organizations might still be running the backdoored software. According to RSA, the victims included five major defense contractors; four major telecommunications providers; 10+ western military organizations; more than two dozen Fortune 500 companies; 24 banks and financial institutions; and at least 45 higher educational institutions.
RSA declined to name the software vendor whose site was compromised, but said the company issued a security notification on its Web site on June 30, 2016 and updated the notice on July 17, 2016 at RSA’s request following findings from further investigation into a defense contractor’s network. RSA also noted that the victim software firm had a domain name ending in “.net,” and that the product in question was installed as a Windows installer package file (.msi).
Using that information, it wasn’t super difficult to find the product in question. An Internet search for the terms “event log security notification april 2015” turns up a breach notification from June 30, 2016 about a software package called EVlog, produced by an Altair Technologies Ltd. in Mississauga, Ontario. The timeline mentioned in the breach notification exactly matches the timeline laid out in the RSA report.
As far as breach disclosures go, this one is about the lamest I’ve ever seen given the sheer number of companies that Altair Technologies lists on its site as subscribers to eventid.net, an online service tied to EVlog. I could not locate a single link to this advisory anywhere on the company’s site, nor could I find evidence that Altair Technologies had made any effort via social media or elsewhere to call attention to the security advisory; it is simply buried in the site. A screenshot of the original, much shorter, version of that notice is here.
Just some of the customers of Eventid.
Perhaps the company emailed its subscribers about the breach, but that seems doubtful. The owner of Altair Technologies, a programmer named Adrian Grigorof, did not respond to multiple requests for comment.
“This attack is unique in that it appears to have specifically targeted Windows system administrators of large and, perhaps, sensitive organizations,” RSA said in its report. “These organizations appeared on a list of customers still displayed on the formerly subverted software vendor’s Web site. This is likely not coincidence, but unfortunately, nearly two years after the Kingslayer campaign was initiated, we still do not know how many of the customers listed on the website may have been breached, or possibly are still compromised by the Kingslayer perpetrators.”
It’s perhaps worth noting that this isn’t the only software package sold by Altair Technologies. An analysis of Eventid.net shows that the site is hosted on a server along with three other domains, eventreader.com, firegen.com and grigorof.com (the latter being a vanity domain of the software developer). The other two domains — eventreader.com and firegen.com — correspond to different software products sold by Altair.
The fact that those software titles appear to have been sold and downloadable from the same server as eventid.net (going back as far as 2010) suggests that those products may have been similarly compromised. However, I could find no breach notification mentioning those products. Here is a list of companies that Altair says are customers of Firegen; they include 3M, DirecTV, Dole Food Company, EDS, FedEx, Ingram Micro, Northrop Grumman, Symantec and the U.S. Marshals Service.
RSA calls these types of intrusions “supply chain attacks,” in that they provide one compromise vector to multiple targets. It’s not difficult to see from the customer lists of the software titles mentioned above why an attacker might salivate over the idea of hacking an entire suite of software designed for corporate system administrators.
“Supply chain exploitation attacks, by their very nature, are stealthy and have the potential to provide the attacker access to their targets for a much longer period than malware delivered by other common means, by evading traditional network analysis and detection tools,” wrote RSA’s Kent Backman and Kevin Stear. “Software supply chain attacks offer considerable ‘bang for the buck’ against otherwise hardened targets. In the case of Kingslayer, this especially rings true because the specific system-administrator-related systems most likely to be infected offer the ideal beachhead and operational staging environment for system exploitation of a large enterprise.”
A copy of the RSA report is available here (PDF).
Update, 3:35 p.m. ET: I first contacted Altair Technologies’ Grigorof on Feb. 9. I heard back from him today, post-publication. Here is his statement:
“Rest assured that the EvLog incident has been reviewed by a high-level security research company and the relevant information circulated to the interested parties, including antivirus companies. We are under an NDA regarding their internal research though the attack has already been categorized as a supply chain attack.”
“The notification that you’ve seen was based on their recommendations and they had our full cooperation on tracking down the perpetrators. It’s obviously not as spectacular as a high visibility, major company breach and surely there wasn’t anything in the news – we are not that famous.”
“I’m sure a DDoS against our site would remain unnoticed while the attack against your blog site made headlines all over the world. We also don’t expect that a large organization would use EvLog to monitor their servers – it is a very simple tool. We identified the problem within a couple of weeks (vs months or years that it takes for a typical breach) and imposed several layers of extra security in order prevent this type of problem.”
“To answer your direct question about notifications, we don’t keep track on who downloads and tries this software, therefore there is no master list of users to notify. Any anonymous user can download it and install it. I’m not sure what you mean by ‘you still haven’t disclosed this breach’ – it is obviously disclosed and the notification is on our website. The notification is quite explicit in my opinion – the user is warned that even if EvLog is removed, there may still be other malware that used EvLog as a bridgehead.”
My take on this statement? I find it to be wholly incongruent. Altair Technologies obviously went to great lengths to publish who its major customers were on the same sites it was using to the sell the software in question. Now the owner says he has no idea who uses the software. That he would say it was never intended to be used in major organizations seems odd in contrast.
Finally, publishing a statement somewhere in the thick of your site and not calling attention to it on any other portion of your site isn’t exactly a disclosure. If nobody knows that there’s a breach notice there to find, how should they be expected to find it? Answer: They’re not, because it wasn’t intended for that purpose. This statement hasn’t convinced me to the contrary.
Update, 11:13 p.m. ET: Altair Technologies now has a link to the breach notification on the homepage for Evlog: http://www.eventid.net/evlog/
Great article.
Nasty stuff. I hope we see a mass exodus from their software if there are any alternatives.
The most obvious choice would be Splunk.
It used to be a great product but has become expensive bloatware, over time.
Wow!
Just, WOW!
I would think this would be front page news.
There should be some consequences for the vendor.
There certainly will be for the rest of us.
Shame on RSA, kudos to Brian!
No shame on RSA – they were reporting on what occurred, and work hard to assist on victim notification and remediation.
None at all. I know RSA worked with a number of organizations to alert apparent victims independently. It says so in the report. My beef was with the software vendor.
Thanks Brian – agreed! And, great work – as always.
Why shame on RSA? More like shame on this company for not looking after their customers. RSA brought light to the situation.
Found the notice on their site under the “Latest Updates” for June 30, 2016. Title is: IMPORTANT: EvLog Security Notitification
http://www.eventid.net/evlog/evlogsecnote.asp
Still not very obvious without knowing to look for a notice.
That notice is linked in the story. However the notice itself doesn’t appear to be linked to from anywhere else on the eventid.net site. I checked archive.org’s cached versions of the site back around the time this notice was published and couldn’t find a link or reference there either.
Whoops, you are correct, sorry about that! Somehow missed the link you provided to the notification found by search engine.
Considering what is written in the second paragraph of the security notification, it is indeed shocking that this wasn’t bigger news.
It appears, in my mind, that you’re saying Altair Technologies’s posted statement of 6/30/2016 was added sometime substantially AFTER the fact. Is that what you’re stating? If so, that really is egregious!
Nope, didn’t say that in the story at all. Not sure how you arrived at that conclusion.
I arrived at that erroneous conclusion because I thought you stated it didn’t “appear to be linked to from anywhere else on the eventid.net site”, thus my jump to, if it wasn’t chronologically listed maybe it was added after the fact. I guess I was trying to make more drama where there wasn’t any. My sincere apologies.
Thoroughly enjoyed the article and the feeling of being in the know with your blogs. Thank you. Well done!
Today it’s linked from the page at http://www.eventid.net/evlog/
but note that the link text there has a typo in “Notitification” which may mislead searching for it:
Jun 30, 2016
IMPORTANT: EvLog Security Notitification
Brian, again you put he OOPS in SCOOPS. Thank you for your efforts. Is this Altair Technologies a real company with employees or just a one man show? What kind of vendor management do these companies deploy????
They certainly don’t keep that list of subscribers current. A number of those banks are long gone:
Bank of New York – to Chase and Mellon in 2006.
Bank of Sacramento – to AmericanWest Bank in 2015.
Chart Bank – to Benjamin Franklin Bancorp in 2004.
Commercial Bank of Florida – to Colonial Bank in 2007.
Fortis Bank – to BNP Paribas in 2008.
…and so on. One could imagine how well the rest of the website might be maintained.
BoNY is now BoNY Mellon so they still use that name.
Anytime I see a vendor website still using .ASP instead of .ASPX I dig further for the reason you mentioned; it makes us wonder how well they would protect our data.
Yeah, like I just said.
https://www.ssllabs.com/ssltest/analyze.html?d=www.eventid.net
HTTP server signature Microsoft-IIS/6.0
Windows Server 2003 on the Internet almost two years after it became unpatchable because it went EOL.
But it’s got a SHA256 cert on it so we’re good, right?
I guess this was one of your easiest disclosures. I did listen to the talk and most of the people were guessing other vendor solutions. Great blog!
Great article – more light needs to be shed across the board on these supply chain attacks to make organizations aware of how to do appropriate vetting of their vendors.
The title is misleading, however, because you don’t include any instructions for how to avoid getting the spotlight of this blog shown upon you.
How *Not to Bury a Major Breach Notification
Thanks for this excellent post–I’m wondering if you might clarify what RSA stands to lose or gain by keeping this hush? I read their white paper; seems they have at least some business interest in getting the word out. (Right?)
My understanding from speaking with the authors of the report was that in exchange for a guarantee that the company would disclose this issue in some way, RSA agreed not to mention the company in its report.
That makes sense. Thanks! This is very interesting. I’m studying for certification exams and I feel like your posts could be standalone case studies. Although I am sorry you’ve had to deal with perps IRL. I’m thankful for your work.
Great story. All transactions need to be validated whether they be software updates, site, changes, payments, communications etc. If validation doesn’t occur for each and every transaction it leaves the network open to exploitation. As this story reveals there’s a long way to go and a lot of work to be done to close the gaps even for those who would be viewed as having the most expertise.
Brian, as always you show the benefit to the world of an active investigative press that has integrity. All companies and individual users of the web owe you a debt. I am glad Google has indirectly sponsored your site (by providing protection from DDOS). I hope other major internet companies are supporting you in other ways. I hope more individuals will support you according the their fiscal abilities.
This also applies to all active investigative journalists with integrity that today must report on their own because the news organizations are going away from investigative journalism.
One quick update: when you go to the Altair Technologies “contact me” page, you get an address in the Bahamas: http://www.altairtech.ca/contact.asp.
JR
According to LinkedIn Adrian Grigorof isn’t the owner of Altair since January 2016.
How did the badguy compromise Altair?
Great post. Would be nice to have social sharing buttons!
Just wondering why ‘owning’ a domain using ones name is considered vanity. Seems prudent to me. 🙂
Vanity domain is an expression, similar to vanity license plates. It’s not a judgement of any kind.
Brian,
Another great scoop! I wonder if there is any reliable data out there on the number of people who are affected by malware from compromised sites/products between discovery and patch, or mitigation, in cases where there is no disclosure?
I am the owner of the site that Brian is highlighting in his blog entry. He sent me an email on Feb 9 asking for details about our security notice. I was preparing an answer, though RSA had not yet released their article and I was under NDA with them and I had to think about can be disclosed. This morning I received another email from Brian: “contact me today or else” – I was adding more info to my response for his Feb 9 email and I did a quick check to see if RSA made their whitepaper public (it wasn’t a few days ago) when I found that Brian went ahead and published this, though I think it is still “today”. I doubt that any answer would’ve made any difference.
We worked with RSA and provided a relevant part of the information in their Kingslayer whitepaper . I’ve been in contact with the author and discussed many aspects of the attack and its aftermath. We didn’t make any “deal” with RSA, they asked us to sign an NDA about their research and volunteered not to mention the company name though anyone can easily find it by searching some of the details in their document (and this blog post is living proof). I was asked to review the whitepaper before being published and I had no problem with it – what happened, happened. If Brian did talk to the authors, he didn’t mention that we fully cooperated with RSA and did all that’s been asked from us (but that would’ve been against the spirit of this blog post). The notification on the site is what RSA recommended. We don’t keep a list of EvLog users, anyone can download it. It is easy for a bank, for a social site, etc. to identify their users. Not so easy when your software is free to download. How many of us are using Linux and when is the last direct email that we received about a security problem with it?
Other software is mentioned in the blog as “potentially” compromised. It was not compromised – is there a notification? No. This was an attack strictly directed at EvLog from what RSA estimated to be a state-sponsored threat actor.
Of course we were not perfect in handling this. In hindsight is much easier to criticize. Should we now start plastering our sites with pop-ups about EvLog being the victim of an attack in 2015? Brian thinks that unless you do this, you are trying to “bury” an attack. Maybe he is right, maybe not. I don’t see any notices on Yahoo’s main page, on Target, on government sites, etc. RSA itself was breached – can anyone navigate to a security notice from their main page? Unless you heard about it and Google it you cannot find any notification. Is there a complain about this? On our site, the security notice is surely not buried. It is on the EvLog home page with an IMPORTANT label next to it. We didn’t have to do this.
Whoever feels like throwing the first stone, good for you, you are a better company. We are still learning from our mistakes.
If you are the real man, I am glad you came here. I think you will find that this is actually a good community to come to, despite your thoughts on how the publishing of the article played out. I hope you read my comment below yours to see where I was coming from prior to seeing this.
If it is indeed true, that the software was completely free to use, and there was never discussions with these agencies/organizations about licencing, sales, or support, then I stand by those previous comments of mine. While I believe there should be certain expectations from vendors, our concern should be with those companies using a free, non-enterprise software, carelessly.
I hope you stay to answer anything you can, this is a good community that could benefit greatly from hearing you out.
I will assume you are the real Mr. Grigorof.
Thanks for coming forward. I applaud two of your statements. The first where you admit that you were not perfect in handling this. The second where you state you are still learning from your mistakes.
In my experience those who live by those statements develop very high levels of expertise. I find it difficult to live by them consistently and wish you the best in doing to.
@Adrian Grigorof
“… I don’t see any notices on Yahoo’s main page…”
Just to set the record straight: Yahoo has had a breach notice on their login page describing the December 2016 breach ever since they first went public with it. Before that, there was a similar notice on their login page describing the September 2016 breach.
Kinda hard to miss. So I guess you must not have a Yahoo account. Or you don’t use it.
Brian – Can we get a response to Mr. Grigorof’s statement?
Thanks
Michael
My response is already in the story.
Very good to see your post.
Thank you
If I understand the article correctly, you have an update mechanism built into your software. Of course hindsight is perfect. Would it be possible to notify your users upon subsequent updates?
For instance, in the past I have updated software and during the first run after the update there will be a screen pop, or a text file loaded that describes the change log, or some such information pertinent to the update. Also, I have seen a notification area in the GUI of the software that will scroll across with an urgent message. Although this is less common now.
I’m not familiar with this software, but the problem is that you’re thinking in terms of end user software with a graphical user interface. This software sounds more like automated log filtering and reporting. There’s no real user interface and there’s essentially an expected output. Which could reasonably be expected to be uniform and which an administrator might be processing with another script. If you change the output to include some sort of non standard notice, you could break those scripts.
As an administrator, I’d be fairly upset if software I used did that.
General practice is for software like this to have an announce list which normally announces updates, but which can be borrowed to announce urgent security updates. By convention, administrators are responsible for monitoring such lists. We obviously can’t do that if such a list doesn’t exist.
“Target Breach” Google search yields:
https://corporate.target.com/about/shopping-experience/payment-card-issue-faq
Entitled: “Data Breach FAQ”
“Answers to commonly asked questions for guests impacted by the recent data breach.”
Just sayin. . .
The question would then be has the malware propagated down the supply chain in as yet undiscovered ways? Has it opened targets downstream in the supply chain to other malware? Clients are using equipment or software from Dell, Cisco, Intuit, just to name a few. Having read the PDF from RSA it appears that this installed a password dump executable. But if it grabbed the URL for the customer database, the user name, and the database password, how far did it go?
Those customers of Altair are big name companies that every mid-range and above enterprise is going to have had contact with…
That is my concern as well. There seems to be no doubt left that this is indeed state-sponsored. What that tells me is that what you said could very well be the case. God only knows how what they were able to deploy or switch up tactics, using the access they already had. They’re probably having a field day these supply chain attacks. I don’t want to really think about it too much..
Great review! 🙂
Wow! This would make an IT tech pretty embarrassed to find out the event logger he just downloaded was the source of a major network breach. I can think of times I’ve used similar software, and the thought of it is scary! Years ago there was no way to evaluate the reputation of a software company, and with this kind of business practice, it looks like it is still going to be a difficult mission.
It isn’t very difficult at all. It does require someone to take the blunders off though. It isn’t very difficult to understand how much of an issue software like Java and Flash really is. Then there is ActiveX. These things ALL have a very clear and well defined history. No one wants to deal with it. In fact, the demand for these things gets even deeper embedded. These are just examples.
One of the biggest things we mention to new IT employees is the mandatory requirement to only download software on our approved list and only from the vendor’s own site. Doesn’t help because they’re all smarter than us (IT Security).
The last clown who railed against every control grabbed some software from the first search engine link and yeah, it had some guests inside it. One rebuilt administration workstation and one written warning letter later and he has a new attitude.
If we give this Grigorof the benefit of the doubt, and say that he wasn’t keeping any record of purchases to specific industries/agencies, then this issue lay right at the foot of our government. Accepting the use of such software (not for gov’t/enterprise, not supported, ect) was the issue, regardless of Grigorof’s claims.
Private enterprises without ties to the public sector being breached is bad enough. Who would have thought this would have ever been acceptable? We have government agencies, public research organizations and critical/sensitive industry, apparently using software from vendors who have no intention of supporting enterprise and government environments. Our approach is still flawed. We’re missing the mark here all together.
I think our critical infrastructure, sensitive government agencies and defense contractors need a rapid and aggressive change in cyber policy. This typical slow adaptation is no longer acceptable. This goes for sensitive organizations and the people holding your credit card info from that pizza you grabbed last night. They will say it’s too difficult, too expensive, too much too fast. We need to make it all happen, or our future may be one of everlasting cyber war decay.
Good work, Brian! Looks to be another case of a vendor hiding his head (and other body parts in the sand), hoping nothing bad will happen to him.
Some software vendors seem to think that their responsibility ends with the sale!
Regards,
Canada is still waiting for a data breach law to be enforced over the privacy act (http://ottawacitizen.com/news/national/canada-will-soon-force-companies-to-disclose-hacking-attempts-data-breaches)
But in this type of attack, there is no privacy/data breached. A proper incident response here is needed and this will be totally different of a IR related to data exfiltration.
Both cases need to be addressed in the regulation, but governments trend to focus on data privacy only…
Please explain to me how ANY update I could have acquired would have prevented any of this OR protected me from any of this?????
In fact, from the way I’m reading this article…..’updating’ would have helped spread the problem.
IMO, the only update that would have helped is an email sent to the product’s announcement list notifying all interested parties of the breach. The help is that you are informed that your may now be breached and should immediately investigate.
You are correct.
Where so many people get confused is that this ‘help’ does not come from Microsoft, Apple, Sony, Facebook, or the federal government.
What is this? did someone got killed or what ? ?
This is a great write up Brian. I appreciate the perspective and thoughtful process to truth-discovery you bring to our field.
You always inspire curiosity and that’s a valuable characteristic to build if one is in the field of technology and in particular, cyber security.
Keep on keeping on!
– Clay
While there are mandatory breach disclosure notice laws for compromise of end-users PII information (credit cards, etc), it seems to me that this was some type of Freeware software, or pay after use type software. Regardless, the software itself was compromised, thus opening the door to further footholds into compromised systems.
I think we are missing the point: how many breaches are needed before we FINALLY take this seriously? Brian does a fantastic job calling attention to this. But yet it continues, because no one learns from the mistakes. But, what does that mean? What am I getting at?
We see these breachs all the time, but I think we are beyond a “we could have done better” mentality. No one wants to take responsibility anymore. I’m not faulting anyone. But we have seen this so much in recent history, that I’m thinking people should start getting out in front of these incidents with a “Excuse me, we may have been breached and here is what we are doing to investigate…”, and “here is the results of our investigation; we failed, and here is what we are doing to correct the issues”.
Get out in front of the incident as soon as possible. Better to be forthcoming than play silly semantic games. I think we are in a surreal world currently in which this will continue until the paradigm shifts, and people FINALLY take IT Security seriously.
I agree wholeheartedly with your comment. However, companies are so focused on making quarterly numbers & shareholder value that IT security will always receive insufficient attention (thereby, money) because it doesn’t add perceived value. Business continues to ignore it, and they pay up anyway, when cleaning up after a breach.
Judging from the intent of the hackers, it looks state-sponsored … doesn’t it look like that?
I haven’t read the RSA report, but a criminal enterprise could earn a lot of money off this customer list too.
The only kind of attacker that seems unlikely is the young prankster.
Part of the,any government system, is the legacy hardware involved. The old stuff just is unable to run newer or the latest programs, and with push updates, is inundated with stuff it cannot handle. Is that the users fault, not being able to afford the latest machines or services? And then getting stuck with updates that disable the functionality of the used machine?
But, did enjoy the article, interesting response, but, nothing said on, removal of the whatever ware software. Did the patch remove the was, or is it possible we are all yet germie?
There are perhaps two kinds of malware. The lazy kind does one specific thing. The other can be used as a foothold. Once a hacker had a foothold in a system, they can (and probably will) do lots of things. Installing additional software, creating additional accounts, stealing the password file, stealing credentials, stealing other PII, attacking other devices.
The way someone “fixes” this is not by simply installing an update. That’s why external companies like RSA are hired to investigate: you need to know what was done in order to determine what needs to be undone. If there’s any chance the password file was stolen then you need to tell all users that any place they reused their passwords (they don’t follow the rules) and you need to reset their passwords. If new accounts were added, you need to at least remove them. You might need to rebuild computers from backups or if the backups were corrupted/compromised from scratch. Etc.