Data Breaches


23
Feb 19

Payroll Provider Gives Extortionists a Payday

Payroll software provider Apex Human Capital Management suffered a ransomware attack this week that severed payroll management services for hundreds of the company’s customers for nearly three days. Faced with the threat of an extended outage, Apex chose to pay the ransom demand and begin the process of restoring service to customers.

Roswell, Ga. based Apex HCM is a cloud-based payroll software company that serves some 350 payroll service bureaus that in turn provide payroll services to small and mid-sized businesses. At 4 a.m. on Tuesday, Feb. 19, Apex was alerted that its systems had been infected with a destructive strain of ransomware that encrypts computer files and demands payment for a digital key needed to unscramble the data.

The company quickly took all of its systems offline, and began notifying customers that it was trying to remediate a security threat. Over a series of bi-hourly updates, Apex kept estimating that it expected to restore service in a few hours, only to have to walk back those estimates almost every other time a new customer update went out.

Contacted Wednesday by an Apex client who was nervous about being unable to make this week’s payroll for his clients, KrebsOnSecurity reached out to Apex for comment. Ian Oxman, the company’s chief marketing officer, said the ransomware never touched customer data, but instead encrypted and disrupted everything in the company’s computer systems and at its off-site disaster recovery systems.

“We had just recently completed a pretty state-of-the-art disaster recovery plan off-site out and out of state that was mirroring our live system,” Oxman said. “But when the ransomware bomb went off, not only did it go through and infect our own network, it was then immediately picked up in our disaster recovery site, which made switching over to that site unusable.”

Oxman said Apex hired two outside security firms, and by Feb. 20 the consensus among all three was that paying the ransom was the fastest way to get back online. The company declined to specify how much was paid or what strain of ransomware was responsible for the attack.

“We paid the ransom, and it sucked,” Oxman said. “In respect for our clients who needed to get their businesses up and running that was going to be obviously the quicker path.”

Unfortunately for Apex, paying up didn’t completely solve its problems. For one thing, Oxman said, the decryption key they were given after paying the ransom didn’t work exactly as promised. Instead of restoring all files and folders to their pre-encrypted state, the decryption process broke countless file directories and rendered many executable files inoperable — causing even more delays.

“When they encrypt the data, that happens really fast,” he said. “When they gave us the keys to decrypt it, things didn’t go quite as cleanly.” Continue reading →


18
Feb 19

A Deep Dive on the Recent Widespread DNS Hijacking Attacks

The U.S. government — along with a number of leading security companies — recently warned about a series of highly complex and widespread attacks that allowed suspected Iranian hackers to siphon huge volumes of email passwords and other sensitive data from multiple governments and private companies. But to date, the specifics of exactly how that attack went down and who was hit have remained shrouded in secrecy.

This post seeks to document the extent of those attacks, and traces the origins of this overwhelmingly successful cyber espionage campaign back to a cascading series of breaches at key Internet infrastructure providers.

Before we delve into the extensive research that culminated in this post, it’s helpful to review the facts disclosed publicly so far. On Nov. 27, 2018, Cisco’s Talos research division published a write-up outlining the contours of a sophisticated cyber espionage campaign it dubbed “DNSpionage.”

The DNS part of that moniker refers to the global “Domain Name System,” which serves as a kind of phone book for the Internet by translating human-friendly Web site names (example.com) into numeric Internet address that are easier for computers to manage.

Talos said the perpetrators of DNSpionage were able to steal email and other login credentials from a number of government and private sector entities in Lebanon and the United Arab Emirates by hijacking the DNS servers for these targets, so that all email and virtual private networking (VPN) traffic was redirected to an Internet address controlled by the attackers.

Talos reported that these DNS hijacks also paved the way for the attackers to obtain SSL encryption certificates for the targeted domains (e.g. webmail.finance.gov.lb), which allowed them to decrypt the intercepted email and VPN credentials and view them in plain text.

On January 9, 2019, security vendor FireEye released its report, “Global DNS Hijacking Campaign: DNS Record Manipulation at Scale,” which went into far greater technical detail about the “how” of the espionage campaign, but contained few additional details about its victims.

About the same time as the FireEye report, the U.S. Department of Homeland Security issued a rare emergency directive ordering all U.S. federal civilian agencies to secure the login credentials for their Internet domain records. As part of that mandate, DHS published a short list of domain names and Internet addresses that were used in the DNSpionage campaign, although those details did not go beyond what was previously released by either Cisco Talos or FireEye.

That changed on Jan. 25, 2019, when security firm CrowdStrike published a blog post listing virtually every Internet address known to be (ab)used by the espionage campaign to date. The remainder of this story is based on open-source research and interviews conducted by KrebsOnSecurity in an effort to shed more light on the true extent of this extraordinary — and ongoing — attack.

The “indicators of compromise” related to the DNSpionage campaign, as published by CrowdStrike.

PASSIVE DNS

I began my research by taking each of the Internet addresses laid out in the CrowdStrike report and running them through both Farsight Security and SecurityTrails, services that passively collect data about changes to DNS records tied to tens of millions of Web site domains around the world.

Working backwards from each Internet address, I was able to see that in the last few months of 2018 the hackers behind DNSpionage succeeded in compromising key components of DNS infrastructure for more than 50 Middle Eastern companies and government agencies, including targets in Albania, Cyprus, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Saudi Arabia and the United Arab Emirates.

For example, the passive DNS data shows the attackers were able to hijack the DNS records for mail.gov.ae, which handles email for government offices of the United Arab Emirates. Here are just a few other interesting assets successfully compromised in this cyber espionage campaign:

-nsa.gov.iq: the National Security Advisory of Iraq
-webmail.mofa.gov.ae: email for the United Arab Emirates’ Ministry of Foreign Affairs
-shish.gov.al: the State Intelligence Service of Albania
-mail.mfa.gov.eg: mail server for Egypt’s Ministry of Foreign Affairs
-mod.gov.eg: Egyptian Ministry of Defense
-embassy.ly: Embassy of Libya
-owa.e-albania.al: the Outlook Web Access portal for the e-government portal of Albania
-mail.dgca.gov.kw: email server for Kuwait’s Civil Aviation Bureau
-gid.gov.jo: Jordan’s General Intelligence Directorate
-adpvpn.adpolice.gov.ae: VPN service for the Abu Dhabi Police
-mail.asp.gov.al: email for Albanian State Police
-owa.gov.cy: Microsoft Outlook Web Access for Government of Cyprus
-webmail.finance.gov.lb: email for Lebanon Ministry of Finance
-mail.petroleum.gov.eg: Egyptian Ministry of Petroleum
-mail.cyta.com.cy: Cyta telecommunications and Internet provider, Cyprus
-mail.mea.com.lb: email access for Middle East Airlines

The passive DNS data provided by Farsight and SecurityTrails also offered clues about when each of these domains was hijacked. In most cases, the attackers appear to have changed the DNS records for these domains (we’ll get to the “how” in a moment) so that the domains pointed to servers in Europe that they controlled.

Shortly after the DNS records for these TLDs were hijacked — sometimes weeks, sometimes just days or hours — the attackers were able to obtain SSL certificates for those domains from SSL providers Comodo and/or Let’s Encrypt. The preparation for several of these attacks can be seen at crt.sh, which provides a searchable database of all new SSL certificate creations.

Let’s take a closer look at one example. The CrowdStrike report references the Internet address 139.59.134[.]216 (see above), which according to Farsight was home to just seven different domains over the years. Two of those domains only appeared at that Internet address in December 2018, including domains in Lebanon and — curiously — Sweden.

The first domain was “ns0.idm.net.lb,” which is a server for the Lebanese Internet service provider IDM. From early 2014 until December 2018, ns0.idm.net.lb pointed to 194.126.10[.]18, which appropriately enough is an Internet address based in Lebanon. But as we can see in the screenshot from Farsight’s data below, on Dec. 18, 2018, the DNS records for this ISP were changed to point Internet traffic destined for IDM to a hosting provider in Germany (the 139.59.134[.]216 address).

Source: Farsight Security

Notice what else is listed along with IDM’s domain at 139.59.134[.]216, according to Farsight:

The DNS records for the domains sa1.dnsnode.net and fork.sth.dnsnode.net also were changed from their rightful home in Sweden to the German hosting provider controlled by the attackers in December. These domains are owned by Netnod Internet Exchange, a major global DNS provider based in Sweden. Netnod also operates one of the 13 “root” name servers, a critical resource that forms the very foundation of the global DNS system.

We’ll come back to Netnod in a moment. But first let’s look at another Internet address referenced in the CrowdStrike report as part of the infrastructure abused by the DNSpionage hackers: 82.196.11[.]127. This address in The Netherlands also is home to the domain mmfasi[.]com, which Crowdstrike says was one of the attacker’s domains that was used as a DNS server for some of the hijacked infrastructure.

As we can see in the screenshot above, 82.196.11[.]127 was temporarily home to another pair of Netnod DNS servers, as well as the server “ns.anycast.woodynet.net.” That domain is derived from the nickname of Bill Woodcock, who serves as executive director of Packet Clearing House (PCH).

PCH is a nonprofit entity based in northern California that also manages significant amounts of the world’s DNS infrastructure, particularly the DNS for more than 500 top-level domains and a number of the Middle East top-level domains targeted by DNSpionage. Continue reading →


12
Feb 19

Email Provider VFEmail Suffers ‘Catastrophic’ Hack

Email provider VFEmail has suffered what the company is calling “catastrophic destruction” at the hands of an as-yet unknown intruder who trashed all of the company’s primary and backup data in the United States. The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.

Founded in 2001 and based in Milwaukee, Wisc., VFEmail provides email service to businesses and end users. The first signs of the attack came on the morning of Feb. 11, when the company’s Twitter account started fielding reports from users who said they were no longer receiving messages. VFEmail’s Twitter account responded that “external facing systems, of differing OS’s and remote authentication, in multiple data centers are down.”

Two hours later, VFEmail tweeted that it had caught a hacker in the act of formatting one of the company’s mail servers in The Netherlands.

“nl101 is up, but no incoming email,” read a tweet shortly thereafter. “I fear all US based data my be lost.”

“At this time, the attacker has formatted all the disks on every server,” wrote VFEmail. “Every VM [virtual machine] is lost. Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”

In an update posted to the company’s Web site, VFEmail owner Rick Romero wrote that new email was being delivered and that efforts were being made to recover what user data could be salvaged.

“At this time I am unsure of the status of existing mail for US users,” Romero wrote. “If you have your own email client, DO NOT TRY TO MAKE IT WORK. If you reconnect your client to your new mailbox, all your local mail will be lost.”

Reached by KrebsOnSecurity on Tuesday morning, Romero said he was able to recover a backup drive hosted in The Netherlands, but that he fears all of the mail for U.S. users may be irreparably lost.

“I don’t have very high expectations of getting any U.S. data back,” Romero said in an online chat. Continue reading →


2
Jan 19

Cloud Hosting Provider DataResolution.net Battling Christmas Eve Ransomware Attack

Cloud hosting provider Dataresolution.net is struggling to bring its systems back online after suffering a ransomware infestation on Christmas Eve, KrebsOnSecurity has learned. The company says its systems were hit by the Ryuk ransomware, the same malware strain that crippled printing and delivery operations for multiple major U.S. newspapers over the weekend.

San Juan Capistrano, Calif. based Data Resolution LLC serves some 30,000 businesses worldwide, offering software hosting, business continuity systems, cloud computing and data center services.

The company has not yet responded to requests for comment. But according to a status update shared by Data Resolution with affected customers on Dec. 29, 2018, the attackers broke in through a compromised login account on Christmas Eve and quickly began infecting servers with the Ryuk ransomware strain.

Part of an update on the outage shared with Data Resolution customers via Dropbox on Dec. 29, 2018.

The intrusion gave the attackers control of Data Resolution’s data center domain, briefly locking the company out of its own systems. The update sent to customers states that Data Resolution shut down its network to halt the spread of the infection and to work through the process of cleaning and restoring infected systems.

Data Resolution is assuring customers that there is no indication any data was stolen, and that the purpose of the attack was to extract payment from the company in exchange for a digital key that could be used to quickly unlock access to servers seized by the ransomware.

A snippet of an update that Data Resolution shared with affected customers on Dec. 31, 2018.

The Ryuk ransomware strain was first detailed in an August 2018 report by security firm CheckPoint, which says the malware may be tied to a sophisticated North Korean hacking team known as the Lazarus Group.

Ryuk reportedly was the same malware that infected the Los Angeles Times‘ Olympic printing plant over the weekend, an attack that led to the disruption of newspaper printing and delivery services for a number of publications that rely on the plant — including the Los Angeles Times and the San Diego Union Tribune.

A status update shared by Data Resolution with affected customers earlier today indicates the cloud hosting provider is still working to restore email access and multiple databases for clients. The update also said Data Resolution is in the process of restoring service for companies relying on it to host installations of Dynamics GP, a popular software package that many organizations use for accounting and payroll services.  Continue reading →


3
Dec 18

Jared, Kay Jewelers Parent Fixes Data Leak

The parent firm of bling retailers Jared and Kay Jewelers has fixed a bug in the Web sites of both companies that exposed the order information for all of their online customers.

In mid-November 2018, KrebsOnSecurity heard from a Jared customer who found something curious after receiving a receipt via email for a pair of earrings he’d just purchased as a surprise gift for his girlfriend.

Dallas-based Web developer Brandon Sheehy discovered that slightly modifying the link in the confirmation email he received and pasting that into a Web browser revealed another customer’s order, including their name, billing address, shipping address, phone number, email address, items and total amount purchased, delivery date, tracking link, and the last four digits of the customer’s credit card number.

Sheehy said after discovering the weakness, his mind quickly turned to the various ways that crooks might exploit it.

“My first thought was they could track a package of jewelry to someone’s door and swipe it off their doorstep,” he said. “My second thought was that someone could call Jared’s customers and pretend to be Jared, reading the last four digits of the customer’s card and saying there’d been a problem with the order, and if they could get a different card for the customer they could run it right away and get the order out quickly. That would be a pretty convincing scam. Or just targeted phishing attacks.”

Concerned that his own information was similarly exposed, Sheehy contacted Jared parent company Signet Jewelers and asked them to fix the data exposure. When several weeks passed and Sheehy could still view his information and that of other Jared customers, he reached out to KrebsOnSecurity.

Scott Lancaster, chief information security officer at Signet, said the company did fix the problem for all future orders shortly after receiving a customer’s complaint. But Lancaster said Signet neglected to remedy the data exposure for all past orders until contacted by KrebsOnSecurity.

“When a customer first brought this matter to our attention in early November, we fixed it for all new orders going forward,” Lancaster said. “But we didn’t notice at the time that this applied to all past orders as well as future orders.” Continue reading →


1
Dec 18

What the Marriott Breach Says About Security

We don’t yet know the root cause(s) that forced Marriott this week to disclose a four-year-long breach involving the personal and financial information of 500 million guests of its Starwood hotel properties. But anytime we see such a colossal intrusion go undetected for so long, the ultimate cause is usually a failure to adopt the most important principle in cybersecurity defense that applies to both corporations and consumers: Assume you are compromised.

TO COMPANIES

For companies, this principle means accepting the notion that it is no longer possible to keep the bad guys out of your networks entirely. This doesn’t mean abandoning all tenets of traditional defense, such as quickly applying software patches and using technologies to block or at least detect malware infections.

It means accepting that despite how many resources you expend trying to keep malware and miscreants out, all of this can be undone in a flash when users click on malicious links or fall for phishing attacks. Or a previously unknown security flaw gets exploited before it can be patched. Or any one of a myriad other ways attackers can win just by being right once, when defenders need to be right 100 percent of the time.

The companies run by leaders and corporate board members with advanced security maturity are investing in ways to attract and retain more cybersecurity talent, and arranging those defenders in a posture that assumes the bad guys will get in.

This involves not only focusing on breach prevention, but at least equally on intrusion detection and response. It starts with the assumption that failing to respond quickly when an adversary gains an initial foothold is like allowing a tiny cancer cell to metastasize into a much bigger illness that — left undetected for days, months or years — can cost the entire organism dearly.

The companies with the most clueful leaders are paying threat hunters to look for signs of new intrusions. They’re reshuffling the organizational chart so that people in charge of security report to the board, the CEO, and/or chief risk officer — anyone but the Chief Technology Officer.

They’re constantly testing their own networks and employees for weaknesses, and regularly drilling their breach response preparedness (much like a fire drill). And, apropos of the Marriott breach, they are finding creative ways to cut down on the volume of sensitive data that they need to store and protect.

TO INDIVIDUALS

Likewise for individuals, it pays to accept two unfortunate and harsh realities:

Reality #1: Bad guys already have access to personal data points that you may believe should be secret but which nevertheless aren’t, including your credit card information, Social Security number, mother’s maiden name, date of birth, address, previous addresses, phone number, and yes — even your credit file.

Reality #2: Any data point you share with a company will in all likelihood eventually be hacked, lost, leaked, stolen or sold — usually through no fault of your own. And if you’re an American, it means (at least for the time being) your recourse to do anything about that when it does happen is limited or nil.

Marriott is offering affected consumers a year’s worth of service from a company owned by security firm Kroll that advertises the ability to scour cybercrime underground markets for your data. Should you take them up on this offer? It probably can’t hurt as long as you’re not expecting it to prevent some kind of bad outcome. But once you’ve accepted Realities #1 and #2 above it becomes clear there is nothing such services could tell you that you don’t already know.

Once you’ve owned both of these realities, you realize that expecting another company to safeguard your security is a fool’s errand, and that it makes far more sense to focus instead on doing everything you can to proactively prevent identity thieves, malicious hackers or other ne’er-do-wells from abusing access to said data.

This includes assuming that any passwords you use at one site will eventually get hacked and leaked or sold online (see Reality #2), and that as a result it is an extremely bad idea to re-use passwords across multiple Web sites. For example, if you used your Starwood password anywhere else, that other account you used it at is now at a much higher risk of getting compromised. Continue reading →


30
Nov 18

Marriott: Data on 500 Million Guests Stolen in 4-Year Breach

Hospitality giant Marriott today disclosed a massive data breach exposing the personal and financial information on as many as a half billion customers who made reservations at any of its Starwood properties over the past four years.

Marriott said the breach involved unauthorized access to a database containing guest information tied to reservations made at Starwood properties on or before Sept. 10, 2018, and that its ongoing investigation suggests the perpetrators had been inside the company’s networks since 2014.

Marriott said the intruders encrypted information from the hacked database (likely to avoid detection by any data-loss prevention tools when removing the stolen information from the company’s network), and that its efforts to decrypt that data set was not yet complete. But so far the hotel network believes that the encrypted data cache includes information on up to approximately 500 million guests who made a reservation at a Starwood property.

“For approximately 327 million of these guests, the information includes some combination of name, mailing address, phone number, email address, passport number, Starwood Preferred Guest account information, date of birth, gender, arrival and departure information, reservation date and communication preferences,” Marriott said in a statement released early Friday morning.

Marriott added that customer payment card data was protected by encryption technology, but that the company couldn’t rule out the possibility the attackers had also made off with the encryption keys needed to decrypt the data.

The hotel chain did not say precisely when in 2014 the breach was thought to have begun, but it’s worth noting that Starwood disclosed its own breach involving more than 50 properties in November 2015, just days after being acquired by Marriott. According to Starwood’s disclosure at the time, that earlier breach stretched back at least one year — to November 2014. Continue reading →


21
Nov 18

USPS Site Exposed Data on 60 Million Users

U.S. Postal Service just fixed a security weakness that allowed anyone who has an account at usps.com to view account details for some 60 million other users, and in some cases to modify account details on their behalf.

Image: USPS.com

KrebsOnSecurity was contacted last week by a researcher who discovered the problem, but who asked to remain anonymous. The researcher said he informed the USPS about his finding more than a year ago yet never received a response. After confirming his findings, this author contacted the USPS, which promptly addressed the issue.

The problem stemmed from an authentication weakness in a USPS Web component known as an “application program interface,” or API — basically, a set of tools defining how various parts of an online application such as databases and Web pages should interact with one another.

The API in question was tied to a Postal Service initiative called “Informed Visibility,” which according to the USPS is designed to let businesses, advertisers and other bulk mail senders “make better business decisions by providing them with access to near real-time tracking data” about mail campaigns and packages.

In addition to exposing near real-time data about packages and mail being sent by USPS commercial customers, the flaw let any logged-in usps.com user query the system for account details belonging to any other users, such as email address, username, user ID, account number, street address, phone number, authorized users, mailing campaign data and other information.

Many of the API’s features accepted “wildcard” search parameters, meaning they could be made to return all records for a given data set without the need to search for specific terms. No special hacking tools were needed to pull this data, other than knowledge of how to view and modify data elements processed by a regular Web browser like Chrome or Firefox.

A USPS brochure advertising the features and benefits of Informed Visibility.

In cases where multiple accounts shared a common data element — such as a street address — using the API to search for one specific data element often brought up multiple records. For example, a search on the email addresses for readers who volunteered to help with this research turned up multiple accounts when those users had more than one user signed up at the same physical address.

“This is not good,” said one anonymous reader who volunteered to help with this research, after viewing a cut-and-paste of his USPS account details looked up via his email address. “Especially since we moved due to being threatened by a neighbor.”

Nicholas Weaver, a researcher at the International Computer Science Institute and lecturer at UC Berkeley, said the API should have validated that the account making the request had permission to read the data requested.

“This is not even Information Security 101, this is Information Security 1, which is to implement access control,” Weaver said. “It seems like the only access control they had in place was that you were logged in at all. And if you can access other peoples’ data because they aren’t enforcing access controls on reading that data, it’s catastrophically bad and I’m willing to bet they’re not enforcing controls on writing to that data as well.”

A cursory review by KrebsOnSecurity indicates the promiscuous API let any user request account changes for any other user, such as email address, phone number or other key details.

Fortunately, the USPS appears to have included a validation step to prevent unauthorized changes — at least with some data fields. Attempts to modify the email address associated with my USPS account via the API prompted a confirmation message sent to the email address tied to that account (which required clicking a link in the email to complete the change).

It does not appear USPS account passwords were exposed via this API, although KrebsOnSecurity conducted only a very brief and limited review of the API’s rather broad functionality before reporting the issue to the USPS. The API at issue resides here; a copy of the API prior to its modification on Nov. 20 by the USPS is available here as a text file. Continue reading →


1
Nov 18

Equifax Has Chosen Experian. Wait, What?

A year after offering free credit monitoring to all Americans on account of its massive data breach that exposed the personal information of nearly 148 million people, Equifax now says it has chosen to extend the offer by turning to a credit monitoring service offered by a top competitor — Experian. And to do that, it will soon be sharing with Experian contact information that affected consumers gave to Equifax in order to sign up for the service.

The news came in an email Equifax is sending to people who took the company up on its offer for one year of free credit monitoring through its TrustedID Premier service.

Here’s the introduction from that message:

“We recently sent you an email advising you that, until further notice, we would be extending the free TrustedID® Premier subscription you enrolled in following the September 7, 2017 cybersecurity incident. We are now pleased to let you know that Equifax has chosen Experian®, one of the three nationwide credit bureaus, to provide you with an additional year of free credit monitoring service. This extension is at no cost to you , and you will not be asked to provide a credit card number or other payment information. You have until January 31, 2019 to enroll in this extension of free credit monitoring through IDnotify™, a part of Experian.”

Equifax says it will share the name, address, date of birth, Social Security number and self-provided phone number and email address with Experian for anyone who signed up for its original TrustedID Premier offering. That is, unless those folks affirmatively opt-out of having that information transferred from Equifax to Experian.

But not to worry, Equifax says: Experian already has most of this data.

“Experian currently has and is using this information (except phone number and email address) in the fulfillment of the Experian file monitoring which is part of your current service with TrustedID Premier,” Equifax wrote in its email. “Experian will only use the information Equifax is sharing to confirm your identity and securely enroll you in the Experian product, and will not use it for marketing or solicitation.”

Even though people who don’t opt-out of the new IDnotify offer will have their contact information automatically shared with Experian, TrustedID Premier users must still affirmatively enroll in the new program before then end of January 2019 — the date the TrustedID product expires.

Equifax’s FAQ on the changes is available here. Continue reading →


2
Oct 18

When Security Researchers Pose as Cybercrooks, Who Can Tell the Difference?

A ridiculous number of companies are exposing some or all of their proprietary and customer data by putting it in the cloud without any kind of authentication needed to read, alter or destroy it. When cybercriminals are the first to discover these missteps, usually the outcome is a demand for money in return for the stolen data. But when these screw-ups are unearthed by security professionals seeking to make a name for themselves, the resulting publicity often can leave the breached organization wishing they’d instead been quietly extorted by anonymous crooks.

Last week, I was on a train from New York to Washington, D.C. when I received a phone call from Vinny Troia, a security researcher who runs a startup in Missouri called NightLion Security. Troia had discovered that All American Entertainment, a speaker bureau which represents a number of celebrities who also can be hired to do public speaking, had exposed thousands of speaking contracts via an unsecured Amazon cloud instance.

The contracts laid out how much each speaker makes per event, details about their travel arrangements, and any requirements or obligations stated in advance by both parties to the contract. No secret access or password was needed to view the documents.

It was a juicy find to be sure: I can now tell you how much Oprah makes per event (it’s a lot). Ditto for Gwyneth Paltrow, Olivia Newton John, Michael J. Fox and a host of others. But I’m not going to do that.

Firstly, it’s nobody’s business what they make. More to the point, All American also is my speaker bureau, and included in the cache of documents the company exposed in the cloud were some of my speaking contracts. In fact, when Troia called about his find, I was on my way home from one such engagement.

I quickly informed my contact at All American and asked them to let me know the moment they confirmed the data was removed from the Internet. While awaiting that confirmation, my pent-up frustration seeped into a tweet that seemed to touch a raw nerve among others in the security industry.

The same day I alerted them, All American took down its bucket of unsecured speaker contract data, and apologized profusely for the oversight (although I have yet to hear a good explanation as to why this data needed to be stored in the cloud to begin with).

This was hardly the first time Troia had alerted me about a huge cache of important or sensitive data that companies have left exposed online. On Monday, TechCrunch broke the story about a “breach” at Apollo, a sales engagement startup boasting a database of more than 200 million contact records. Calling it a breach seems a bit of a stretch; it probably would be more accurate to describe the incident as a data leak.

Just like my speaker bureau, Apollo had simply put all this data up on an Amazon server that anyone on the Internet could access without providing a password. And Troia was again the one who figured out that the data had been leaked by Apollo — the result of an intensive, months-long process that took some extremely interesting twists and turns.

That journey — which I will endeavor to describe here — offered some uncomfortable insights into how organizations frequently learn about data leaks these days, and indeed whether they derive any lasting security lessons from the experience at all. It also gave me a new appreciation for how difficult it can be for organizations that screw up this way to tell the difference between a security researcher and a bad guy.

THE DARK OVERLORD

I began hearing from Troia almost daily beginning in mid-2017. At the time, he was on something of a personal mission to discover the real-life identity behind The Dark Overlord (TDO), the pseudonym used by an individual or group of criminals who have been extorting dozens of companies — particularly healthcare providers — after hacking into their systems and stealing sensitive data.

The Dark Overlord’s method was roughly the same in each attack. Gain access to sensitive data (often by purchasing access through crimeware-as-a-service offerings), and send a long, rambling ransom note to the victim organization demanding tens of thousands of dollars in Bitcoin for the safe return of said data.

Victims were typically told that if they refused to pay, the stolen data would be sold to cybercriminals lurking on Dark Web forums. Worse yet, TDO also promised to make sure the news media knew that victim organizations were more interested in keeping the breach private than in securing the privacy of their customers or patients.

In fact, the apparent ringleader of TDO reached out to KrebsOnSecurity in May 2016 with a remarkable offer. Using the nickname “Arnie,” the public voice of TDO said he was offering exclusive access to news about their latest extortion targets.

Snippets from a long email conversation in May 2016 with a hacker who introduced himself as Adam but would later share his nickname as “Arnie” and disclose that he was a member of The Dark Overlord. In this conversation, he is offering to sell access to scoops about data breaches that he caused.

Arnie claimed he was an administrator or key member on several top Dark Web forums, and provided a handful of convincing clues to back up his claim. He told me he had real-time access to dozens of healthcare organizations they’d hacked into, and that each one which refused to give in to TDO’s extortion demands could turn into a juicy scoop for KrebsOnSecurity.

Arnie said he was coming to me first with the offer, but that he was planning to approach other journalists and news outlets if I declined. I balked after discovering that Arnie wasn’t offering this access for free: He wanted 10 bitcoin in exchange for exclusivity (at the time, his asking price was roughly equivalent to USD $5,000).

Perhaps other news outlets are accustomed to paying for scoops, but that is not something I would ever consider. And in any case the whole thing was starting to smell like a shakedown or scam. I declined the offer. It’s possible other news outlets or journalists did not; I will not speculate on this matter further, other than to say readers can draw their own conclusions based on the timeline and the public record. Continue reading →