A phishing attack last week gave attackers access to email and files at the California State Controller’s Office (SCO), an agency responsible for handling more than $100 billion in public funds each year. The phishers had access for more than 24 hours, and sources tell KrebsOnSecurity the intruders used that time to steal Social Security numbers and sensitive files on thousands of state workers, and to send targeted phishing messages to at least 9,000 other workers and their contacts.
A notice of breach posted by the California State Controller’s Office.
In a “Notice of Data Breach” message posted on Saturday, Mar. 20, the Controller’s Office said that for more than 24 hours starting on the afternoon of March 18 attackers had access to the email records of an employee in its Unclaimed Property Division after the employee clicked a phishing link and then entered their email ID and password.
“The SCO has reason to believe the compromised email account had personal identifying information contained in Unclaimed Property Holder Reports,” the agency said, urging state employees contacted by the agency to place fraud alerts on their credit files with the major consumer bureaus. “The unauthorized user also sent potentially malicious emails to some of the SCO employee’s contacts.”
The SCO responded in an email that no state employee data was compromised.
“A single employee email account was briefly compromised by a spear phishing attack and promptly disabled,” SCO spokesperson Jennifer Hanson said. “SCO has notified the employee’s contacts who may have received a potentially malicious email from the unauthorized user. SCO team members have identified all personal information included in the compromised email account and begun the process of notifying affected parties. The Controller is going over and beyond the notification requirements in law by providing both actual mailed notification and substitute notification in an effort to ensure the broadest possible notification.”
A source in an adjacent California state agency who’s been tracking the incident internally with other employees says the SCO forgot to mention the intruders also had access to the phished employee’s Microsoft Office 365 files — and potentially any files shared with that account across the state network.
“This isn’t even the full extent of the breach,” said the California state employee, who spoke on condition of anonymity.
The source claims the intruders stole several documents with personal and financial data on thousands of state employees, and then used the phished employee’s inbox to send targeted phishing emails to at least 9,000 California state workers and their contacts. In a follow-up response to those claims, the SCO said its “IT security staff were able to determine — based on the same logs that identified the intrusion — that no access was made to any Office 365 files other than the employee’s mailbox.”
The State Controller is the Chief Fiscal Officer of California, the sixth largest economy in the world. Source: sco.ca.gov.
Many attackers can do a great deal of damage with 24 hours of access to a user’s account. And spear-phishing others that frequently interact with the SCO via email could land the bad guys even more access to state systems. The SCO holds an enormous amount of personal and financial information on millions of people and companies that do business with or in the state.
Organizations hoping to improve internal security often turn to companies that help employees learn how to detect and dodge email phishing attacks — by sending them simulated phishing emails and then grading employees on their responses. The employee said that until very recently California was using one such company to help them conduct regular employee training on phishing.
Then in October 2020, the California Department of Technology (CDT) issued a new set of guidelines that effectively require all executives, managers and supervisors to know all of the details of a phishing exercise before it occurs. Which suggests plenty of people who definitely should get phish tested along with everyone else won’t get the same ongoing training.
“Meaning, such people will not be tested ever again,” the state agency source said. “It’s utterly absurd and no one at CDT is taking ownership of this kludge. The standard was also written in such a way to effectively ban dynamic testing like you see in KnowBe4, where even an administrator won’t know what phishing template they might receive.” [Full disclosure: KnowBe4 is an advertiser on this site].
The CDT issued the following statement in response: “SCO informed CDT they have contained the phishing attack. The characterization of the CDT phishing exercise standard is incorrect. Before phishing tests in any state agency are performed, internal business units are advised to coordinate to avoid disruption or operational impact to public services. Supervisors and managers are routinely tested without advance notice to ensure employees at every level are aware of security hazards and can learn how to avoid them.”
Update, 3:44 p.m. ET: Added comment and response from the California SCO.
Update, 5:38 p.m. ET: Added additional comment from SCO about cloud access.
Update, 6:58 p.m. ET: Added response from CDT.
BK, quick patch: “such people will be tested ever again”…
Thanks.
How are they not using MFA in this day and age?
…your tax (CA residents) dollars at work…
…even most federal contractors now have 2fa for email…
Where do you get reliable statistics on contractor 2FA?
When I search for something actually mandated for the feds like DMARC, I get numbers between 70 to 80 percent. DMARC takes very little work for the sysadmin and zero friction for the users.
…there’s not reliable stats on 2fa (the DoD calls it multifactor)…
…NIST SP 800-171 and the new CMMC require it for level 3 and the DFARS mandates it in the contract boilerplate…
…but like all things it’s one thing to mandate it, another to enforce it, I remember testifying for it literally years ago…
…they’re hoping competitive pressure will bring compliance along…
…but at least it’s written down unlike in most States…
As yes, like the DMARC mandate that seems to take forever to reach compliance. I can just imagine the headaches maintaining MFA unless something like a Yukikey is used. People have multiple devices these days.
I catch the occasional clown spoofing my email via the DMARC reports. As if a complaint to mail.ru would do anything.
One would be shocked at the number of state level government entities that are NOT using MFA.
Reasons are many, but high hitters IMHO are:
1) Budget constraints.
2) Lack of state policy or law from the legislative level (remember that FISMA does not apply to states) to force the effort.
3) Lack of cohesion of though between state agencies (some agencies still think they don’t need such security where others get why it is a good thing but cannot swing the cost to go it alone).
4) Lack of technical resources to implement a secure and sustainable MFA solution.
5) Reluctance to address identity proofing as a policy for state workers AND citizens/workers.
6) In some states, union resistance to change is a real thing.
I just realized I am ranting… my apologies.
It just frustrates me to no end that my own state cannot get their act together on MFA for state systems and services.
LOGIN/PASSWORD AUTHENTCIATION MUST DIE!!!
”union resistance to change is a real thing.”
pls explain YOUR reasoning of Unions {I’m one}
not wanting to implement MultiFactor Authen within this context,
on a website
or as a tool to protect their workers and others.
AWS – interesting set of conjectures 1-6 – but #4 ? – I run IT for a 60 seat non-profit and we have had MFA in use for 4 years ( … prob cannot mention my provider here)
I don’t see how 2FA/MFA would help here.
A web site that prompts for a username and password can also mimic prompting for the 2FA, and then they can authenticate w/ the real site.
Users don’t understand how 2FA works (and web sites do a bad job of enabling users to understand when to expect it).
I’m not saying 2FA shouldn’t be used (it should!), just I don’t think it helps for phishing-MITM attacks which is effectively what this is.
Why can someone access the state network without coming in through a VPN?
Why can someone access the mail server without MFA?
Their email ID and password should maybe auth to the Exchange server (since they are not using Duo auth as an extra measure), but not the state network.
“had access to the phished employee’s Microsoft Office 365 files — and potentially any files shared with that account across the state network”
O365 is not the same as the internal “state network”.
O365 has email and file sharing. Teams, Sharepoint, OneDrive, etc.
These files are accessible once you are authenticated to O365.
Duo or Microsoft’s own Authenticator MFA solution would both be nice. But against a good phishing page that relays creds in real-time… the phishing would still work.
Seems like a bad idea to have “documents with personal and financial data on thousands of state employees” on OneDrive, which can be accessed through Office 365 auth, even through a browser at onedrive.live.com .
Stuff like that should be on an internal network and accessed only through a VPN.
Yeah… this has been the fight against “cloud first” for a long time. Cloud first is winning though.
Many enterprises do want to move more to the cloud, have lower costs, and be less reliant on their own data centers. Many of them rightly believe Microsoft can do security better than their own infrastructure people.
It’s a trade off. And depends greatly on what security they have on-premises.
O365 or their own VPN portal? I’ve seen both have horrible security.
the struggle is real to keep management from being attracted to the cloud like a bug to a lamp. The marketing is unlimited, the risk are too. I will continue to try and keep our data on prem, where millions of people aren’t accessing the same equipment with hundreds of layers of exploitable software. Your financial security should not go to the lowest bidder or the loudest salesperson.
Seems like the umpteenth story I am reading today about another breach that could have been prevented.
DMARC email domain security was something Brian mentioned 5 years ago and our firm wrote an entire step-by-step instruction set to help clients resolve themselves, also 5 years ago. But still, the majority of email domains are not protected with this simple configuration.
Even if employees were trained every day for security awareness, someone could still click on the wrong thing once and infect the whole place. Unless the organization spent $2 per month per employee to protect all browsing activities with a simple DNS filtering service, that would have prevented the malicious site from even being accessed by anyone in the organization.
Same thing with MFA or complex passwords or account lockouts for too many attempts or account privilege escalation notification. None of these best practices cost anything, or much at all. But organizations don’t implement them because they are “inconvenient”.
Seems like organizations really will not learn from the experience of others, they just have to experience the pain and suffering themselves before they might consider doing something about it.
You mention a lot of security features that might be simple to implement…. but aren’t really going to stop this kind of attack.
1) DMARC – This wasn’t email domain spoofing. So no protections there.
2) DNS filtering would not help most targeted phishing attacks, because it takes some time before a phishing site becomes blacklisted.
3) MFA – As mentioned before, good phishing sites are more than just offline credential harvesting for another time. They have real-time login relays that also steal OTP codes or Push notification tokens.
Why are so many comments here focused on MFA?
MFA is not a panacea… and against phishing attacks like this…. not as effective as people think.
The employee clicked on a link in his email, which went to a cloned login page. He entered his username/password thinking it was a legit login page.
Even with MFA (push to device, typed OTP, etc…) the attack would still be effective since the victim would supply that information too.
Good phishing sites are often real-time relays, so expiration of OTP codes or Push authentication are simply passed to the email service login page without delay.
…doh…
…training is used to teach users not to click on things, even they look real…
…mfa just slows them down until the brain (hopefully) engages…
…technology can never fix dumb…
Hope based security
…as pascal said it’s better to be wrong…
…then depend on technology (ok, i added that)…
Are push relays accepting known bad or cloud ips/proxies? We can place no assumptions that they block that, obviously, but I’m curious what studies have found
This is one area where U2F or FIDO has an advantage. The browser must enforce same origin policy for any code calling the U2F API and the website origin and the authentication origin must be identical.
Indeed, phishing is thwarted with U2F or FIDO2 devices.
Of course, good luck giving all state employees a FIDO2 key. Just not in the budget.
Mobile phones are at least ubiquitous now and personal phones can be used as a secure authenticator.
There is some work being done to get a phone app to do FIDO2 over Bluetooth, but it sucked when I tested it.
And Windows Hello can now act as a FIDO authenticator. But then that does limit access to corporate Windows devices, which may be a non-starter too.
yes…you get it. well said on point short concise
This “story” seems more like gossip than a news story.
google Pepe Silvia and watch the video lol DoD…i cant wait for my negotiation for contract LOL reading up on CDT. when ive been syncing using atomic clocks hahaha this isnt gossip this a timebomb in waiting…im glad Kerbs wrote about it, its a concern that need to be addressed def.
this is why i said read about OTP and why RT OTP is so important4:14 AMthis will eventually be implemented into all e-commerce by geo location approval apples system to system code to confirm isnt going to cut it cause of that server app and Virtualization….. RMM’s on Mac = Mac not happy…this is why I got a Razor computer Mac hates Razor computers they truly arnt to great except for a application called synapse….this is the same thing as dswservice ie agent rmm api custom github it…. your CEO knows this…..thats why the dont give a fuck about apple or windows permissions anymore…..logging will auto mate an alert if something unsual is happening be it whatever the coder builds into the SQL sequence tomanager the Agents of the RMM ie the Server host which goes threw the network admin the Sysadmin sets so the network admin doesnt like that now the sys admin can move across the WAN without his Security router being tiggered….the sysadmin knows this….thats why I said learn MSTSC from any IP you can pw ps:push a command to force o365 to delete any email someone reports RT…..faster then the Security router can re-route to the fail over DNS IF they have one … when people RDP…..you only see the screen…assume your the driving instructor and your training break pedal is cut….what do you do to pre 2021 to close a intrusion thats spreading fasthttps://www.youtube.com/watch?v=MYtjpIwamosYEEEEEEEE HAWWWWW hahahahaha
I couldnt be more serious….
These clips are the best way to explain what is goin on to non IT users….
https://www.youtube.com/watch?v=5-79mMCkNbQ
^ -Life of a Admin
https://www.youtube.com/watch?v=1yOrrRhUEyE
^ the bad Actor
https://www.youtube.com/watch?v=zZn0njzFL68
^ Day in the life of a Breach for Audit…
The Noc/Admin/Support
Phishing testing and training does not work. The interaction needs to engineered around the human, as opposed to requiring endless ineffective training that demands that human change cognitive patterns developed over eons.
All the training in the world will not fix a systematically broken interaction.
Hey, very intersting info but me neeed thinks about this. Thanks for sharing.
Data breaches are simply common place. The focus seems to be on mitigation measures and compartmentalizing data so if one category is accessed by a third party, not all data is lost. Thank you for the detailed post Krebs on Security, and wish everyone here a healthy 2021! Darren Chaker
All great comments… but lets put a foot forward. Ok….Budget. Security. SourceCode. Testing. Usability
1. https://freeotp.github.io/
– apple & android
2. Custom API – FreeIPA
3. Backup – Cloud, hybrid and MSP – Monthly patch/update/backup
Exchange Servers – Dont cheap it on the RAM….even on a VM….allocate plenty….4096 aint enough, 8192, aint enough you need to dedicate a pci-e SSD to handle it exclusively – Make sure your Admin, and Net admin have RDP’s ready to a Virtual Host on the main Host.
Install Barracuda or a sophos xg blade, or sonicwall….or 2+3+1 any all or 1…. hardware…vs software.. ok im more about the hardware….
greylisting for rebound & bounce back…..
This is Free and very very simple….if your employee can not email the IT support desk…. ask any email they are not sure of that slips there or has an attachment to make it common practice to sandbox it…..ie open in a secure container.
https://sandboxie-plus.com/
This is simple implementation that can be automated or emailed out to a SMB in a memo with a read reply back to the admin…. therefore liability will fall on the employee for a breach…simple. This is the reality companies and DoD dont want to take 5-15min to explain to the company during a weekly or monthly meeting. Now accidents happen so ….admins come-on check the logs. RMM’s chck the log.
No reason why a outside NS needs to be checked up…why…cause it should mainly be coming via AuthTokens on a GAL by known mail DMARC exchange records.
So…..fux a DMARC….why?!
Did anyone watch the video I posted…simple someone cut the breaks ie. DNS NS or DNS spoofing…due to someone checking a outside email mainly gmail….so metadata collects….welcome to the world of the browser…..
Admins dont harden or update this push and they should….ditch google chrome #1, and understand that if you use OWA to check your email and dont use the suite….all the BS inside any browser you use is like a leech in a pond reach to suck up what it can. So….again…. simple
To much MFA….people cant get to there email. Simple
IT needs to be kept simple….keep it simple FreeOTP….a Security Router with a Hidden DNS bounce relay on either hardware or software to filter….
and if alllll that fails…..open the email in a sandbox…
This is simple practice….you can even implement OWA /365 to automate this….for the GAL and the sub-domains.
Admins come on set simple inbound/outbound rules and block outside emails. If people want to do that they can on there own time or in a VM to there home computer. or personal phone. tablet whatever.
https://www.youtube.com/watch?v=zZn0njzFL68
Only so much of us that kno what the deal is …. so DoD…know what the deal is ANTE up when you see Cheshire : )
https://www.youtube.com/watch?v=MYtjpIwamos
Wildcards <—-the leech the enemy
The Gmail search functionality works based on what I would describe as tokens. A token is any sequence of alphanumeric characters separated by a space or by other non-alphanumeric characters such as underscore, full stop (period), "@", dash, etc.. So in peter.ford23@example.com there are 4 tokens: "peter", "ford23", "example" and "com".
Wildcards within tokens do not work. Wildcards outside of tokens are unnecessary and misleading.
These work with the write syntx and attempts trust me
Thats why I said you need to check the web browser backend ie.
XML
About:config
Subject: @dodemail:active
whatever…..thank you nck files. esp ones of computers where they didnt blow away the old AD account.
So who cares about DMARC…check the SPF
https://www.dmarcanalyzer.com/spf/checker/
Kind of ironic I got an email from ca.gov today with my email to them included in the response.
This part caught my eye.
CAUTION: This email originated from outside of the organization. Do not click on any links or open any attachments unless you recognize the sender and know the content is safe.
———–
I’m tempted to put
Warning: this email it going to the state of California which has known to be hacked
in my reply.
I was recently notified by the University of California they were hacked. I find it most disturbing there was not even an apology when they notified me my data was hacked due to them running the accellion file sharing appliance – https://abcnews.go.com/Technology/wireStory/university-california-victim-nationwide-hack-attack-76847800. Are we to the point now that it’s just assumed this is routine? IMO, this is the issue with moving files to any data base type product. I’ll stick with file shares and add security portals in front like myworkdrive.com with two factor and DLP protection with no password hashes stored on the appliance..