Networking software giant Citrix Systems says malicious hackers were inside its networks for five months between 2018 and 2019, making off with personal and financial data on company employees, contractors, interns, job candidates and their dependents. The disclosure comes almost a year after Citrix acknowledged that digital intruders had broken in by probing its employee accounts for weak passwords.
Citrix provides software used by hundreds of thousands of clients worldwide, including most of the Fortune 100 companies. It is perhaps best known for selling virtual private networking (VPN) software that lets users remotely access networks and computers over an encrypted connection.
In March 2019, the Federal Bureau of Investigation (FBI) alerted Citrix they had reason to believe cybercriminals had gained access to the company’s internal network. The FBI told Citrix the hackers likely got in using a technique called “password spraying,” a relatively crude but remarkably effective attack that attempts to access a large number of employee accounts (usernames/email addresses) using just a handful of common passwords.
In a statement released at the time, Citrix said it appeared hackers “may have accessed and downloaded business documents,” and that it was still working to identify what precisely was accessed or stolen.
But in a letter sent to affected individuals dated Feb. 10, 2020, Citrix disclosed additional details about the incident. According to the letter, the attackers “had intermittent access” to Citrix’s internal network between Oct. 13, 2018 and Mar. 8, 2019, and that there was no evidence that the cybercrooks still remain in the company’s systems.
Citrix said the information taken by the intruders may have included Social Security Numbers or other tax identification numbers, driver’s license numbers, passport numbers, financial account numbers, payment card numbers, and/or limited health claims information, such as health insurance participant identification number and/or claims information relating to date of service and provider name.
It is unclear how many people received this letter, but the communication suggests Citrix is contacting a broad range of individuals who work or worked for the company at some point, as well as those who applied for jobs or internships there and people who may have received health or other benefits from the company by virtue of having a family member employed by the company.
Citrix’s letter was prompted by laws in virtually all U.S. states that require companies to notify affected consumers of any incident that jeopardizes their personal and financial data. While the notification does not specify whether the attackers stole proprietary data about the company’s software and internal operations, the intruders certainly had ample opportunity to access at least some of that information as well.
Shortly after Citrix initially disclosed the intrusion in March 2019, a little-known security company Resecurity claimed it had evidence Iranian hackers were responsible, had been in Citrix’s network for years, and had offloaded terabytes of data. Resecurity also presented evidence that it notified Citrix of the breach as early as Dec. 28, 2018, a claim Citrix initially denied but later acknowledged.
Iranian hackers recently have been blamed for hacking VPN servers around the world in a bid to plant backdoors in large corporate networks. A report released this week (PDF) by security firm ClearSky details how Iran’s government-backed hacking units have been busy exploiting security holes in popular VPN products from Citrix and a number of other software firms.
ClearSky says the attackers have focused on attacking VPN tools because they provide a long-lasting foothold at the targeted organizations, and frequently open the door to breaching additional companies through supply-chain attacks. The company says such tactics have allowed the Iranian hackers to gain persistent access to the networks of companies across a broad range of sectors, including IT, security, telecommunications, oil and gas, aviation, and government.
Among the VPN flaws available to attackers is a recently-patched vulnerability (CVE-2019-19781) in Citrix VPN servers dubbed “Shitrix” by some in the security community. The derisive nickname may have been chosen because while Citrix initially warned customers about the vulnerability in mid-December 2019, it didn’t start releasing patches to plug the holes until late January 2020 — roughly two weeks after attackers started using publicly released exploit code to break into vulnerable organizations.
How would your organization hold up to a password spraying attack? As the Citrix hack shows, if you don’t know you should probably check, and then act on the results accordingly. It’s a fair bet the bad guys are going to find out even if you don’t.
An interesting example of what can be done with just a little bit of customer information in the news today: https://www.bbc.co.uk/news/uk-england-hampshire-51536747
The worst form of reporting I see on this article.
What’s so bad about the reporting?
More bad news, it seems it never ends
“More bad news, it seems it never ends”
Like any other form of crime, you’re right, it will never end. Globally, it’s probably impossible to stop criminals from attempting to infiltrate networks, but at least you have the power to stop your accounts from being accessed because of a weak password.
There are many suggestions available for password security, but my absolute favourites are:
1) Be a moving target. Moving (password) targets are extremely difficult to hit. Changing one’s password often verily increases the difficulty for anyone to crack your PW,
2) Never reuse a previous PW. Be unique; by always using a new PW, you’ll never be compromised by someone simply retrying a previously used PW,
3) Randomise your PWs. By using a PW consisting of randomly selected characters, you remove yourself from essentially all dictionary attacks (and spraying),
4) Use special characters. Each special character exponentially compounds the difficulty of guessing a PW and eliminates all dictionary attacks based only on real words,
5) Longest PW possible. Each site has a different rule on PW length, take advantage of their limit and use the longest PW permitted – each extra character in a PWs exponentially increases the difficulty to crack a PW,
6) Phrases. Don’t strain your brain trying to memorise complex PWs with special characters. Using an oddly constructed phrase makes your PW extremely difficult, if not impossible to guess – “mygoldfishhasawarmembrace” is not something anyone, anywhere will ever guess, especially if you don’t own a fish,
7) Complexity. Never use a simple one-word PW – any existing word, or altered word using numbers in place of letters, can be guessed or quickly cracked. If you’re going to use real words, similar to phrases, increase your security by chaining two or more random words together like “horsetreadmillwhiteboard”
There are many other rules, look for suggestions on this site and elsewhere.
Depending on the site’s limits, I use base-95 PWs up to 128 characters in length and I change my PWs frequently – I never worry about anyone cracking my passwords, even if it’s only 8 characters long on my Netflix account.
All you can really control is you not being the cause of a password breach from a weak PW.
My last suggestion is to use an encrypted PW manager on your computer – not a plain text file. Anyone can steal and read a text file, but an encrypted DB file is going to be virtually impossible to guess as long as it’s protected using a combination of the above PW rules – don’t undo the protection provided by an encrypted PW manager by using a single word, guessable, or reused PW.
I encode words or phrases with either Base64 or MD5 and then use the result as a password
For example MD5 of “sunshinestate” is:
Password = “a3f01ff6a58987ffbb8ef6f436b5f8d0”
It would take tens of thousands of years to crack the above password . If the website hashed and salted the above password the results would be impossible to crack if stolen in a breach
I think this is a common misconception. Assuming you MD5 hash these two values:
secret=5ebe2294ecd0e0f08eab7690d2a6ee69
password=5f4dcc3b5aa765d61d8327deb882cf99
You are always given the exact same length of 32. To an attacker, I can guarantee you that this is going to be spotted immediately — and I would argue that it’s less secure because this would mean that someone would just use even more weak passwords like “cat” or “dog”, MD5 hash it, then think they’re secure…
It all boils down to what the attacker knows. In other words, if the attacker knows your password is a hash, like you suggested, they know it’s a hash of some value, but they don’t know the value. On the other had, if the attacker knows what hash you are using (again, MD5 is extremely easy to spot), then they know what you used to hash this value and what actually is needed to decode it. You can come up with dozens of online resources to decode an MD5 hash by doing a 3 second Google search.
Similarly, you can still see weak cookie values that are just MD5 hashes of some value which are used for role-based access control. Probably not nearly as common today, but it still does occur. More commonly years ago, they would hash “admin” or “user” for your role type. So, your value would be:
ee11cbb19052e40b07aac0ca060c23ee (MD5 of “user”)
What if you change it to:
21232f297a57a5a743894a0e4a801fc3 (MD5 of “admin”)
The whole point here is, don’t be under the false assumption that you’re more secure just because you MD5 encode a value. There are ways around it and it’s trivial to spot. In addition, if you thought of it, don’t you think an attacker has as well?
Except that doesn’t help for sites that require upper+lower+number+symbol
Dictionary attacks use hashed passwords as well. If you hash a simple password, you can almost guarantee that it is in the dictionary. It literally takes seconds to scan millions of hashed an unhashed passwords. Why not just use a password generator that is built into many password managers like KeepassXC? Also, if you have hundreds of passwords, are you going to remember them all or just use the same hash on each site? This is really bad security practice. The solution to the problem in the article is requiring 2FA for all employees, contractors, interns and others who many have access to the network, preferably using a hardware key.
One of the most important suggestions is missing from that list. Never use the same password for more than one site. You have no idea if a site has weak security, and if hackers can steal and decrypt it off one site, they’ll try it on others.
If your trusted organizations are irresponsible, even the best password habits amount to a hanging stainless steel gate on a split rail fence.
I used to use long passwords, but as our company keeps putting more rules on password creation, it’s gotten tough.
Must include: at least 8 characters, upper case letters, lower case letters, numbers, special characters, must not contain more than two consequtive letters (FGH, REW, CFT, QAZ type sequences), must not contain words (they must have a huge dictionary cause there’s words I never learned, and it includes names), must not contain all or part of contact phone, email or name, must not match one of the previous 30 passwords.
I just use the minimum 8 characters because it’s too hard to use a longer password with all those rules.
Having a strong password is good, but a lot of times it doesn’t matter how strong it is because attackers simply fish an easy target in the environment and dump passwords from memory. Then they pivot and do it again until they get domain admin.
MFA is more crucial than strong passwords in my mind.
Pen testing should be as common as a employee’s physical exam.
…pen testing and hunting for break-ins are two very different skills…
…they failed at hunting and detecting…
…there are only two kinds of organizations in the world – those that know they’ve been broken into, and those that don’t…don’t be the latter kind…
Any pen test is going to look for weak passwords. If they got caught by “password spraying” , they had some damn weak passwords.
Unlike brute force guessing, which tries many thousands of passwords, “password spraying” only tries a few of the most common, and likely weakest, passwords.
An IT stalwart like Citrix failing a “password spraying” hack should be unfathomable.
…weak passwords may or may not have been found by pen testing, getting in and going sideways would have been found by hunting/detecting…again, two very different skill sets…
WHO EVER IS CALLING THE SHOTS NEED TO BE FIRED. THEY SHOULD HAVE SOME IF THE BEST HACKERS TESTING THE SYSTEM WEEKLY IF NOT DAILY. WHY DO WE KNOW THE OBVIOUS AND THEY DON’T?
Seems your keyboard is defective.
And your brain, as well. This is a nearly perfect example of the Dunning–Kruger effect… when idiots don’t know enough to even realize they don’t know anything, and therefore believe themselves to be experts.
Having admin’d Citrix VPN/app/server solutions for a decade I can confidently assert none of the Citrix security issues of the last year are surprising in the least.
Also 2FA for remote access is basic common sense.
If their IT does not have 2FA implemented, they shouldn’t even work in IT…
To some extent, I get where you’re coming from.
But that’s just not how it always works in real life. Most likely, the IT professionals has recommended this (and other security recommendations) for years, and never got the funding from upper management.
I used to work for a company where management didn’t want to invest in IT, resulting not only in poor equipment but certainly also in poor security, yet the IT crowd, including myself, was always blamed for outages and breakdowns, which was why I eventually quit. Now I’m happy in my position, and while I’m not exactly happy that my former workplace is out of business, I can’t really say I’m surprised, because of the way it was (mis-)managed.
So, don’t blame IT… 🙂
Well said. Security professionals point to potential and most times “if not when” scenarios and are told this is not a priority, funding not available.
This is most common in local Governments where hackers first steal all critical data THEN ask for ransom.
It’s not always about funding. I’ve worked at places where 2FA was shot down because it would inconvenience somebody with a 3 letter title.
Please tell me that the individual’s three letters were not C, I, and O. I just want a tiny bit of solace.
Well said…
MFA is a question of money. Most companies wants to buy a solution, there are additional issues with employee password reset etc.
But even simple consumer software, like Home Assistant for the Raspberry Pi (and other platforms) comes with MFA support out of the box. Just check that you want MFA, scan the QR code, enter the MFA code from your phone, and you will have MFA.
So MFA is a free commodity, but expensive if you want to buy it.
All that aside, I think most companies has lots of technical debt, that at some point will come back and bite them hard.
For now it saves them money, but if something comes in and exploits it, it might cost way more than were saved.
As I see it, the biggest problem with IT is that it is impossible to get skilled staff. You can always hire people, but skilled people are getting rare. Especially skilled people under the age of 40-50 years.
Companies are now all running lean aka mean aka short profits over common sense. Make sure not to have a single employee too many. Do not hire freshmen out of school and give them real world training, they might switch to another employer. Do not fix anything today you can wait to fix until tomorrow. Don’t make the right solutions, they are more expensive than the fast solutions. We are all waiting for a shift towards common sense again. We can not wait forever
Why do skilled people have to be under the age of 40 to 50 years? Does it matter of their 25 or 55 if they can do the job?
Reading between the lines, I believe that the reason why you would want someone under the 40s-50s age range is so that they can be taught by the ones who are the experts, hence the older security professionals who have what it takes so they can pass that on down. To stop looking at dollars and start looking at sense (yes that was an intentional homophone) The Generation Gap needs to be filled so the leaders of tomorrow will be better equipped to help mitigate, contain, prevent, be pro-active, and be diligent.
So, you want skilled people, and are arbitrarily disqualifying the people who have been at it for the longest amount of time from your search? Yep, that makes sense. *facepalm*
I’m 63, and I’ll run circles around MOST security professionals under 40. Yeah, I’m more expensive. But you won’t pay nearly as much with my solutions.
+1
We recently had a zero day detected in a version of Citrix receiver installer file. Its unclear weather this was a false positive or not since we were not able to recover the same file to submit for further research. I’ve long suspected that Citrix would be a target for embedded malware. Who know what the extent of this really is?
After almost a week with no Citrix support, they have stopped answering there phones during this crisis. Still unable to schedule work or process docs or payments. What to do?
Honestly if you are still having that many issues I would recommend looking elsewhere. I recently switched to FileCloud and they have some of the best support out there with a main focus on security.
Check out a free trial of our data governance platform; no one is even close to what we’re doing in this area.
Gartner, G2, PCMag, IDC all rate us through the roof.
https://www-sjc.egnyte.com/corp/registration/lp-enterprise-file-sharing.html?georedirect=true
I don’t understand how such a big company can fail so hard, even we’re using since years strict password rules..
At least one Special Symbol, Numbers, Small & Big Letters and no Common Words / Names.
Also the password expires every 3 months and the account permissions are limited, if you don’t login from the local network.
Luckily we don’t use Citrix.
Strict password rules never help. They force a culture of keyboard walks. Length and entropy are the only way to go. A strict minimum of 32 characters and no more than 1 repeat of each character is way better.
People just end up using things like
1qaz2wsx!QAZ@WSX with “strict rules”
Correct.
https://securityboulevard.com/2019/03/nist-800-63-password-guidelines/
https://pages.nist.gov/800-63-3/sp800-63-3.html
I may be misunderstanding you, but it would seem 32 character minimum and no repeat characters is begging for “keyboard walk” passwords. Disallowing repeats also massively reduces entropy compared to generated passwords.
According to a search online, it says
“Citrix Systems
citrix.com
Citrix Systems, Inc. is an American multinational software company that provides server, application and desktop virtualization, networking, software as a service, and cloud computing technologies. Citrix solutions are claimed to be in use by over 400,000 clients worldwide, including 99% of the Fortune 100, and 98% of the Fortune 500.Wikipedia
Type:Public
Traded as:NASDAQ: ctxs CTXS, NASDAQ-100 Component, S&P 500 Component
Industry:Cloud computing, Computer software”
……..google……..
github – dwsagent….
cross-plat. Hybrid – In-house[stop buying enterprise is they dont know how to scale….branch….
SD-WAN [ sonic-wall if you need the extra opsec ] 2WAN
put it alllllllll – on a custom mini with a NAS-backup and how but fux endpoint..ever heard of Cobian? https://www.cobiansoft.com/ best backup utility easy…..save DOD a few bucks contract him to update it..flawless victory (or go with Datto….and RMM….pay the 3rd party & Lazy MSP…..
Now back to the inhouse Hybrid – setup a GP for entry via VPN – File server….or you can do a internal shave P2P – using DwsAgent create a custom group for those Registered with the Agent file share. so in the policy on the Agent – if your not on the Hyrid{(pull – of the DyDNS – using no-ip to mask the public address you can switch on the fly and control who has access to what and you dont have to break a tree to share…..its like Ventrillo – on the hybrid you can automate by the agents running on the back end. and basically in order to get access to the share file via https://newshare1RT1800hr.net <—– if your not under the group policy on the hybrid that gets the call – via Proxy to allow the individual access its cause they arnt allow access to that DyDNS…..which is masked. it also has a TLS2.1 back for the P2P…………..
Fux FTP, WebNAV it – and fux SMB…thats how you get jacked via SMTP keeping ports open fux that create a proxy landing zone….. God gov is slow…… HIGHER SOME GREY HATS!!!! Were ready!!!!!! sick of china and .RU making us look like we didnt start this hax thing….Tandy days Radioshack runing ford simulator
Fux Citrix – Fux even a Sonicwall to access a VPN = why use any of them create a burner hub that can be called up RT by pinging the moderator, or the Bot that send RT – verifications Text to approve or disaprove the user via 1 other in the group clicking there sms approve….you kno we have 10T now…..right….
AHHHHH the frustration its no wonder were getting rolled……. Watch Death Note…..You need L you need undergound guys that have ethics and can get dirty to out shine your White hat punks that are all fat redbulll egotistical punks that think they know it alll when they say ohhh the Node is down ping it and see where it drops…..nah momo ima pathping it and get all the info
You guys wont let in the Marines of the IT world……when people said you cant run Avermedia C127 on win10…….I DID!!!!!! When people said you can remove a RMM off apple…….I DID!!!!
When people said you cant rip youtube mp3 driect on the site without a redirect I DID….
What are you all doing in the DoD……and at DHS…EMAIL ME PUT ME ON….Im a Navy Vet – jobless doing contract work……I wana get on the front I wana help stop child sex trafficking, narcotics…and show what I go to give, and learn from what others have to teach…..
Citrix old VPN – secconect (dead tekin) let it go its over for them…..
ughhhh!!! sorry just sad to read dissapointmet after another and another cause our IT is weak cause they only hire CS-Degree dudes who cant event create a simple .bat
Cheshire147, game streaming, removing software from a Mac, ripping YouTube? Who cares. You sound like a script kiddie. Those cs dudes you denigrate, can most likely at least put a sentence together.
dude. The big brain know it all routines just make you look like a Tier 0 NOC monkey who wants to fit in with the engineers way too bad.
You obviously think word salad makes you sound knowledgeable. it does not.
Your not gonna run my last post….if you dont kno you should already kno
Your the man Kerbs on Sec!
Nothing safe anymore
A bit misleading, I expect more from a respected community member. You did not need to patch in order to be protected. The remediation steps were out and communicated before the exploit kits were released. If you ignored the remediation communication, bad on you. Oh and for more background on how fast the response actually was see here: https://www.techzine.eu/blogs/security/44687/exclusive-interview-citrix-ciso-fermin-serna-where-did-it-go-wrong/
I see nothing “misleading” about what I wrote. In regards to patches released by Citrix, it is factual. The difference between mitigations and patches is the former is “here are the steps you must take to fix/mitigate this,” versus “here, we fixed it for you.”
Thousands of Citrix (NetScaler) servers worldwide remain vulnerable to CVE-2019-19781. This critical vulnerability allows unauthenticated remote attackers to execute arbitrary commands on the targeted server – enabling them to steal credentials and install ransomware.
Our latest vulnerability scan results are freely available for authorized government CERT, ISAC, and law enforcement teams. Submit request here: https://forms.gle/eDf3DXZAv96oosfj6
You may want to think about UX, The date next to your title can be misread, e.g. 19 Hackers… I know the colour is different but not by much.
This is true and should be changed.
It’s a good thing that you were able to identify this awful flaw that the rest of us were too stupid to notice.
I just realized I’ve been using the same work email password for the past 10 years. It’s not a common one that anyone would be able to randomly guess, but still… bad me.
Brian, this is not new information. The 5-month timframe and the type of data exposed have been known since July last year.
https://www.citrix.com/blogs/2019/07/19/citrix-concludes-investigation-of-unauthorized-internal-network-access/
Citrix Receiver?
Citrix replaced that at least a year ago with Citrix Workspace.
You’re not supposed to be using Receiver anymore.
Well, receiver it’s just a name. Citrix changed the name product to Workspace App or WPA which is AKA receiver. Just FYI
Emm! Good job Brian. Interestingly, have you seen any of the articles on 2 fa breaches in the wild. And, why are you, the open u as everyone keeping important documents on a web machine? Ever heard of off web? Must be a ” B” school topic.
Or even an ” Rest of the alphabet” school topic. And remember, most of the managers didn’t go to those classes when going for their mba’s. They were taught management and budget. No wonder they didn’t see it happening. Or wonder why security didn’t catch it, it wasn’t their job.
Here is the official response from Citrix on the breach. Have a Fantastic Day! https://www.citrix.com/blogs/2019/07/19/citrix-concludes-investigation-of-unauthorized-internal-network-access/
I’m sure lots of companies have this same issue. Nothing is 100% safe so we need to focus more on offline content and ready backups.
It’s only going to get worse.
If Citrix CONCLUDED their investigation 2019-07-19, why am I now being informed about it 2020-02-10. It’s now 7 MONTHS later.
On what planet is that acceptable?
Clearly, their data governance is in question.
I’m looking at you, ShareFile users. Run. Run like the wind.
…point me at the thing that says they have to make a public announcement…oh, btw, LE can tell them to keep it quiet…
Nigga
“…a little-known security company Resecurity”…not for long. I’ve known these guys for years and they are the best. They just spend their money on analysts and not marketing. Top shelf crew.
The bad guys probably downloaded the Citrix VPN source code and found some chinks in the armor… one or more ways to subvert, or just plain remote code execute.
These resent turn of events warrants everyone completely ditching ALL Citrix products forever. Failure to do so means you don’t care, and you are ready to accept this type of incompetence as normal. These failures at Citrix are basic security 101 no-nos. Why do you think these make the light of day? Because the bottom dollar matters, nothing else.
…anyone that still uses citrix today deserves this…even 5 years ago…why would you use a horse drawn carriage…
I’m willing to bet that Citrix kept backdoor passwords to their products on the hacked network. It’s only a matter of time before the hundreds of medical offices and hospitals running Citrix products have their medical files exposed.
Another devastating breaches that could have been PREVENTED. Yeah, we know that’s it IMPOSSIBLE to suffer catastrophic and devastating breaches such as the case of Citrix without ACTIVE DIRECTORY been compromised. Hackers requires accounts or credentials, privilege or rights and access or connectivity to resources to cause cyber mayhem.
Once you’re able to place some controls around AD you’ve succeeded in disrupting, slow down and frustrates these hackers. If by hence you’re interested AD security platform that works, see this
1- Compromise Security Assessment,
https://www.smsam.net/compromise-assessment-service-csa
2- Active Directory Security Assessment, https://www.smsam.net/active-directory-security-assessment-adsa/
What’s worse with these breaches is that they are disclosed well after they have been discovered and that they are typically reported early to not be as severe as they end up being. It should be noted that any large system has many doors that might be potentially exposed and you can secure all but that one and still lose the fight. Should we expect zero failure of security, or is this simply impossible.
To that one person or persons out there who really need a true and efficient Blackhat hacker i would advice you to contact wizcyber on Wickr (might have to download wicker) or send a text message on +447537129768 he just offered me top notch services and he is capable of offering hacking services of any sort. P .s i am only doing this to help other from meeting this fake so called hackers who soil the good names of powerfull and genuine hacker. Do mention Amy when contacting him as he can be understandably wary…