Tyler Technologies, a Texas-based company that bills itself as the largest provider of software and technology services to the United States public sector, is battling a network intrusion that has disrupted its operations. The company declined to discuss the exact cause of the disruption, but their response so far is straight out of the playbook for responding to ransomware incidents.
Plano, Texas-based Tyler Technologies [NYSE:TYL] has some 5,300 employees and brought in revenues of more than $1 billion in 2019. It sells a broad range of services to state and local governments, including appraisal and tax software, integrated software for courts and justice agencies, enterprise financial software systems, public safety software, records/document management software solutions and transportation software solutions for schools.
Earlier today, the normal content on tylertech.com was replaced with a notice saying the site was offline. In a statement provided to KrebsOnSecurity after the markets closed central time, Tyler Tech said early this morning the company became aware that an unauthorized intruder had gained access to its phone and information technology systems.
“Upon discovery and out of an abundance of caution, we shut down points of access to external systems and immediately began investigating and remediating the problem,” Tyler’s Chief Information Officer Matt Bieri said. “We have since engaged outside IT security and forensics experts to conduct a detailed review and help us securely restore affected equipment. We are implementing enhanced monitoring systems, and we have notified law enforcement.”
“At this time and based on the evidence available to us to-date, all indications are that the impact of this incident is limited to our internal network and phone systems,” their statement continues. “We currently have no reason to believe that any client data, client servers, or hosted systems were affected.”
While it may be comforting to hear that last bit, the reality is that it is still early in the company’s investigation. Also, ransomware has moved well past just holding a victim firm’s IT systems hostage in exchange for an extortion payment: These days, ransomware purveyors will offload as much personal and financial data that they can before unleashing their malware, and then often demand a second ransom payment in exchange for a promise to delete the stolen information or to refrain from publishing it online.
Tyler Technologies declined to say how the intrusion is affecting its customers. But several readers who work in IT roles at local government systems that rely on Tyler Tech said the outage had disrupted the ability of people to pay their water bills or court payments.
“Tyler has access to a lot of these servers in cities and counties for remote support, so it was very thoughtful of them to keep everyone in the dark and possibly exposed if the attackers made off with remote support credentials while waiting for the stock market to close,” said one reader who asked to remain anonymous.
Depending on how long it takes for Tyler to recover from this incident, it could have a broad impact on the ability of many states and localities to process payments for services or provide various government resources online.
Tyler Tech has pivoted on the threat of ransomware as a selling point for many of its services, using its presence on social media to promote ransomware survival guides and incident response checklists. With any luck, the company was following some of its own advice and will weather this storm quickly.
Update, Sept. 24, 6:00 p.m. ET: Tyler said in an updated statement on its website that a review of its logs, monitoring, traffic reports and cases related to utility and court payments revealed no outages with those systems. However, several sources interviewed for this story who work in tech roles at local governments which rely on Tyler Tech said they proactively severed their connections to Tyler Tech systems after learning about the intrusion. This is a fairly typical response for companies that outsource payment transactions to a third party when and that third party ends up experiencing a ransomware attack, but the end result (citizens unable to make payments) is the same.
Update, 11:49 p.m. ET: Tyler is now encouraging all customers to change any passwords for any remote network access for Tyler staff, after receiving reports from some customers about suspicious logins. From a statement Tyler sent to customers:
“We apologize for the late-night communications, but we wanted to pass along important information as soon as possible. We recently learned that two clients have report suspicious logins to their systems using Tyler credentials. Although we are not aware of any malicious activity on client systems and we have not been able to investigate or determine the details regarding these logins, we wanted to let you know immediately so that you can take action to protect your systems.”
I admit, I’m no expert in this area. But this statement confuses me:
“We currently have no reason to believe that any client data, client servers, or hosted systems were affected.”
If you know there is/has been an intrusion in your network and have not done all the necessary investigative work as of yet, you have reason to be concerned that client data and hosted systems may have been affected.
It’s these boiler plate, cookie-cutter responses that are so infuriating. If nothing else, just say you are very concerned and will not comment further until you have more information.
Lines like the one you highlight
“We currently have no reason to believe that any client data, client servers, or hosted systems were affected.”
are almost always meant to calm public reaction in an effort to minimize damage to the company’s public image (and stock value). They likely also hope to be able to quietly pay off any ransom demands for exfiltrated data without the public knowing about it.
Saying something like “we’re (very) concerned about data security” and/or “no further comments” in this situation is likely going to be interpreted as an admission that things are way worse than the above.
Tyler has a very strict access restriction into their hosted environments. I used to work there. The internal network is completely segregated and heavily firewalled so access is extremely limited into the hosted section of the environment.
Also, as an IT professional, its pretty easy to find impacted systems of a ransomware attack.
…all easily defeated by one successful phishing email…
qed
I’ll always opt for TRUTH over platitudes. If you don’t know, don’t speculate on the “optimistic” side.
As you probably well know, many companies use this line (or something similar) and then the truth comes out that hundreds/thousands/millions of people’s data was compromised.
Now you may be right, and their external systems may be safe. But until they KNOW that, they shouldn’t pretend they do,
IMO.
We have become a people way too complacent with liars.
“We are implementing enhanced monitoring systems…”
A little late for that now, isn’t it?
It’s not unlike businesses that make such statements as “we are implementing multi-factor authorization.” Why hadn’t they done it long ago, before the damage was done?
Where do they get their CIO, CSO and other “leaders”?
It’s comments like these that bother me… I work with Tyler a lot, they are not a perfect company, but they are a good one. It’s not Tyler’s fault that they got hacked, it’s the hacker’s fault they got hacked. It’s like blaming the victim of a gunshot wound of getting shot. Any large company IT department will tell you the same thing… It’s not a question of if you get hacked, its a matter of when you get hacked and how to respond. Obviously you mitigated the possibility as much as possible. It’s clear that Tyler had a plan in place and that they are following that plan. What more can you ask of them? Hopefully clients aren’t compromised and if they are hopefully they respond appropriately. The real test of integrity will be when the crisis is completely over. Will they be transparent to the clients on what happened.
Bad analogy to a ‘gunshot’ victim, not even close!
TYL is a bank, protect your deposits, you are RESPONSIBLE, where was your security.
Crisis is ‘never’ over , a real IT dept operates in crisis-mode with
with def-con levels of security.
Learn from the military,
C3I – Communications,Command, Control & Intelligence
All traders will short this ‘baby’ down 20 – 30%
Thanks KrebsOnSecurity
‘The TRuth is Out There’ -X-files
Their security was woefully inadequate when I left there 2 years ago. I’m talking about highly sensitive CJIS data protected by one word passwords. Remote access all over the place. Shared passwords for everything. Ransomware could put them out of business depending on where they got into. Not to mention criminal charges for loosing a bunch of CJIS data.
This is why I don’t feel sorry for companies that get hacked like this anymore. I’ve seen so many companies ignoring basic security principals because of cost, complexity or ignorance. They just take the risk thinking it won’t happen to them, until it does.
Yes, they’ll probably blame the IT guy that was warning them the whole time, but that won’t stop the millions in losses. Sooner or later, companies are going to learn that REAL security is complicated, time consuming and expensive. Realize this or suffer the consequences.
Ah, yeah… earlier in my career, I was that IT guy… got blamed for an incident, despite my repeated warnings to management regarding security issues.
Finance guy fell for phishing email, led to BEC that ultimately caused millions of dollars in losses.
User training isn’t effective if there is a culture of fear.
Threatening termination and litigation just makes people less likely to report incidents.
You can spend all sorts of money on security software, but really it is the stupid stuff that will get you. Remember Equifax? All they had to do was patch. Don’t expose ports to the internet if they should only be exposed to the lan. That would have stopped all those SMB attacks.
A nation state will hit you with a zero day or several. The cyber criminals will send you a booby trapped link or exploit software that should be patched.
Networking 101
Yes, well said. you need proactive security soluitons than a reactive one.
I used to work there 5 years ago in the hosted environment. I have friends who still work there. I can tell you one thing, they are 100 times better now then they used to be. They’ve invested HEAVILY into security. They have their own SIEM now for client data, they do rigorous penetration testing, they have multiple certified white hat hackers and ethical hackers on their staff.
They are leaps and bounds better off than they were 5 years ago.
If Tyler is “leaps and bounds” ahead and have such a cutting-edge SIEM product…why didn’t they use that product on their internal business network?
If we’re to believe that the client data is SO SAFE, why are they currently a IT Security headline?
I fully understand that nobody is perfect, but for a company that tried to sell me their security product, they should have a better track record. I’m glad we passed on Tyler’s security product.
Ok, that’s it. In my next life I’m going to be a carpenter. At least I’ll be able to sleep at night.
I wish this site had Like buttons for comments. As it is, I think I’ll come back as your apprentice.
Agree, hand crafts are so satisfying. You can see your work completed (security never is). Other people can see an appreciate the work you do, nobody really sees all the work we do.
And you can take a picture of your work and show it to others, big no no in security lol
It pays well and the stress is far lower. One of my children went into construction. After college and few years in corporate life he left and is now a home and SMB electrician.
So this next life carpenter apprenticeship is there an application or do I just get in line. ^_^
That’s one of the reasons why I am classing myself as semi-retired from IT and now work in an academic library. It’s absolutely lovely having a mostly stress-free job that makes zero demands of your time after 5pm and on weekends.
Carpenter? Didn’t one get nailed up a couple of thousand years ago? Seems like a risky profession.
That only happens when you challenge secular authority, and you’ve decided to give up your divinity…
The simpletons could have reassigned their clueless CSO to guard the staff car park. However that would have been predicate on them having any staff left after this fiasco. Truly a case of the blind leading the blind.
I’m sure Bleeping Computer will have a write up on this one tomorrow : –(
It’s already there:
https://www.bleepingcomputer.com/news/security/government-software-provider-tyler-technologies-hit-by-ransomware/
Such a crying shame they couldn’t spend any of that billion dollars on securing their customer’s resources.
One despairs. The bean-counters won’t be the ones fired, either, I bet.
I’m sure the beancounters who didn’t spend any of their billion dollars on securing their customers’ resources will not be fired. Some hapless IT guy who warned them about it will get the axe instead.
One despairs.
It is always the guy who tried to warn that gets the blame.
Those that ignored him are considered clueless, and can not be held responsible. If anything, they get a promotion, because they defended the company, and helped recover the company.
Sounds familiar, like anyone at the NSA that grows a conscience.
I can tell you point blank this isn’t true. I used to work for Tyler. Tyler spends more in IT than any company I’ve ever worked for. They have basically a blank check to buy whatever they need…and they do it. The problem is, IT professionals are not evolving as fast as the hackers are. Services available to protect against something like this are few and far between.
With the advent of work from home due to COVID, its exposing many large organizations to intrusion. There simply is no good way to protect a system from infection once it leaves the corporate network. When a worker puts it on their home network, their personal computer could already be infected. Once they connect their work computer, their personal computer can infect that machine. They connect into the corporate network via VPN or some other means and BAM, the infection gets into the internal infrastructure.
Most anti virus applications are losing ground to the hackers. There are more and more zero day threats than ever before, meaning hackers are developing new bugs every day that anti virus companies simply can’t keep up with.
I will say, as far as data breaches go, ransomware attacks are far more harmless than other types. Ransomware simply encrypts the data and has a pre-hashed key to unlock it. In most of these attacks, the attacker doesn’t actually have access to the data that gets encrypted and your data is usually still secure, its just a mater of being able to access it behind an encrypted cipher.
Tyler may be spending lavish amounts of money, but their security is still extremely poor, lax, and implemented haphazardly. That money could very well be used to train their staff.
I don’t mean to discount your experience working for them, Jon, yet these are some of the issues that we’ve been dealing with up to this day:
When setting up our server for the police department, the advice from Tyler Tech was “[d]isable the Windows firewall.” Seriously. Disable the firewall on a critical server. For the police department. I know of no better way to have your CJIS access revoked than to turn off your firewall.
And then just below the instructions to disable the firewall is the following tidbit, “If you do not want to disable the firewall, you need exceptions for the following ports.” This document was part of our workstation preparation instructions. You can view the PDF here: https://paste.c-net.org/GobletEnvious
If Tyler Tech is handing this out, what do you want to bet that when they set up a server for a client they are just going to disable the firewall since it will be less of a hassle? Plus, it’s the first thing listed, so it should be better, right?
The document wasn’t even correct with the port numbers as we still had to open over 25,000 more after further discussions with them. Well, there is over a third of the firewall right there out the window. I had tried limiting the ports, but then the Tyler Tech software doesn’t work. When I begged the “Deployment Engineer” to go to programming and see if he could get them to narrow the number of open ports, I got a brushoff. What kind of security is this?
Tyler Tech keeps disabling UAC on our computers even though I have explicitly told them not to. I’ve had to set up a group policy object to re-enable it.
Tyler Tech leaves debug files in the C:\TMP folder that contain plain-text usernames and passwords.
In that same C:\TMP folder, Tyler Tech puts registry files that they then run to disable the use of strong cryptography on the computers.
Tyler Tech uses configuration files for the SQL server that have hard-coded plain-text usernames and passwords. Not only that, but the passwords used are simple. Knowing that Tyler uses such simple passwords doesn’t give me any warm and fuzzy feelings.
Tyler Tech expects and demands that all shared folders are given full permission to the EVERYONE user. I’ve tried limiting this in our network by setting up specific users and computers that can access the network, but every time tech support gets on the server they want to add EVERYONE again.
Tyler Tech installed Apache Tomcat on one of our servers. In the three (3) years I’ve been working with Tyler, they have updated Tomcat once. Tomcat has had multiple vulnerabilities since then and I can’t get Tyler to update it any more, even with an urgent support ticket sitting in their help desk queue.
When setting up our online interface with them, Tyler Tech gave us a list of IP addresses to whitelist in our firewall. I did a reverse check on them and found that several had unrelated websites being used on them. I ended up using a single IP address that they said was the only one required. I didn’t want our network to be probed by a hacked website.
With all of these issues that we are facing, I feel that Tyler Tech is a burning mess of insecurity right now. I’m surprised it took this long for them to get hacked and I wouldn’t count on any of their customer’s data as being safe regardless of it being on another system. They don’t secure our data on our premises, and I can’t believe they would do anything more on their side.
I do not recommend their software (or services) to anyone.
We are a Tyler Customer and I concur with Bill that there are serious deficiencies in Tyler’s architecture. We too, were advised to turn off the Windows firewall on the servers. When we confronted them about this, we were told that we’d have to figure it out on our own. Seriously?
So now, after the ransom of their network, we’re told it may take as long as 4 hours, on a dedicated support call, to get our passwords changed. We tried to do it on our own, on their notification, and it totally broke the system. I can only make one conclusion, and it’s worse than not good. Hardcoded passwords, un-needed service accounts? Who knows. They are not forthcoming about anything related to security.
“Ransomware survival guides and incident response checklists” don’t seem like a sound reason for taxpayers to trust their tax dollars and private information to this firm.
I don’t think they have any say in the matter.
Our business uses some of their services (prior to Tyler buying the smaller company we utilize, not the original Tyler in TX). Their pen testing for us on vendors and internal has been nothing short of excellent. Reporting and response is top notch too.
I really hope they are as prepared as they coach others to be, and recover as fast as they would expect their customers to be able to.
It’s make it or break it time, if management let them practice what they preach.
I can tell you this, client data is FAR more secure than their internal IT. This bug is only impacting their internal infrastructure, NOT their client data. That is heavily firewalled and HEAVILY protected using multiple different services.
If you’re a customer, you should feel pretty confident that your data is secure. I should know, I used to work there 🙂
Enough. Elected officials have abdicated their responsibilities to provide record keeping and data for the public sector they are supposed to serve. Events such as the mandatory e-filing of court cases. Tyler has $1 billion dollars(2019) that could have been used by the states/counties to do their OFFICIAL JOBS. Exposing these sensitive records to TYLER and the world was and remains a mistake.
TYLER could have been hired to make the local record keeping more productive in house. TYLER is another port for private information to be distributed for all to see.
How about some consequences for TYLER and each State/County?
Whining and belly-aching aren’t sufficient.
You sound highly uneducated to how municipalities operate. You are probably the same person who complains about high taxes? Guess what…the reason Tyler is so heavily successful is because municipalities are being forced to go with a product such as there’s to cut personnel expenses. Buying Tyler’s software is far cheaper than staffing the amount of people it would take to do the same job.
Hate to break it to ya, but the same people complaining about high taxes (I would assume you are one of those people) would also complain about using an ERP solution such as this to perform the duties of dozens of people.
Sorry to break it to ya, but you should educate yourself on the processes before you comment. It makes you look uneducated and uniformed.
Municipalities operate on the ‘good ole boy’ network- you know someone/you in the family then the contracts fly by the ‘open bidding process’ into the power brokers wands to be divided out based on the ‘pay to play’ mechanisms.
Every muni has a story to tell, over the century.
Absolutely not the case, at least in the muni’s I worked for. The purchasing and bidding process is so heavily regulated and oversighted that an average IT purchase took is about fourteen months to get done. Fourteen months! Try predicting technology requirements, not to mention security requirements that far out and see how close you come.
No problem. Just remember what you are asking for when the tax increase comes around.
I worked in municipal government for almost 20 years. The reason their security stinks is that they have zero discretionary budget. Most municipalities are trying to weather these storms on tax bases that haven’t changed in fifteen years or more. They can’t even afford to hire top tier employees, much less invest in security tools.
They also offer a security service called Tyler Detect, which is supposed to help detect these types of attacks and to kill the effected service or process.
We started using Tyler Detect at the beginning of our fiscal year, July 1st, to supplement something provided through the state. So far the service from the state has stopped at least one attack before it became an issue, but Tyler Detect did not alert on it at all.
Until a federal law is in place that threatens CEOs and CSOs with a good, old-fashioned prison stretch for breaches, nothing will change. The threat of prison seems to concentrate the mind wonderfully.
How is this the CSO / CIOs fault? I take it you don’t work in IT and have never been witness to a ransomware attack. Do you know how it happens? Because average idiots open emails they shouldn’t be. They browse sites they shouldn’t be. There is only so much you can do to get around stupid people being stupid.
This was almost assuredly someone opening a file on their system they weren’t supposed to. With work remote being rampant, the IT group can’t control a system once it leaves their campus and gets placed on Karen’s network at home where anything can happen. Then, once Karen infects her machine, connects into the corporate network via VPN, the infection runs rampant. There are technologies to combat these types of attacks but they are still in their infancy and many organizations have not implemented them as of yet.
This isn’t simply a matter of a CIO or CSO saying “yeah, we are going to stop cyber attacks”. If that were the case, cyber attacks wouldn’t be a thing. Educate yourself before forming an opinion.
To be fair and I am not saying CSO and CEO is blameless but there are actual criminals who launched the attack who should go to jail. Assuming of course they are ever caught. I know of plenty of cyber related crimes that after getting reported are never followed up on by law enforcement.
So your thought is that a nation state using tools stolen from the USA’s NSA breaks into a company’s network, and the CEO and CSO should go to jail?
If the CEO should go to jail, then shouldn’t the USA President go to jail for allowing the NSA hacking tools to get out? I’m sure Trump spends his time discussing cybersecurity with the low level staff at the NSA.
Cybersecurity is like a hard drive, it’s not if it will fail, it’s just a matter of time.
RAID was invented and shared regional VM datacenters are not new either.
Good thing there are new services such as MSPOverwatch.com that are helping MSPs drink their own kool-aid and keep their own house secure. Anyone can say they are an expert at cybersecurity or provide cybersecurity solutions but who’s testing them, who’s authorizing them to say that? Where’s the proof?
These companies always claim “an abundance of caution…” after an infection. Where is it prior to the infection?
Regards,
I’m lead sysadmin for an org that uses INCODE, one of Tyler’s large accounting system packages. Zero communications from Tyler the entire day, all while our servers and clients were continuously calling out to Tyler for software updates. The updates say they failed but considering our end was attempting communications all day with a compromised network, some kind of follow-up would be fantastic. We’re treating this as if we’re compromised and doing a full review of the systems. I’m likely going to keep them offline until I hear back from Tyler.
The website is still down. It makes you wonder if the attackers were simply trying to embarrass a firm that claimed to be security conscious. Maybe they thought they would be a good mark because they would try to hide the attack; but I think it is more just the fact that these more sophisticated criminals like to put down the companies fighting them, just to get the publicity out there to make them seem invincible.
Last year, I retired, having spent 10 years working for a federal contractor at the NIH.
I’ve never heard of Tyler Technologoies. They say they’re the biggest provider? Bigger than M$, or Oracle? or IBM, or GDIT, or SAIC, or….?
They started in 1966 and are now the largest provider of software for the public sector, topping $1B in sales in 2019 –
$1B? Pardon me while I laugh.
I started working at the NIH for SRA, whose annual revenue was well over $1B, more like a number of times that amount. Then we got sold/merged with CSC-G (which CSC spun off for government work), and it was still more. Finally, we were bought by GDIT, and we’re long past $10B/yr.
Tyler is couch change.
Their primary play is the state/local market, not federal, so it makes sense that you might not have needed to interact.
MS had #140 Billion in revenue for 2019, Oracle had $40 Billion. Tyler’s $1 Billion is not in the same ballpark.
Given the uncertainties on the services this company provides to election administration around the country, is it even possible to assess the risk this poses to the current election cycle?
I have worked with them and their passwords are horrible. They use a standard password for all their customers, ie customer number with a common prefix or suffix. So if you know the template you can hack them all… I have tried to get them to change it and it is always an act of god to do it.
Can we circle back to the inherent risks of A divine carpentry profession? I perceive there is much more commentary heretofore undiscovered.
You should ask Tyler why they don’t use MFA for their Odyssey court management system. Nor do they require complex passwords (you can use as little as 3-characters) and they don’t have password expirations!
After this incident Odyssey users were asked to reset their passwords…..
Source: Friends who use the system.
Good thing there are new services such as MSPOverwatch.com that are helping MSPs drink their own kool-aid and keep their own house secure. Anyone can say they are an expert at cybersecurity or provide cybersecurity solutions but who’s testing them, who’s authorizing them to say that? Where’s the proof?
Right now it appears like Expression Engine is the best blogging platform availabe right now.
(from what I’ve read) Is that what you are using on your blog?
webpage
LnOUveyMo
Is anyone else shocked that their stock has not taken any real hit? $TYL – I’d think this would be a strong SELL for anyone paying attention, no?