TTEC, [NASDAQ: TTEC], a company used by some of the world’s largest brands to help manage customer support and sales online and over the phone, is dealing with disruptions from a network security incident resulting from a ransomware attack, KrebsOnSecurity has learned.
While many companies have been laying off or furloughing workers in response to the Coronavirus pandemic, TTEC has been massively hiring. Formerly TeleTech Holdings Inc., Englewood, Co.-based TTEC now has nearly 60,000 employees, most of whom work from home and answer customer support calls on behalf of a large number of name-brand companies, like Bank of America, Best Buy, Credit Karma, Dish Network, Kaiser Permanente, USAA and Verizon.
On Sept. 14, KrebsOnSecurity heard from a reader who passed on an internal message apparently sent by TTEC to certain employees regarding the status of a widespread system outage that began on Sunday, Sept. 12.
“We’re continuing to address the system outage impacting access to the network, applications and customer support,” reads an internal message sent by TTEC to certain employees.
TTEC has not responded to requests for comment. A phone call placed to the media contact number listed on an August 2021 TTEC earnings release produced a message saying it was a non-working number.
[Update, 6:20 p.m. ET: TTEC confirmed a ransomware attack. See the update at the end of this piece for their statement]
TTEC’s own message to employees suggests the company’s network may have been hit by the ransomware group “Ragnar Locker,” (or else by a rival ransomware gang pretending to be Ragnar). The message urged employees to avoid clicking on a file that suddenly may have appeared in their Windows start menu called “!RA!G!N!A!R!”
“DO NOT click on this file,” the notice read. “It’s a nuisance message file and we’re working on removing it from our systems.”
Ragnar Locker is an aggressive ransomware group that typically demands millions of dollars worth of cryptocurrency in ransom payments. In an announcement published on the group’s darknet leak site this week, the group threatened to publish the full data of victims who seek help from law enforcement and investigative agencies following a ransomware attack.
One of the messages texted to TTEC employees included a link to a Zoom videoconference line at ttec.zoom.us. Clicking that link opened a Zoom session in which multiple TTEC employees who were sharing their screens took turns using the company’s Global Service Desk, an internal TTEC system for tracking customer support tickets.
The TTEC employees appear to be using the Zoom conference line to report the status of various customer support teams, most of which are reporting “unable to work” at the moment.
For example, TTEC’s Service Desk reports that hundreds of TTEC employees assigned to work with Bank of America’s prepaid services are unable to work because they can’t remotely connect to TTEC’s customer service tools. More than 1,000 TTEC employees are currently unable to do their normal customer support work for Verizon, according to the Service Desk data. Hundreds of employees assigned to handle calls for Kaiser Permanente also are unable to work.
“They’ve been radio silent all week except to notify employees to take another day off,” said the source who passed on the TTEC messages, who spoke to KrebsOnSecurity on condition of anonymity. “As far as I know, all low-level employees have another day off today.”
The extent and severity of the incident at TTEC remains unknown. It is common for companies to disconnect critical systems in the event of a network intrusion, as part of a larger effort to stop the badness from spreading elsewhere. Sometimes disconnecting everything actually does help, or at least helps to keep the attack from spreading to partner networks. But it is those same connections to partner companies that raises concern in the case of TTEC’s ongoing outage.
In the meantime, if you’re unlucky enough to need to make a customer service call today, there’s a better-than-even chance you will experience….wait for it…longer-than-usual hold times.
This is a developing story. Further details or updates will be noted here with a date and time stamp.
Update, 5:37 p.m. ET: TTEC responded with the following statement:
TTEC is committed to cyber security, and to protecting the integrity of our clients’ systems and data. We recently became aware of a cybersecurity incident that has affected certain TTEC systems. Although as a result of the incident, some of our data was encrypted and business activities at several facilities have been temporarily disrupted, the company continuous to serve its global clients. TTEC immediately activated its information security incident response business continuity protocols, isolated the systems involved, and took other appropriate measures to contain the incident. We are now in the process of carefully and deliberately restoring the systems that have been involved.
We also launched an investigation, typical under the circumstances, to determine the potential impacts. In serving our clients TTEC, generally, does not maintain our clients’ data, and the investigation to date has not identified compromise to clients’ data. That investigation is on-going and we will take additional action, as appropriate, based on the investigation’s results. This is all the information we have to share until our investigation is complete.
“TTEC’s Service Desk reports that hundreds of TTEC employees assigned to work with Bank of America’s prepaid services are unable to work because they can’t remotely connect to TTEC’s customer service tools. More than 1,000 TTEC employees are currently unable to do their normal customer support work for Verizon, according to the Service Desk data. Hundreds of employees assigned to handle calls for Kaiser Permanente also are unable to work.”
Sooooo….. Customer Service as usual
Add Progressive to the list.
Your last point struck a chord with me, as I experience it more often than not from companies with whom I attempt to communicate by phone. (Yeah, I know… legacy technology.) If a company has since the pandemic began — and continues to — “experience higher than normal call volumes”, then that call volume is no longer “higher than normal” but is instead the NEW “normal”, and they should staff up accordingly.
This is a scheme to force people to use their various online “customer service” tools, which are easier for them to ignore, since they are not dependent on real-time interaction.
No, it really isn’t.
It’s really not. It’s just that people are so awful to customer service reps that no one wants to do it anymore. And I don’t blame them. I’m sure you’re one of those that starts the call with “do you know I had to wait an hour to get you”?! And then proceeds to to bitch about it for another 30 minutes which is most of the reason for long wait times. Does it even occur to you that we could handle your problem in two minutes but we can’t even get to the problem until 30 minutes in.
When we have to work a 9 hour shift with back to back calls with 4 seconds of between calls is very exhausting. Then adding callers who are angry for the wait and mad the cost of the insurance has gone up. I understand the frustration on the customer’s side just remember me I did not make your policy go up. I wish people would educate themselves on insurance. Then they would understand more and be in less shock and angry.
“They’ve been radio silent all week except to notify employees to take another day off,”
This isn’t true for USAA adjusters working for TTEC. They’ve had nearly constant zoom calls since Monday.
Yea. I was suppose to start training tomorrow. No responses from my numerous emails since a week ago. Pitiful.
Yes Janice, they should stop in the middle of a corporate-wide crisis to address all emails.
And if they don’t announce/come clean about the breach, and don’t answer queries, just how is she (or anyone else with any connection to them) supposed to know what’s happening?
Right. It’s a major ordeal going on that with the company and you’re worried about your hiring process. If this have affected the company systems when and how are they answering emails. Nothing is working…. They’re not able to be access their systems. I’ve experienced it first hand.
When I was hired in June but only have been able to work not even 3 weeks of that time and info is NOT ever provided up front, yea. Right!!!
And this is all due to mostly technical issues with them. And disorganization like I’ve never experienced in almost 45 years working.
Email has been affected Janice. That is most likely why you haven’t heard.
It’s not true for banking either. We’ve been in Zoom all week training. We have PowerPoints going all of the reps have been participating and engaged.
Can confirm. I work for them. We’re on constant standby for when the system comes back up.
We are able to get on Global Protect, and Windows but our main system Citrix we cannot get on, we are ademently told not even even open citrix.
It has effected the pay of all employees across the board. missing pay and hours. Not cool TTEC.
I agree with that even the promise pay out for this sep 15 they delay it because of this incident..not cool…
Yes and that is definitely not cool
I do know that leadership is telling us that there should be a second round of paychecks to make up for the difference since the entire timecard system was down until yesterday. But I know in our program (Bank of America), official pay day is on Fridays, even though we normally get paid before then.
i got 2 checks but still missing holiday pay and 13 hours i want my money lol
It has not affected anyone’s pay this far employees in banking are still showing up for work, logging into Zoom and getting paid for it.
Not everyone, they are completely disregarding that last shift. It’s basically only getting on zoom to says whether you’re going to use your earned vacation or are you taking the pay cut by leaving early
I am a Ttec employee. We are being paid. You don’t know what you’re talking about.
Nor affected from effected despite such storytelling skills, tsk.
(oral disapproval clicking sounds)
2 years!?! Wow – #metoo – and I would not, and cannot, at all use the word “reputable” when describing TTEC.
Until a heavy hand (not the law) punished these scum, they will continue to make life hard for millions of others.
All those “soft on computer crime” refuse to acknowledge the intent and repercussions of the crimes and call them “computer crimes” no – they effect millions of people and those scum need to be put down like a rabid dog.
I have worked there for 2 years and I assure you it’s a reputable company we can’t help that hackers got into the system are work email has been down.
Did you seriously just call TTEC “reputable”?!?! I seriously would not call them “reputable”…. and I don’t think the people in California who lost everything they owned would call TTEC “reputable” if they knew what level TTEC played in their accounts being blocked.
Brian,
Not sure if the error is yours or theirs, but you may want to add a [sic] to their reply:
They have:
Although as a result of the incident, some of our data was encrypted and business activities at several facilities have been temporarily disrupted, the company continuous[sic] to serve its global clients.
Should be:
Although as a result of the incident, some of our data was encrypted and business activities at several facilities have been temporarily disrupted, the company continue[s] to serve its global clients.
There is also an extra space between “the” and “incident” at the start of the sentence. I assume you copied and pasted it verbatim so that too is their error.
Sofa
You’re correct. It’s a copy/pasta thing. Their words, not mine.
Hi, is it possible to have a comment deleted from here? It’s using my name and I did not make a post. I just don’t know who commented. Please, if possible, I need to know how to get this taken care of.
Very clever trolling JgregoryJ /s
This could be solved by endpoint security system and as well as if they could have adopt zero trust security model. Also they would have noticed the traffic from non-reputed public IPs and as their port communication through inbound and outbound. Hopefully they implement the zero trust network security.
This seems like good feedback, but it’s so easy to arm-chair quarterback without knowing any of the actual details of what exactly is going on here.
Buzzwords that don’t necessarily even apply.
Often these companies get a false sense of security when some security product vendor tells them they’re “secure” from ransomware because they have some “nexgen firewall”, “Machine learning EDR”, “Zero Trust”, etc.
Lots of marketing fluff. Lots of people trying to make money with buzzwords.
They probably already have all of those things lmao.
I would be surprised if they didn’t already have those things along with a dedicated security operations center considering they have 60k employees. Don’t allow marketing buzzwords like “Zero trust” give you false hope that things like this can simply be averted by yet another appliance or by putting the employees entire workflow into some SaaS webapp, it won’t happen.
All you can do is make the attack surface smaller by hiring less and automating more. And I mean smaller, not eliminated.
Mrinal, and do you suppose that a zero-trust security model could be implemented overnight?
Do you have any sense of what is involved in systems that connect 60,000+ people globally, and whether all of those technologies are even modern or legacy (or rely on modern vs legacy protocols)?
It’s very easy to sling terminology around. Much harder to architect, implement and manage it.
No comment
It’s interesting to see how quiet this story has remained given the size of the company, and those companies they support.
That is because they are paying people off – otherwise they will have a massive lawsuit on their hands from every single person in California who had their unemployment accounts blocked… especially if those hundreds of thousands of people in multiple states knew that TTEC was having personal account information put into non-encrypted Share Point documents. Seriously, would you pleased if you found out that your banks 3rd party company was putting your personal information into a Share Point document along with thousands of other people that was not encrypted??
I informed my spouse 48 hours before TTEC as to why the system was down.
thank you, Mr. Brian Krebs,
for another example of a possibly fubar situation. but not thrilled to hear about it. I have at least two of those companies.
the issue for me: there is no longer one approved and preferred system of communication. ( but a multiplicity of ways, but if absolutely everything is internet based and it goes down, then there are no other options) It all seems like we have built a very fragile system and not a robust or redundant one.
It does not automatically keep flying or floating because it is too big to fail, but it fails because it is too big to manage effectively. Where’s the plan B ?
Not all programs are handling the incident the same. Ttec is doing it’s best to keep employees working. Some are in meetings, others are training. Ttec is really showing it cares about it’s employees! I don’t care if they are communicating with the public as long as they communicate with employees.
The management of TTEC that was working over the various projects handled the situation differently because they are all different in their methods of management. While some in one project did one thing others in another project did another when it came down to managing the communication to front-line employees. Many employees are already facing issues of having their utilities cut off or getting eviction notices because they are not getting a paycheck until after the dates in which they were to have their rent paid. That does not make for a stable company at all! That is dysfunctional! TL’s are losing out as well. AND, too boot – TTEC has lost projects because of their dysfunction (as they deserve).
En León Guanajuato ,México , nos hacen entrar en nuestro horario normal y después nos dicen que tomemos el día libre sin goce de sueldo de echo un directivo nos cargo unpaid systemdown hasta el día lunes 20 de septiembre , en México eso no es legal .
Al principio de que empezamos a trabajar en casa nos dijeron que si TTEC tenía problemas técnicos no nos afectaría en nuestro salario sólo nos afectaría si tuviéramos problemas con el servicio de internet o electricidad ese tiempo no se pagaría.
Así es cómo manejan sus intereses .
I stand behind Ttec 100% they have always been supportive and honest. Lets be mad at the people caused all this and place blame where it belongs. lets do something about that.
I have been with ttec for almost 6 years. I have no dought that they will do the right thing by us. It may not be at the pace everyone wants but they will make it right, they always have. And those saying your not going to get paid your hours from past pay period thats not true. And I hear they are getting everyone’s hours in now for next check. Stay tuned..I would put a confident bet that they will make it right for the employees. They did us right at the start of the pandemic, they will do us right this time too.
“They did us right at the start of the pandemic, they will do us right this time too.” How? If employees come down sick with Covid they DO NOT PAY or offer ANY help during that time. Ask me how I know….
I’m a current employee and We are not working and the company it’s not paying us even is a problem with their system, also they are not saying the true, I found it out over here!!!
What client are you working for? USAA? Bank of America? Who?
My client is fedex. They paid us for 80 whether we worked those hours or not. But please believe they did state if they overpaid us they would be taking the money back the following pay period. What money are they going to take if they are no longer paying us. Do I believe they will do right by us, Im just going to watch and see.
I work for ttec on the front lines so not getting kudos to post this..my site is handling it the best they can it seems. Maybe those who create ransomware should take the blame and get life wo messing with ours.. tell me, what else is ttec to do?
Not everyone, they are completely disregarding that last shift. It’s basically only getting on zoom to says whether you’re going to use your earned vacation or are you taking the pay cut by leaving early
TTEC has had their employee’s back. They went out of their way to pay us on time. Criminals are criminals whether they rob you at gun point or hide in darkness online sooner or later they all get careless and make a mistake!
Some shifts are out days of pay. They’re being told they can make it up but how are you going to make up a week or more of time.
Still telling us we are not able to work today.
I’m a current trainee of TTEC and I don’t see the point in blaming TTEC for a ransomware attack. I’ve worked all week during training, and got my paycheck friday. (Yes, it was short on holiday pay for my training on labor day but they already addressed it and are working towards getting that to me.) As far as I’ve seen, TTEC has been 100% upfront with their employees on what’s going on and are doing their best to fix the issues. Complaining at this point is useless, but issues like this aren’t generally just passed out to the general public until all things are known and figured out. That’s just common sense. But my trainers were sure to keep me and the others working with me informed of what was going on every step of the way so far.
I feel they should at least tell us something like..it will be at least a few more days or .we are getting close or something..
Hi, is there an update about the system? is it okay now?
Still haven’t got paid smh
Hey have anyone still not received there check from Friday? I’m still waiting to get paid
I work at ttec and haven’t been paid yet! This is so crazy
Still haven’t been paid either
I am an Employee with Ttec . I have not been paid for my hours worked. It is a shame for a sophisticated company as such, to operate in this manner.
I haven’t been paid either. We didn’t even get any answers until today. And we found out ourselves about the hack from doing our own Googling. All we were told was “This has never happened before” and “Systems have been down and payroll is effected.” We keep getting told “Maybe tomorrow you’ll be paid” USAA Deposits, BTW.
The story my team got from our Operations Manager was that there was an issue with our overuse of VPN services. I knew the story was a load of garbage, but am polite enough not to call my boss out on it in a meeting with 50 other people. So, at least know I know that it was her making stuff up instead of corporate incompetence.
What I simply cannot abide is that we were lied to when there was no reason to do so. It would have been just as acceptable to me for her to have said either “I don’t know exactly what’s going on,” or “I’m really not at liberty to discuss the reasons for the current issues.”. Sure, some people would have whined about it, but there’s no law that says the company has to tell you everything about everything. Especially given go likelihood of some dopey CSR telling a.customer about the problem.
Still, I’ve found TTEC to be neither better or worse than most BPOs I’ve worked for over the years as a whole. And I’m going to escalate my concerns about the way it was handled through the correct process.