If you sell Web-based software for a living and ship code that references an unregistered domain name, you are asking for trouble. But when the same mistake is made by a Fortune 500 company, the results can range from costly to disastrous. Here’s the story of one such goof committed by Fiserv [NASDAQ:FISV], a $15 billion firm that provides online banking software and other technology solutions to thousands of financial institutions.
In November 2020, KrebsOnSecurity heard from security researcher Abraham Vegh, who noticed something odd while inspecting an email from his financial institution.
Vegh could see the message from his bank referenced a curious domain: defaultinstitution.com. A quick search of WHOIS registration records showed the domain was unregistered. Wondering whether he might receive email communications to that address if he registered the domain, Vegh snapped it up for a few dollars, set up a catch-all email account for it, and waited.
“It appears that the domain is provided as a default, and customer bank IT departments are either assuming they don’t need to change it, or are not aware that they could/should,” Vegh said, noting that a malicious person who stumbled on his discovery earlier could have had a powerful, trusted domain from which to launch email phishing attacks.
At first, only a few wayward emails arrived. Ironically enough, one was from a “quality assurance” manager at Fiserv. The automatic reply message stated that the employee was out of the office “on R&R” and would be back to work on Dec. 14.
Many other emails poured in, including numerous “bounced” messages delivered in reply to missives from Cashedge.com, a money transfer service that Fiserv acquired in 2011.
Emails get bounced — or returned to the sender — when they are sent to an address that doesn’t exist or that is no longer active. The messages had been sent to an email address for a former client solutions director at Fiserv; the “reply-to:” address in those missives was “email@example.com”.
The messages were informing customers of CashEdge’s main service Popmoney — which lets users send, request and receive money directly from bank accounts — that Popmoney was being replaced with Zelle, a more modern bank-to-bank transfer service.
Each CashEdge missive included information about recurring transfers that were being canceled, such as the plan ID, send date, amount to be transferred, the name and last four digits of the account number the money was coming from, and the email address of the recipient account.
Incredibly, at the bottom of every message to CashEdge/Popmoney customers was a boilerplate text: “This email was sent to [recipient name here]. If you have received this email in error, please send an e-mail to firstname.lastname@example.org.”
Other services that directed customers to reply to the researcher’s domain included Fiserv customer Netspend.com, a leading provider of prepaid debit cards that require no minimum balance or credit check. The messages from Netspend all were to confirm the email address tied to a new account, and concerned “me-to-me transfers” set up through its service.
Each message included a one-time code that recipients were prompted to enter at the company’s website. But from reading the many replies to these missives, it seems Netspend didn’t make it terribly obvious where users were supposed to input this code. Here’s one of the more profane examples of a customer response:
Many others emailed by Netspend expressed mystification as to why they were receiving such messages, stating they’d never signed up for the service. From the gist of those messages, the respondents were victims of identity fraud.
“My accounts were hacked and if any funding is gone your [sic] sued from me and federal trade commission,” one wrote. “I didn’t create the account. Please stop this account and let me know what’s going on,” replied another. “I never signed up for this service. Someone else is using my information,” wrote a third.
Those messages also concerned me-to-me transfers. Other emails came from Detroit-based TCF National Bank.
New York-based Union Bank also sent customer information to the researcher’s domain. Both of those messages were intended to confirm that the recipient had tied their accounts to those at another bank. And in both cases, the recipients replied that they had not authorized the linkage.
In response to questions from KrebsOnSecurity, Fiserv acknowledged that it had inadvertently included references to defaultinstitution.com as a placeholder in software solutions used by some partners.
“We have identified 5 clients for which auto-generated emails to their customers included the domain name “defaultinstitution.com” in the “reply-to” address,” Fiserv said in a written statement. “This placeholder URL was inadvertently left unchanged during implementation of these solutions. Upon being made aware of the situation we immediately conducted an analysis to locate and replace instances of the placeholder domain name. We have also notified the clients whose customers received these emails.”
Indeed, the last email Vegh’s inbox received was on Feb. 26.
This is not the first time an oversight by Fiserv has jeopardized the security and privacy of its customers. In 2018, KrebsOnSecurity revealed how a programming weakness in a software platform sold to hundreds of banks exposed personal and financial data of countless customers. Fiserv was later sued over the matter by a credit union customer; that lawsuit is still proceeding.
Vegh said he found a similar domain goof while working as a contractor at the Federal Reserve Bank of Philadelphia back in 2015. In that instance, he discovered an unregistered domain invoked by AirWatch, a mobile device management product since acquired by VMWare.
“After registering that domain I started getting traffic from all around the world from Fortune 500 company devices pinging the domain,” Vegh said.
Vegh said he plans to give Fiserv control over defaultinstitution.com, and hand over the messages intercepted by his inbox. He’s not asking for much in return.
“I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one,” he said of his interaction with AirWatch. “This time, I am hoping to actually receive a t-shirt!”
Update, 12:44 p.m. ET: The lead paragraph has been updated to reflect Fiserv’s 2020 revenues, which were nearly $15 billion.
“Fiserv February” seems to be an annual event. That is the month Fiserv caused a data breach at my credit union. I am the CEO of the consumer owned credit union mentioned in this article that is taking Fiserv to court over it’s pervasive security failures. My heart goes out to all the other impacted consumers and institutions that have been victimized by yet another grossly irresponsible privacy blunder by Fiserv. I encourage everyone to take their banking business to a financial institution like ours that dumped Fiserv and is willing to advocate for strong consumer privacy. We look forward to getting justice for our members as part of ongoing litigation against Fiserv.
You didn’t switch to an FIS product instead, did you?
No. I said I’m an advocate for consumer privacy LOL!
What’s wrong with FIS, curious!
I am an FIS employee, and therefore a little biased… While FIS is far from perfect, the organization is fanatical about info security (sometimes to the detriment of hitting delivery timeframes and usability). We were the first processor to receive formal MRA’s from the FFIEC about 10 years ago, which was more of a shot across the bow from the regulators to our industry as a whole (in their next audits Fiserv and Jack Henry also received MRA’s). But as a response, we made a commitment to our clients and shareholders (that continues today) to be the leader in info security, and we invest more in people, technology, and governance than anyone else in the industry. From the inside, I can tell you info security is now a central part of our culture. All employees are constantly required to take courses on information security, we have ongoing phishing tests (3 strikes and your fired), and our CISO and Chief Risk Office have robust oversight and ultimate authority over all of our lines of business. Again, I am not saying FIS is perfect, but I would strongly disagree with any assertion that we are not taking security seriously (just ask our partners at the DHS Cyber Crime Center).
FIS is the only software system I have ever witnessed where timestamps went from 1 to 12, and back again with no way to distinguish AM from PM.
While it’s more typical and convenient for computers to use 24h time, that’s fine most humans are not used to 24h time anyway.
What’s decidedly NOT fine is FIS not including the AM/PM period in these timestamps! I vividly remember the elaborate heurstic hacks needed to reconstruct that information as a consumer of that information, and very much hope that bug has been fixed by now. Slow progress, indeed.
I think most humans live outside the English-speaking world.
Dear Americans, there is world outside United States. Most humans use 24h system, metric system and have price tags with included tax. Please keep your stone age to yourself.
What is an MRA?
My bank changed its online service provider and I was requested to sign off on Jack Henry’s terms of service before I could access my account.
I read the TOS and declined to accept when it said they were not responsible for the data accuracy and had nothing to do with my account but yet here there were getting ready to pass my info on from the bank to me. They also said they are cool with google analytics knowing which pages I visit. Why does google need to know where I bank?
For those of you not familiar with FiServ’s amazing security go check this site out.
This is Fiserv’s file transfer site. You download the entire portfolio and all it’s info including unencrypted Card Numbers, SSN’s etc, from here in plain text. TLS 1.0. etc
We opened tickets with FiServ on this and they have yet to find anyone on their staff that knows what TLS is. We sent them a screenshot of SSL labs and they said we were in violation of contract for “hacking” their sites.
FiServ is that f*** bad at security.
Fiserv threatened me also, when I reported the security problem with their online banking site. They didn’t want me alerting their other customers or the press about their failure. To me it seems that Fiserv prides itself on scaring security researchers and Fiserv clients into silence. Evidently they think it’s wise to use the same “intimidate the victim” playbook used by sexual abusers.
Time’s up, Fiserv! We won’t be silenced. You’ve got big problems and it’s time to start addressing them. If anyone has information related to security concerns with Fiserv, please feel free to e-mail me at email@example.com.
FWIW, Bessemer Federal Credit Union is suing Fiserv over alleged security flaws.
So much for not commenting on pending litigation!
Right? I hope Fiserv Corporate Legal Affairs is tracking the comments.
I hope Fiserv Corporate Legal Affairs is tracking the comments.
Exactly my thought.
I would not recommend posting your email here as it will get picked up by web crawlers and used to send you spam.
I used to work for Fiserv. What you have to understand is that Fiserv is just a holding company that buys up all the other companies to capitalize off of their success or shut down competition. I wound up there unfortunately by acquisition. You’d be flabbergasted how they force you to make changes that do not fit your existing infrastructure, run people off with all the domain knowledge and then assign you some Indians that have no idea of how the product works, much less work on it. They sell way more than they can deliver. It’s all about the stock price. I’m surprised there hasn’t been a colossal security failure due to their software yet.
I work at Fiserv and have contributed to OpenSSL. Perhaps I’m the exception, but at least one person at Fiserv knows what TLS 1.0 is.
It looks like they’re listening now Jose. There is a message on the site this morning that TLS 1.0 and 1.1 are disabled! I assume they will be sending you a T-shirt also LOL! Nice job! Seriously though, this isn’t the way financial institutions want to go about getting issues addressed. It would be nice if we could work as partners without threats and hostility. We’re just trying to protect our customers.
Yeah, lets just say as a former employee the new FD leadership at Fiserv doesn’t know how to run a complex data center. Also, having a MRA from the Fed doesn’t show that consumers security is first place in their mind. In general, security is an after thought at this place and they don’t have many seasoned security engineers but if they did, they wouldn’t consult with them anyways.
“I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one”
They probably sent it to defaultrecipient.com 🙂
Ha! Best thing I’ve read all day. Thanks for that.
It would have taken quite a bit of restraint on my part to not respond to that email with an equally profane response. Just for fun.
I now work as a contractor for the institution from which I retired in 2017. I’m paid by direct deposit. For literally years the deposit advice messages had a From: address of firstname.lastname@example.org.
That’s the right way to do a default, though, since that’s a specially reserved domain where email is dumped: https://www.iana.org/domains/reserved
It was literally created for this purpose.
I wouldn’t say this is the right way since you hope that it is dumped. What is stopping IANA from changing this policy in the future or accidentally registering, even for a short period of time.
The correct way, IMHO, is it should be sent to a registered address owned by the company with the server designed to receive it to just bitbucket it. This allows the company to change those emails to be actually received if need be for review.
I believe this is an issue with companies and customers of those companies looking to push a product out as quick as possible, while reducing costs. Why isn’t FISERV not requiring a full setup that changes these defaults to what the customer really wants? Why don’t customers test, validate the products they install and use? Costs and the willing to not push back at those that actually pay them is a big part. It’s a community issue just as much as it is a lazy programming trying to make a buck.
example.com is permanently registered already to the IANA so it cannot be registered.
The best way for a programmer to do it is to build validation into the automated email. When you put in an email address it will send an email and you need to enter a code to save this email as valid. I had a vendor whose software setup required it before the product could go live. I was rather impressed.
…well yes and no…
…example.com was setup for documentation, not operational systems…
…it was setup in case some idi10t clicked on the link in the document…
I’ve got stories about companies whose documentation used non-example domain names that they didn’t own.
Others who used names they didn’t own in production and couldn’t figure why some things didn’t work. Others who used a non-owned domain name, then trouble came when an acqusition required that they enable DNS internally.
…don’t get me started on people that used ip addresses that belonged to someone else on networks that “will never be connected to the Internet”…
Fiserv as a company is a total fucking joke now that First Data runs it
Fiserv is a massive company with many different products. It would be nice if Joy Peterson would identify the specific product line that had an issue. An open-ended “Fiserv” demeans everyone employed by Fiserv, which isn’t really fair. I think I know the system Bessemer runs on, and they’ve been on that system since before Fiserv acquired it. I believe they used to be a very happy client, too. Whatever the case, know that there are great Fiserv employees all over this country that take pride in their work and are extremely embarrassed by the degradation of service caused by continued layoffs of the most experienced people and the outsourcing of their work to places like India, The Philippines, and Costa Rica.
How can we find out if our banks or other financial institutions use FubarServ, I mean Fiserv?
Ask them. One word of caution though, the financial institution doesn’t have to be using Fiserv for it’s core processing for consumer’s information to be at risk. They have their hands in numerous other areas such as debit and credit card processing, online banking, bill pay, peer to peer transfers, etc. That’s what makes these issues so terrifying. It is the perfect demonstration of the saying “with great power comes great responsibility”. Unfortunately they aren’t taking their responsibility to protect American consumers seriously enough.
In years past I installed/configured bank software. Many of the institutions utilized Fiserv for processing, and some had tried to use their free software, but quickly decided it wasn’t up to task. Not sure if this is what was being used in this case or not. Testing as well as configuration and training/support was our strength because we knew many banks didn’t have the expertise. The sad thing is many smaller banks had little/no training or IT person on staff and were not following security best practices. A head teller or VP would also wear the IT hat and some reluctantly. Even when they were trying it was difficult if they lacked training. So issues with code and configuration are compounded when end users are not being properly trained.
Yes, blame the teller for issues with a bank. Blame MBA’s for not knowing programming languages. And blame the guard at the door, for lack of security training. But, who is be blamed, let’s see, the programmer who created the script, they forgot to put a hold and a note where a input should be, how about a trainer, or book that didn’t emphasize the importance of changing that input to a proper one, but, yes, read the first sentence again.
Good article Brian.
Thanks for that Jim! The whole point of paying big bucks to these vendors is that we are supposed to be paying for expertise. In fact, Fiserv even advertises their “Domain Expertise”. Google Fiserv Domain and you will see what I mean.
Not only that, they maintain dictatorial control over everything. In most cases this wouldn’t be something we would have access to change or even see on our side.
You do realize that the term “Domain Knowledge” doesn’t mean internet domains, it actually refers to knowledge of a specific, specialized discipline or field, in contrast to general knowledge, as in ACH or ATM or Bill Payment (and others) in Fiserv’s case.
Thank you for clarifying that. This seems to have turned into a Bash Fiserv site instead of looking at all the great things that are being done to provide world-class offerings for the credit unions/banks and their members/customers by the employees. Angst about operations and client satisfaction should be directed, specifically, at the leadership. Additionally, Congress. Yes, Congress passed that horrific Dodd/Frank bill that is causing these institutions to consolidate at rapid rates making less revenue for every systems and services provider out there. With less revenue, there is less room for innovation and ongoing operations.
It’s not the programmer. When you develop applications that are used by multiple clients, you use a bunch of defaults that are configurable during implementation time. See my earlier comments about how they just throw new resources on a product that has no idea.. working with other teams at Fiserv was like working with an entirely different company, because they pretty much are!
Anybody in the fin tech industry knows Fiserv is now Firs Data, the merger info was just Wall Street talk. I cant comment on what security was like before, but they brought their whole security leadership over based on a LinkedIn search. This was less than two years ago through the pandemic, so there is likely considerable restructuring happening. One can only hope positive changes based on First Data’s history.
First Data is forcefully shoving antiquated techniques down our throats and is going to end up causing more security holes in the long run. Avoid Fiserv at all costs.
I was able to observe the security team chaos after the merger, and lets just say they moved the First Data leadership to run the security teams. It has been chaos and a lack of direction since. The senior leaders are making decisions based on vendors and a non-standard cybersecurity pillar program designed by Jason Dewaz. The company has been bleeding talent since, and don’t even consult the existing talent.
IHG mandates using FISERV for credit card processing. You have no other chose in the matter. Not good!
I can tell you firsthand that the majority of Fiserv’s products follow outdated, insecure practices. It’s frankly embarrassing and they should be ashamed. I have raised concerns in the past but they get swept under the rug.
I’d say it’s less sweeping under the rug and more adding it to the backlog. These programs are thousands (sometimes hundreds of thousands) of lines of code, all of which were written by tons of different developers who either haven’t touched it in years or are no longer with the company.
It’s kind of like that saying… You can pick two:
The problem is that with Fiserv, cheap is always one of the two.
That is why we say WTFiserv
We had a catastrophe when we switched from C……. Bank to M…. Bank years ago. Our online service was down and worked intermittently for months. All because FISERV decided it was ok to go live without testing all aspects. 2009 was a clusterF.
A quick Google search (and doublechecked with a Godaddy search) just confirmed that another similar domain name, defaultbankdotcom, is available for sale. Maybe you should buy it and hold it for research purposes, Brian. 🙂
I see dumbarsebankers.com is also available.
Gary! Thank you for that, I’ll jump on it ….soon.
Does anybody know how the hell Fiserv got to be a $15 billion company??? Magic? Luck? Good genes?
Unfortunately, Fiserv is more focused on hiring “enterprise agile coaches” and “scrum masters” in order to further their rinky-dink agile agenda, instead of investing that money into something useful such as security upgrades. There has been a ramp-up in the number of outages and it’s getting worse because First Data is laying off product experts in order to funnel a TON of money into NASCAR hood logo Clover advertisements. Nice one, Fiserv.
So, someone needs to determine if any of recipients of these leaks would be covered by GDPR?
I know nothing of IT Security, but do work for a Mortgage Lender who uses FISERV and wtf is wrong with these people???!!!!
This is truly a sh*t show, how do these people stay in business?
Rant over, thanks!
Yikes – Fiserv is imploding at the seams at record rates. I reckon it’s time some federal regulators step in and pull the plug on the whole operation, to set an example that insecure low-quality rubbish software simply cannot be tolerated when it comes to sensitive customer data. Hopefully this is a wake-up call for Fiserv “leadership” and HR.
Everything is about making a release date or a revenue target. Upper management doesn’t care about the employees or the product, just about signing new clients to boost revenue.
Having worked at Fiserv before the FirstData merger, and still being there today, I can attest that security standards now are significantly more stringent than before. We do take security seriously, we do have mandates around TLS 1.2, and we do care about privacy. Troll away, y’all.
First Data is shoving this archaic “Voltage” product down our throats like it’s the second coming, but in reality it will bring us back to the Stone Age. They literally have no clue what they are doing when it comes to security.
You are right. Fiserv has taken a lots of steps towards to enhanced the security and but you can’t do anything if people don’t want to upgrades its like Iphone has launched lots of new versions and you still using Iphone4 so who will be responsible. if you are in some business clients also need to invest to upgrade them selfs.
Fiserv has gone backwards with their security “policies.” It’s only getting worse with First Data’s “our way or the highway” style of leadership.
How about Fiserv’s core pride and joy, Premier, and the requirement for Internet Explorer (ActiveX)? Using Compatibility Mode!
We were once told that bank T1 VPNs to Fiserv would be going away for wireless. Why? Because it was more secure. And they really believed that.
That’s just the tip of the ice berg. For their hosted environment, we’ve demonstrated/reported numerous ways that a single bank can access other bank’s customers’ data. So it only takes one tiny hosted bank with a tiny (or no) IT staff to be compromised then even the multi-billion $ FI’s go down, too. Wonderful. Oh yeah, and those vulnerabilities still exist, and reports have fallen on deaf ears.
They’re holding our data for ransom and making deconvesrion inaccessible.
Why would a financial institution like fidelity score an F in SSL test?!!!
Not sure, but it may be the case that it’s not their fault. If they are using networksolutions, that registrar became a joke after web.com bought them.
I had a bank that employed FIServe to handle online bill payments. Their “Cheesehead Wisconsin” support was so horribly bad that I closed my account and went looking for a bank that did not use them.
BTW – TCF is a Minnesota bank. Major jerks. Many years ago my gf missed a car payment. They took two fulltime employees and had them harass her every month multiple times a month to make sure she was going to pay, for almost two years, until the car was paid off.
I work at a bank that uses Fiserv’s services. Not only is their software extremely buggy, but we have also run into a LOT of problems with their poorly configured firewalls. There are constant denial of service attacks, and then they will add rules to the firewall that end up blocking our own customers. My team is pretty fed up with Fiserv as a whole.
I used to manage one of Fiserv’s Online Banking systems and found that their admin panel allowed me to insert HTML in fields where we could customize text. This included a script tag that let me link back to a website of my choice. This let me crawl the DOM, extract account numbers and balances and send them to any other website of my choice. (This was all just a test in a test environment.) but if I were an attacker that gained access to the system I could install something similar and harvest user information.
I bet senior Fiserv execs are reading every comment posted here.
Any company that has a relationship with a bank that holds money for consumers has to clear every single email template (any customer communication) with the bank before being sent to customers. So, netspend for instance has to clear any content with their bank previous to sending it to customers. This incident likely caused numerous TOS and regulatory violations which open up all of those financial service providers to fines from the CFPB, and perhaps law suits from customers.
“I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one”
They probably sent it to defaultrecipient.com
Krebs could write an article all about Jason Dewez. I left the company middle of last year, and I can say with the way he rules the kingdom, its no surprise the security org has been hemorrhaging talent. The man is a menace, rules by fear, and has an ego as fragile as a butterfly wing. I have never, in my 16 years of
industry experience, seen people get fired for asking questions in the wrong tone during town hall meetings.
His “open door” policy is a trap, the second you walk in that door you have a mark on your back
There are some genuinely talented engineers at Fiserv that are stuck between a rock and a hard place. Jason is the chaos of Nero and vindictiveness of Caesar all rolled into one beady eyed package.
The sad thing is that, instead of fixing obvious issues like unregistered domains, Fiserv’s security teams primarily focuses on personal reputation of leadership. I guarantee that some minion has already sent a screenshot of the above comment to Jason Dewez.
Most definitely – because they (he) cares about his reputation; not the security or success of the organization. There is perpetual nepotism, and several multi-million dollar products that have been purchased, in which he, in particular, is a board member. If this doesn’t scream pilfering the coffers, I’m not sure what does.