March 17, 2021

If you sell Web-based software for a living and ship code that references an unregistered domain name, you are asking for trouble. But when the same mistake is made by a Fortune 500 company, the results can range from costly to disastrous. Here’s the story of one such goof committed by Fiserv [NASDAQ:FISV], a $15 billion firm that provides online banking software and other technology solutions to thousands of financial institutions.

In November 2020, KrebsOnSecurity heard from security researcher Abraham Vegh, who noticed something odd while inspecting an email from his financial institution.

Vegh could see the message from his bank referenced a curious domain: defaultinstitution.com. A quick search of WHOIS registration records showed the domain was unregistered. Wondering whether he might receive email communications to that address if he registered the domain, Vegh snapped it up for a few dollars, set up a catch-all email account for it, and waited.

“It appears that the domain is provided as a default, and customer bank IT departments are either assuming they don’t need to change it, or are not aware that they could/should,” Vegh said, noting that a malicious person who stumbled on his discovery earlier could have had a powerful, trusted domain from which to launch email phishing attacks.

At first, only a few wayward emails arrived. Ironically enough, one was from a “quality assurance” manager at Fiserv. The automatic reply message stated that the employee was out of the office “on R&R” and would be back to work on Dec. 14.

Many other emails poured in, including numerous “bounced” messages delivered in reply to missives from Cashedge.com, a money transfer service that Fiserv acquired in 2011.

Emails get bounced — or returned to the sender — when they are sent to an address that doesn’t exist or that is no longer active. The messages had been sent to an email address for a former client solutions director at Fiserv; the “reply-to:” address in those missives was “donotreply@defaultinstitution.com”.

The messages were informing customers of CashEdge’s main service Popmoney — which lets users send, request and receive money directly from bank accounts — that Popmoney was being replaced with Zelle, a more modern bank-to-bank transfer service.

Each CashEdge missive included information about recurring transfers that were being canceled, such as the plan ID, send date, amount to be transferred, the name and last four digits of the account number the money was coming from, and the email address of the recipient account.

Incredibly, at the bottom of every message to CashEdge/Popmoney customers was a boilerplate text: “This email was sent to [recipient name here]. If you have received this email in error, please send an e-mail to customersupport@defaultinstitution.com.”

Other services that directed customers to reply to the researcher’s domain included Fiserv customer Netspend.com, a leading provider of prepaid debit cards that require no minimum balance or credit check. The messages from Netspend all were to confirm the email address tied to a new account, and concerned “me-to-me transfers” set up through its service.

Each message included a one-time code that recipients were prompted to enter at the company’s website. But from reading the many replies to these missives, it seems Netspend didn’t make it terribly obvious where users were supposed to input this code. Here’s one of the more profane examples of a customer response:

Many others emailed by Netspend expressed mystification as to why they were receiving such messages, stating they’d never signed up for the service. From the gist of those messages, the respondents were victims of identity fraud.

“My accounts were hacked and if any funding is gone your [sic] sued from me and federal trade commission,” one wrote. “I didn’t create the account. Please stop this account and let me know what’s going on,” replied another. “I never signed up for this service. Someone else is using my information,” wrote a third.

Those messages also concerned me-to-me transfers. Other emails came from Detroit-based TCF National Bank.

New York-based Union Bank also sent customer information to the researcher’s domain. Both of those messages were intended to confirm that the recipient had tied their accounts to those at another bank. And in both cases, the recipients replied that they had not authorized the linkage.

In response to questions from KrebsOnSecurity, Fiserv acknowledged that it had inadvertently included references to defaultinstitution.com as a placeholder in software solutions used by some partners.

“We have identified 5 clients for which auto-generated emails to their customers included the domain name “defaultinstitution.com” in the “reply-to” address,” Fiserv said in a written statement. “This placeholder URL was inadvertently left unchanged during implementation of these solutions. Upon being made aware of the situation we immediately conducted an analysis to locate and replace instances of the placeholder domain name. We have also notified the clients whose customers received these emails.”

Indeed, the last email Vegh’s inbox received was on Feb. 26.

This is not the first time an oversight by Fiserv has jeopardized the security and privacy of its customers. In 2018, KrebsOnSecurity revealed how a programming weakness in a software platform sold to hundreds of banks exposed personal and financial data of countless customers. Fiserv was later sued over the matter by a credit union customer; that lawsuit is still proceeding.

Vegh said he found a similar domain goof while working as a contractor at the Federal Reserve Bank of Philadelphia back in 2015. In that instance, he discovered an unregistered domain invoked by AirWatch, a mobile device management product since acquired by VMWare.

“After registering that domain I started getting traffic from all around the world from Fortune 500 company devices pinging the domain,” Vegh said.

Vegh said he plans to give Fiserv control over defaultinstitution.com, and hand over the messages intercepted by his inbox. He’s not asking for much in return.

“I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one,” he said of his interaction with AirWatch. “This time, I am hoping to actually receive a t-shirt!”

Update, 12:44 p.m. ET: The lead paragraph has been updated to reflect Fiserv’s 2020 revenues, which were nearly $15 billion.


77 thoughts on “Fintech Giant Fiserv Used Unclaimed Domain

  1. Joy Peterson

    “Fiserv February” seems to be an annual event. That is the month Fiserv caused a data breach at my credit union. I am the CEO of the consumer owned credit union mentioned in this article that is taking Fiserv to court over it’s pervasive security failures. My heart goes out to all the other impacted consumers and institutions that have been victimized by yet another grossly irresponsible privacy blunder by Fiserv. I encourage everyone to take their banking business to a financial institution like ours that dumped Fiserv and is willing to advocate for strong consumer privacy. We look forward to getting justice for our members as part of ongoing litigation against Fiserv.

    Reply
    1. Former Bank IT

      You didn’t switch to an FIS product instead, did you?

      Reply
            1. Mr. D

              I am an FIS employee, and therefore a little biased… While FIS is far from perfect, the organization is fanatical about info security (sometimes to the detriment of hitting delivery timeframes and usability). We were the first processor to receive formal MRA’s from the FFIEC about 10 years ago, which was more of a shot across the bow from the regulators to our industry as a whole (in their next audits Fiserv and Jack Henry also received MRA’s). But as a response, we made a commitment to our clients and shareholders (that continues today) to be the leader in info security, and we invest more in people, technology, and governance than anyone else in the industry. From the inside, I can tell you info security is now a central part of our culture. All employees are constantly required to take courses on information security, we have ongoing phishing tests (3 strikes and your fired), and our CISO and Chief Risk Office have robust oversight and ultimate authority over all of our lines of business. Again, I am not saying FIS is perfect, but I would strongly disagree with any assertion that we are not taking security seriously (just ask our partners at the DHS Cyber Crime Center).

              Reply
              1. Anonymous

                FIS is the only software system I have ever witnessed where timestamps went from 1 to 12, and back again with no way to distinguish AM from PM.

                While it’s more typical and convenient for computers to use 24h time, that’s fine most humans are not used to 24h time anyway.

                What’s decidedly NOT fine is FIS not including the AM/PM period in these timestamps! I vividly remember the elaborate heurstic hacks needed to reconstruct that information as a consumer of that information, and very much hope that bug has been fixed by now. Slow progress, indeed.

                Reply
                1. VasyaPups

                  Dear Americans, there is world outside United States. Most humans use 24h system, metric system and have price tags with included tax. Please keep your stone age to yourself.

                  Reply
              2. jason

                What is an MRA?
                My bank changed its online service provider and I was requested to sign off on Jack Henry’s terms of service before I could access my account.

                I read the TOS and declined to accept when it said they were not responsible for the data accuracy and had nothing to do with my account but yet here there were getting ready to pass my info on from the bank to me. They also said they are cool with google analytics knowing which pages I visit. Why does google need to know where I bank?

                Reply
    2. Jose

      For those of you not familiar with FiServ’s amazing security go check this site out.
      https://www.ssllabs.com/ssltest/analyze.html?d=www.mft1.firstdataclients.com

      This is Fiserv’s file transfer site. You download the entire portfolio and all it’s info including unencrypted Card Numbers, SSN’s etc, from here in plain text. TLS 1.0. etc

      We opened tickets with FiServ on this and they have yet to find anyone on their staff that knows what TLS is. We sent them a screenshot of SSL labs and they said we were in violation of contract for “hacking” their sites.

      FiServ is that f*** bad at security.

      Reply
      1. Joy Peterson

        Fiserv threatened me also, when I reported the security problem with their online banking site. They didn’t want me alerting their other customers or the press about their failure. To me it seems that Fiserv prides itself on scaring security researchers and Fiserv clients into silence. Evidently they think it’s wise to use the same “intimidate the victim” playbook used by sexual abusers.

        Time’s up, Fiserv! We won’t be silenced. You’ve got big problems and it’s time to start addressing them. If anyone has information related to security concerns with Fiserv, please feel free to e-mail me at jpeterson@bessfcu.com.

        Reply
        1. Edward Wong

          FWIW, Bessemer Federal Credit Union is suing Fiserv over alleged security flaws.

          Reply
          1. Keoni

            Right? I hope Fiserv Corporate Legal Affairs is tracking the comments.

            Reply
          2. Keoni

            I hope Fiserv Corporate Legal Affairs is tracking the comments.

            Reply
        2. BB

          I would not recommend posting your email here as it will get picked up by web crawlers and used to send you spam.

          Reply
      2. JDub

        I used to work for Fiserv. What you have to understand is that Fiserv is just a holding company that buys up all the other companies to capitalize off of their success or shut down competition. I wound up there unfortunately by acquisition. You’d be flabbergasted how they force you to make changes that do not fit your existing infrastructure, run people off with all the domain knowledge and then assign you some Indians that have no idea of how the product works, much less work on it. They sell way more than they can deliver. It’s all about the stock price. I’m surprised there hasn’t been a colossal security failure due to their software yet.

        Reply
      3. Eugene Smith

        I work at Fiserv and have contributed to OpenSSL. Perhaps I’m the exception, but at least one person at Fiserv knows what TLS 1.0 is.

        Reply
      4. Joy Peterson

        It looks like they’re listening now Jose. There is a message on the site this morning that TLS 1.0 and 1.1 are disabled! I assume they will be sending you a T-shirt also LOL! Nice job! Seriously though, this isn’t the way financial institutions want to go about getting issues addressed. It would be nice if we could work as partners without threats and hostility. We’re just trying to protect our customers.

        Reply
    3. Cpt America

      Yeah, lets just say as a former employee the new FD leadership at Fiserv doesn’t know how to run a complex data center. Also, having a MRA from the Fed doesn’t show that consumers security is first place in their mind. In general, security is an after thought at this place and they don’t have many seasoned security engineers but if they did, they wouldn’t consult with them anyways.

      Reply
  2. Face Palm

    “I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one”

    They probably sent it to defaultrecipient.com 🙂

    Reply
  3. Gob Bluth

    It would have taken quite a bit of restraint on my part to not respond to that email with an equally profane response. Just for fun.

    Reply
  4. Bob Brown

    I now work as a contractor for the institution from which I retired in 2017. I’m paid by direct deposit. For literally years the deposit advice messages had a From: address of payables@example.com.

    Reply
      1. CPJ

        I wouldn’t say this is the right way since you hope that it is dumped. What is stopping IANA from changing this policy in the future or accidentally registering, even for a short period of time.

        The correct way, IMHO, is it should be sent to a registered address owned by the company with the server designed to receive it to just bitbucket it. This allows the company to change those emails to be actually received if need be for review.

        I believe this is an issue with companies and customers of those companies looking to push a product out as quick as possible, while reducing costs. Why isn’t FISERV not requiring a full setup that changes these defaults to what the customer really wants? Why don’t customers test, validate the products they install and use? Costs and the willing to not push back at those that actually pay them is a big part. It’s a community issue just as much as it is a lazy programming trying to make a buck.

        Reply
        1. masterX244

          example.com is permanently registered already to the IANA so it cannot be registered.

          Reply
        2. Rob

          The best way for a programmer to do it is to build validation into the automated email. When you put in an email address it will send an email and you need to enter a code to save this email as valid. I had a vendor whose software setup required it before the product could go live. I was rather impressed.

          Reply
      2. security vet

        …well yes and no…

        …example.com was setup for documentation, not operational systems…

        …it was setup in case some idi10t clicked on the link in the document…

        Reply
      3. John

        I’ve got stories about companies whose documentation used non-example domain names that they didn’t own.
        Others who used names they didn’t own in production and couldn’t figure why some things didn’t work. Others who used a non-owned domain name, then trouble came when an acqusition required that they enable DNS internally.

        Reply
        1. security vet

          …don’t get me started on people that used ip addresses that belonged to someone else on networks that “will never be connected to the Internet”…

          Reply
  5. The Joker

    Fiserv as a company is a total fucking joke now that First Data runs it

    Reply
    1. Keoni Mokulele

      Fiserv is a massive company with many different products. It would be nice if Joy Peterson would identify the specific product line that had an issue. An open-ended “Fiserv” demeans everyone employed by Fiserv, which isn’t really fair. I think I know the system Bessemer runs on, and they’ve been on that system since before Fiserv acquired it. I believe they used to be a very happy client, too. Whatever the case, know that there are great Fiserv employees all over this country that take pride in their work and are extremely embarrassed by the degradation of service caused by continued layoffs of the most experienced people and the outsourcing of their work to places like India, The Philippines, and Costa Rica.

      Reply
  6. J

    How can we find out if our banks or other financial institutions use FubarServ, I mean Fiserv?

    Reply
    1. Joy Peterson

      Ask them. One word of caution though, the financial institution doesn’t have to be using Fiserv for it’s core processing for consumer’s information to be at risk. They have their hands in numerous other areas such as debit and credit card processing, online banking, bill pay, peer to peer transfers, etc. That’s what makes these issues so terrifying. It is the perfect demonstration of the saying “with great power comes great responsibility”. Unfortunately they aren’t taking their responsibility to protect American consumers seriously enough.

      Reply
  7. D

    In years past I installed/configured bank software. Many of the institutions utilized Fiserv for processing, and some had tried to use their free software, but quickly decided it wasn’t up to task. Not sure if this is what was being used in this case or not. Testing as well as configuration and training/support was our strength because we knew many banks didn’t have the expertise. The sad thing is many smaller banks had little/no training or IT person on staff and were not following security best practices. A head teller or VP would also wear the IT hat and some reluctantly. Even when they were trying it was difficult if they lacked training. So issues with code and configuration are compounded when end users are not being properly trained.

    Reply
  8. Jim

    Yes, blame the teller for issues with a bank. Blame MBA’s for not knowing programming languages. And blame the guard at the door, for lack of security training. But, who is be blamed, let’s see, the programmer who created the script, they forgot to put a hold and a note where a input should be, how about a trainer, or book that didn’t emphasize the importance of changing that input to a proper one, but, yes, read the first sentence again.

    Good article Brian.

    Reply
    1. Joy Peterson

      Thanks for that Jim! The whole point of paying big bucks to these vendors is that we are supposed to be paying for expertise. In fact, Fiserv even advertises their “Domain Expertise”. Google Fiserv Domain and you will see what I mean.

      Not only that, they maintain dictatorial control over everything. In most cases this wouldn’t be something we would have access to change or even see on our side.

      Reply
      1. Odinn

        You do realize that the term “Domain Knowledge” doesn’t mean internet domains, it actually refers to knowledge of a specific, specialized discipline or field, in contrast to general knowledge, as in ACH or ATM or Bill Payment (and others) in Fiserv’s case.

        Reply
        1. Keoni Mokulele

          Thank you for clarifying that. This seems to have turned into a Bash Fiserv site instead of looking at all the great things that are being done to provide world-class offerings for the credit unions/banks and their members/customers by the employees. Angst about operations and client satisfaction should be directed, specifically, at the leadership. Additionally, Congress. Yes, Congress passed that horrific Dodd/Frank bill that is causing these institutions to consolidate at rapid rates making less revenue for every systems and services provider out there. With less revenue, there is less room for innovation and ongoing operations.

          Reply
    2. JDub

      It’s not the programmer. When you develop applications that are used by multiple clients, you use a bunch of defaults that are configurable during implementation time. See my earlier comments about how they just throw new resources on a product that has no idea.. working with other teams at Fiserv was like working with an entirely different company, because they pretty much are!

      Reply
  9. Tony Stark

    Anybody in the fin tech industry knows Fiserv is now Firs Data, the merger info was just Wall Street talk. I cant comment on what security was like before, but they brought their whole security leadership over based on a LinkedIn search. This was less than two years ago through the pandemic, so there is likely considerable restructuring happening. One can only hope positive changes based on First Data’s history.

    Reply
    1. George L.

      First Data is forcefully shoving antiquated techniques down our throats and is going to end up causing more security holes in the long run. Avoid Fiserv at all costs.

      Reply
    2. Cpt America

      Hey Tony,

      I was able to observe the security team chaos after the merger, and lets just say they moved the First Data leadership to run the security teams. It has been chaos and a lack of direction since. The senior leaders are making decisions based on vendors and a non-standard cybersecurity pillar program designed by Jason Dewaz. The company has been bleeding talent since, and don’t even consult the existing talent.

      Reply
  10. JasonT

    IHG mandates using FISERV for credit card processing. You have no other chose in the matter. Not good!

    Reply
  11. George L.

    I can tell you firsthand that the majority of Fiserv’s products follow outdated, insecure practices. It’s frankly embarrassing and they should be ashamed. I have raised concerns in the past but they get swept under the rug.

    Reply
    1. Anon

      I’d say it’s less sweeping under the rug and more adding it to the backlog. These programs are thousands (sometimes hundreds of thousands) of lines of code, all of which were written by tons of different developers who either haven’t touched it in years or are no longer with the company.

      It’s kind of like that saying… You can pick two:

      Fast
      Cheap
      Quality

      The problem is that with Fiserv, cheap is always one of the two.

      Reply
  12. Liz

    We had a catastrophe when we switched from C……. Bank to M…. Bank years ago. Our online service was down and worked intermittently for months. All because FISERV decided it was ok to go live without testing all aspects. 2009 was a clusterF.

    Reply
  13. DelilahTheSober

    A quick Google search (and doublechecked with a Godaddy search) just confirmed that another similar domain name, defaultbankdotcom, is available for sale. Maybe you should buy it and hold it for research purposes, Brian. 🙂

    Reply
      1. Lindy

        Gary! Thank you for that, I’ll jump on it ….soon.

        Does anybody know how the hell Fiserv got to be a $15 billion company??? Magic? Luck? Good genes?

        Reply
  14. Joseph

    Unfortunately, Fiserv is more focused on hiring “enterprise agile coaches” and “scrum masters” in order to further their rinky-dink agile agenda, instead of investing that money into something useful such as security upgrades. There has been a ramp-up in the number of outages and it’s getting worse because First Data is laying off product experts in order to funnel a TON of money into NASCAR hood logo Clover advertisements. Nice one, Fiserv.

    Reply
  15. Global Crosser

    So, someone needs to determine if any of recipients of these leaks would be covered by GDPR?

    Reply
  16. Lwood

    I know nothing of IT Security, but do work for a Mortgage Lender who uses FISERV and wtf is wrong with these people???!!!!

    This is truly a sh*t show, how do these people stay in business?

    Rant over, thanks!

    Reply
  17. Concerned Employee

    Yikes – Fiserv is imploding at the seams at record rates. I reckon it’s time some federal regulators step in and pull the plug on the whole operation, to set an example that insecure low-quality rubbish software simply cannot be tolerated when it comes to sensitive customer data. Hopefully this is a wake-up call for Fiserv “leadership” and HR.

    Reply
  18. Turd Furguson

    Everything is about making a release date or a revenue target. Upper management doesn’t care about the employees or the product, just about signing new clients to boost revenue.

    Reply
  19. Fiserv Employee

    Having worked at Fiserv before the FirstData merger, and still being there today, I can attest that security standards now are significantly more stringent than before. We do take security seriously, we do have mandates around TLS 1.2, and we do care about privacy. Troll away, y’all.

    Reply
    1. George L.

      First Data is shoving this archaic “Voltage” product down our throats like it’s the second coming, but in reality it will bring us back to the Stone Age. They literally have no clue what they are doing when it comes to security.

      Reply
    2. RIght

      You are right. Fiserv has taken a lots of steps towards to enhanced the security and but you can’t do anything if people don’t want to upgrades its like Iphone has launched lots of new versions and you still using Iphone4 so who will be responsible. if you are in some business clients also need to invest to upgrade them selfs.

      Reply
      1. Wrong

        Fiserv has gone backwards with their security “policies.” It’s only getting worse with First Data’s “our way or the highway” style of leadership.

        Reply
  20. Another Unsatisfied Fiserv Bank

    How about Fiserv’s core pride and joy, Premier, and the requirement for Internet Explorer (ActiveX)? Using Compatibility Mode!

    We were once told that bank T1 VPNs to Fiserv would be going away for wireless. Why? Because it was more secure. And they really believed that.

    That’s just the tip of the ice berg. For their hosted environment, we’ve demonstrated/reported numerous ways that a single bank can access other bank’s customers’ data. So it only takes one tiny hosted bank with a tiny (or no) IT staff to be compromised then even the multi-billion $ FI’s go down, too. Wonderful. Oh yeah, and those vulnerabilities still exist, and reports have fallen on deaf ears.

    They’re holding our data for ransom and making deconvesrion inaccessible.

    Reply
    1. kevin

      Not sure, but it may be the case that it’s not their fault. If they are using networksolutions, that registrar became a joke after web.com bought them.

      Reply
  21. Kevin

    I had a bank that employed FIServe to handle online bill payments. Their “Cheesehead Wisconsin” support was so horribly bad that I closed my account and went looking for a bank that did not use them.

    BTW – TCF is a Minnesota bank. Major jerks. Many years ago my gf missed a car payment. They took two fulltime employees and had them harass her every month multiple times a month to make sure she was going to pay, for almost two years, until the car was paid off.

    Reply
  22. SSB

    I work at a bank that uses Fiserv’s services. Not only is their software extremely buggy, but we have also run into a LOT of problems with their poorly configured firewalls. There are constant denial of service attacks, and then they will add rules to the firewall that end up blocking our own customers. My team is pretty fed up with Fiserv as a whole.

    Reply
  23. Former bank online banking admin

    I used to manage one of Fiserv’s Online Banking systems and found that their admin panel allowed me to insert HTML in fields where we could customize text. This included a script tag that let me link back to a website of my choice. This let me crawl the DOM, extract account numbers and balances and send them to any other website of my choice. (This was all just a test in a test environment.) but if I were an attacker that gained access to the system I could install something similar and harvest user information.

    Reply
  24. Dharmadhrt

    I bet senior Fiserv execs are reading every comment posted here.

    Reply
  25. QuickPayPortal

    Any company that has a relationship with a bank that holds money for consumers has to clear every single email template (any customer communication) with the bank before being sent to customers. So, netspend for instance has to clear any content with their bank previous to sending it to customers. This incident likely caused numerous TOS and regulatory violations which open up all of those financial service providers to fines from the CFPB, and perhaps law suits from customers.

    Reply
  26. Paymydoctor

    “I had been promised a t-shirt and a case of beer for my efforts then, but alas, never received one”

    They probably sent it to defaultrecipient.com

    Reply
  27. Ripstun

    Krebs could write an article all about Jason Dewez. I left the company middle of last year, and I can say with the way he rules the kingdom, its no surprise the security org has been hemorrhaging talent. The man is a menace, rules by fear, and has an ego as fragile as a butterfly wing. I have never, in my 16 years of

    industry experience, seen people get fired for asking questions in the wrong tone during town hall meetings.

    His “open door” policy is a trap, the second you walk in that door you have a mark on your back

    There are some genuinely talented engineers at Fiserv that are stuck between a rock and a hard place. Jason is the chaos of Nero and vindictiveness of Caesar all rolled into one beady eyed package.

    Reply
    1. Eugene Smith

      The sad thing is that, instead of fixing obvious issues like unregistered domains, Fiserv’s security teams primarily focuses on personal reputation of leadership. I guarantee that some minion has already sent a screenshot of the above comment to Jason Dewez.

      Reply
      1. Ripstun

        Most definitely – because they (he) cares about his reputation; not the security or success of the organization. There is perpetual nepotism, and several multi-million dollar products that have been purchased, in which he, in particular, is a board member. If this doesn’t scream pilfering the coffers, I’m not sure what does.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *