Fiserv, Inc., a major provider of technology services to financial institutions, just fixed a glaring weakness in its Web platform that exposed personal and financial details of countless customers across hundreds of bank Web sites, KrebsOnSecurity has learned.
Brookfield, Wisc.-based Fiserv [NASDAQ:FISV] is a Fortune 500 company with 24,000 employees and $5.7 billion in earnings last year. Its account and transaction processing systems power the Web sites for hundreds of financial institutions — mostly small community banks and credit unions. According to FedFis.com, Fiserv is by far the top bank core processor, with more than 37 percent market share.
Two weeks ago this author heard from security researcher Kristian Erik Hermansen, who said he’d discovered something curious while logged in to an account at a tiny local bank that uses Fiserv’s platform.
Hermansen had signed up to get email alerts any time a new transaction posted to his account, and he noticed the site assigned his alert a specific “event number.” Working on a hunch that these event numbers might be assigned sequentially and that other records might be available if requested directly, Hermansen requested the same page again but first edited the site’s code in his browser so that his event number was decremented by one digit.
In an instant, he could then view and edit alerts previously set up by another bank customer, and could see that customer’s email address, phone number and full bank account number.
Hermansen said a cybercriminal could abuse this access to enumerate all other accounts with activity alerts on file, and to add or delete phone numbers or email addresses to receive alerts about account transactions.
This would allow any customer of the bank to spy on the daily transaction activity of other customers, and perhaps even target customers who signed up for high minimum balance alerts (e.g., “alert me when the available balance goes below $5,000”).
“I shouldn’t be able to see this data,” Hermansen said. “Anytime you spend money that should be a private transaction between you and your bank, not available for everyone else to see.”
Hermansen said he told his bank about what he found, and that he tried unsuccessfully to get the attention of different Fiserv employees, including the company’s CEO via LinkedIn. But he wasn’t sure whether the flaw he found existed in all bank sites running on Fiserv’s ebanking platform, or just his bank’s installation.
Naturally, KrebsOnSecurity offered to help figure that out, and to get Fiserv’s attention, if warranted. Over the past week I signed up for accounts at two small local banks that each use Fiserv’s online banking platform.
In both cases I was able to replicate Hermansen’s findings and view email addresses, phone numbers, partial account numbers and alert details for other customers of each bank just by editing a single digit in a Web page request. I was relieved to find I could not use my online account access at one bank to view transaction alerts I’d set up at a different Fiserv affiliated bank.
A single digit changed in a Web browser request caused someone else’s alerts to pop up in my account at this small local bank in Virginia.
But it was not difficult to find hundreds of other Fiserv-affiliated banks that would be just as vulnerable. If a bank is using Fiserv’s platform, it usually says so somewhere at the bottom of the bank’s home page. Another giveaway is that most of the bank sites using Fiserv display the same root domain name in the browser address bar after login: secureinternetbank.com.
Fiserv said in a statement that the problem stemmed from an issue with “a messaging solution available to a subset of online banking clients.” Fiserv declined to say exactly how many financial institutions may have been impacted overall. But experts tell KrebsOnSecurity that some 1,700 banks currently use Fiserv’s retail (consumer-focused) banking platform alone.
“Fiserv places a high priority on security, and we have responded accordingly,” Fiserv spokesperson Ann Cave said. “After receiving your email, we promptly engaged appropriate resources and worked around the clock to research and remediate the situation. We developed a security patch within 24 hours of receiving notification and deployed the patch to clients that utilize a hosted version of the solution. We will be deploying the patch this evening to clients that utilize an in-house version of the solution.”
This author confirmed that Fiserv no longer shows a sequential event number in their banking sites and has replaced them with a pseudo-random string.
Julie Conroy, research director with market analyst firm Aite Group, said the kinds of banks that use Fiserv’s platform mostly are those that can’t afford to build and maintain their own.
“These financial institutions use a core banking provider like Fiserv because they don’t have the wherewithal to do it on their own, so they’re really trusting Fiserv to do this on their behalf,” Conroy said. “This will not only reflect on Fiserv’s brand, but also it will impact customer’s perception about their small local bank, which is already struggling to compete with the larger, nationwide institutions.”
Allen Weinberg, partner and co-founder at Glenbrook Partners LLC, said the ability of fraudsters to edit account transaction alerts somewhat negates the value of these alerts in helping consumers fight fraud tied to their online banking accounts.
“If a fraudster can just turn off the alerts, there’s one less protection that consumers think they have,” Weinberg said. “I think consumers do rely in large part on these alerts to help them detect fraudulent activity.”
The weaknesses in Fiserv’s platform is what’s known as an “information disclosure” vulnerability. While these are among the most common types of security issues with Web sites, they are also perhaps the most preventable and easily fixed.
Nevertheless, disclosure flaws can be just as damaging to a company’s brand as other more severe types of security errors. Other notable security incidents involving recent information disclosure issues include a weakness at Panera Bread’s site that exposed tens of millions of customer records, and a bug in identity protection service LifeLock’s site that revealed email addresses for millions of customers.
Update, 12:22 p.m. ET: Updated the second paragraph to include a link to information about Fiserv’s #1 market share. Also updated the story to reflect screen shots shared with KrebsOnSecurity indicating Hermansen was able to see full customer account numbers in his bank’s implementation of Fiserv’s platform. The account numbers I saw in both Virginia banks used in my testing were truncated to the last four digits.
Great article
This is what you get when you hire the cheapest bidder. Doesn’t take long to figure out where Fiserv outsources their development. No experienced developer would do this, and no inexperienced developer should be writing code to support a banking application.
You appear to be making unverified assertions. I can’t prove otherwise, but it would be interesting to see proof backing up at least the claim of “it doesn’t take long”.
Also, a low bidder doesn’t necessarily mean Worst Resourced bidder, a low bidder could easily be a large profitable efficient operation that is able to spread its operating costs over its large business base such that it has competitively low prices.
It is worth noting that at a certain point, individual smaller institutions would not be able to create and maintain performance and cost competitive and secure online platform. For them this essential but non core operation would have to be outsourced to achieve these three things. Indeed, both of my credit unions, one large and one small are outsourced on the Jack Henry platform.
Fiserv isn’t the cheapest bidder. They are one of the top 3 outsourced systems for community banks. They’re actually priced toward the higher side.
fiserv is a joke.
I thought it was pretty obvious, but my comment was in reference to their using inexpensive developers from less developed countries to write their banking applications.
Generally speaking, Matt is right and the proof is in the pudding.
Companies must understand that they have a responsibility to triage and treat every security infomation they get responsibly. So if krebs wasn’t involved, they would probably have paid little attention to this or not even take the researcher or the information seriously.
I had exactly the same thought. I’ve complained to my bank about security issues and they never respond constructively. I just get the boilerplate apology email with a link to their FAQ.
THIS!
Many times when I report a website malfunction, I get denial and/or cluelessness in reply. Very few companies have a means to cut through the customer support filter and provide a means to report a website issue to someone who understands the tech.
Interesting and insightful read, as always. But, does changing to a pseudorandom ID really fix the issue?
Certainly it makes it less trivial, but could an attacker not still enumerate the entire ID space to perform effectively the same attack if only the ID is being used by the platform to tell one user from another?
Interesting and insightful read, as always. But, does changing to a pseudorandom ID really fix the issue?
From a practical standpoint, yes it can. It’s very straightforward to pick a pseudorandom ID in a field large enough (enough bits of entropy) that collision or even enumeration becomes a nonissue. Think large enough where it would take quadrillions of attempts (or more) to even hit one actual entry. This is a fairly common practice these days and it seems to work well. The rate limiting inherent in any server cluster’s maximum performance is sufficient to deter any meaningful attack.
Fixed by applying a pseudo random string – really? If that is the full extent of their patch, then all they did was replace an extremely exploitable flaw and replaced it with an ever so slightly harder to exploit flaw.
Really, this pseudo-random string might be a 32 digit UUID generated with a secure number generator.
I’ll have to see it to know what they did, but that is what is normally done.
The root problem is with the architecture of the solution. If you enter an alert ID that is not linked with your client ID, the page should say alert not found even if one exists under another client. Even if you use a large random string (such as a GUID) to prevent enumeration, that is still no excuse for returning alert IDs not associated with the logged in user’s client ID.
The best mitigation is to return only those alerts associated with the logged in user’s session ID. That way even if the user requests a valid alert ID for another user the site should return nothing.
Using a large random value such as a GUID is an additional measure for obscuring the IDs but is not a complete solution to this problem.
I was thinking the same thing. Did they put in place any authorisation checks to make sure that the user accessing the page has access to the account? Or did they just change the identifier to make it harder to randomly guess?
If they only changed the identifier, it’s not really fixing the insecure direct object reference (IDOR) vulnerability. It’s just making the reference harder to come across by accident. Anyone who can get hold of someone else’s identifier (e.g. by snooping on their email) can still access their alert settings.
Great article. There is no way this is the only issue with this site. A company that makes this kind of amateur hour mistake is not following other best practices. I hope this guy keeps looking.
another job well done if only people got rewards for finding these flaws.
Surely you’ve heard of bug bounties? While there’s no indication a bug bounty was paid in this particular case, or that Fiserv offers bounties, many companies do compensate hackers for finding and responsibly disclosing security holes.
It’s common enough that there are a number of researchers (Probably a relatively small number, and probably only the most skilled/talented researchers.) whose sole income is derived from bug bounties, and there are a number of platforms, the most well known of which is probably hacker one, that exist solely to manage bug bounty programs.
There’s no obvious sign that fiserv participates in bounties. Not surprising, but also not good for the ecosystem/their customers’ clients.
When you say “single digit in a Web page request”, do you mean URL?
Can you please let us know what was triggered by that single digit change?
How do I say this gently? If you don’t know the answer to your question, I’m not going to tell you.
You are asking a question that indicates that you don’t know the basics of query strings. Study how URLs are constructed for dynamic web sites and the answer will become obvious.
You could try to be friendly and informative instead of an asshole. There are no requirements needed to access this webpage and he has a simple question. I have to wonder if you are choosing to be a prick because you dont know the answer yourself.
There are a number of tools — Burp Suite stand alone or Tamper as a browser plugin — that can be used to stutter each page request and alter the data fields that get transmitted. It wasn’t a URL change, it was a parameter inside the HTML of the site.
I ran across one of these possible indicators of vulnerability while examining my bank’s web page during one of their many buyout transitions. I didn’t dare burp it for fear of ending up being accused of computer hacking.
So now I just don’t bother looking at HTML
Take a look at this page on the OWASP web site. It explains things pretty well:
https://www.owasp.org/index.php/Insecure_Direct_Object_Reference_Prevention_Cheat_Sheet
In short, the number in the URL was an identifier. Changing this identifier to a different value allowed Krisitan to access data that belonged to other people. The page was missing access controls to ensure that the data was only accessible by the person that the data belonged to.
Brian,
It might be worth noting that those who bank at a financial institution on the Jack Henry platform, they will see the name of that company prominently displayed in their browser address bar at some point, and from what I can tell, Henry is an unrelated competitor to Fiserv.
The article got me to wondering about my two credit union’s sites (one big and one small, in two different states). I recalled seeing the Henry name but needed to do an Internet check to confirm they are competitors.
It might be worth noting that Fiserv is the market leader with a 37% share followed by Henry with 20% share. A graph of other players is at the following link, and above the graph is a link to a more detailed report:
https://bankinnovation.net/2018/02/fiserv-has-largest-u-s-marketshare-of-top-bank-core-processors/
Robert,
Jack Henry is a competitor of Fiserv. They, FIS, and Fiserv have the market share of core banking for community banks across the US. This risk is related to online banking. Many community banks will use the same provider for online banking that they use for core processing; however, that is not always the case (there are quite a few online banking providers who aren’t also core providers).
This risk appears to be isolated to alerts generated as part of at least one of the Fiserv online banking products (they have multiple platforms that they sell to banks in that space).
Another superb article, Brian. Keep up the great work!
Just for clarification, this issue would affect banks that use Fiserv’s online banking platform. There are banks that use Fiserv’s core software but utilize a third party for their online banking offering. This does not refer to the bank’s regular informational website.
There needs to be regulation for back office software development in regards to outsourcing the work offshore. Banking is just too critical to allow code, customer data and intellectual property to be compromised for the sake of saving on labor costs. And the kicker is that the banks and end-users have no idea where this valuable development is being housed.
Fiserv is steadily working to fire and replace all US based Development. These problems will only continue to worsen. They think security is a “joke”. I have spent a chunk of my life arguing for better security in fiserv products across the board and it almost always falls on deaf ears.
I feel your pain. We have been doing the same here for the last 8 years.
Levi — Please feel free to get in touch with any concerns. krebsonsecurity @ gmail dot com. Thanks!
Just curious; are you an IT administrator at a bank system in Kearney Nebraska?
That must be some other Levi Rogers.
The bigger problem I see is that FISERV was unresponsive to customers who identified a serious problem with their software! How can we get their attention without having to go to the press?
Hi Rex, I don’t know the answer, but if you know of issues that need to be addressed please feel free to ping me at krebsonsecurity @ gmail dot com. Thanks!
Good Evening Brian,
First off, I have a great respect for the work you enable and do here on Krebs. Secondly I am an Information Security Manager working for the Enterprise Technology Group of Fiserv. ETG runs the shared platforms as well as does our best to hold our various products, business units, and applications accountable to our security standards above and beyond the regulatory standards. If any of our customers, clients, or current / prior employees have a security concern related to one of our products, websites, applications, or connections I would sincerely love to hear them so that I can use what influence and means I have at our company, above and beyond our consumer support and outreach branches, to address their concerns. My linkedin profile is here: https://www.linkedin.com/in/adkinder/ please feel free to message me.
While I can not, under any circumstances speak on behalf of the company, I can offer my thoughts, opinions, and influence as an individual as it relates to concerns. It is events like these, and people like Hermansen and yourself that enable people like me, who work in the Security industry, to bring awareness and attention to fundamental, foundational, or even (as in this case) situational/limited scope issues and give us the means to highlight, address, and fix them in a timely manor.
Adam,
Does Fiserv have a web page that gives guidance on how to report security issues with Fiserv products?
Do we know dates of when this issue was originally discovered and subsequently reported to the bank/Fiserv?
Being a Fiserv hosted bank that uses this internet banking product, I can tell you that, as of now, they have not notified us of this issue or resolution.
Look for Bulletin 2018-1473. I received it yesterday morning, but it looks like they are proactively applying a patch rather than responding to a threat.
Gina is this a problem with the WebFederal product?
Julie Conroy took this opportunity to disparage smaller financial institutions. She says they cannot afford to write their own core- software. So, by her logic, software written for a single large financial institution is less likely to have a security flaw than software written by Fiserv for their 1700+ financial institutions. How does that make sense? I suggest Julie do a little more investigation before she condemns small banks.
Steve, I didn’t get that at all in my conversation with Julie. If anything, she seemed to me sympathetic to the smaller banks, who as she said already face tough competitive challenges vis-a-vis the bigger banks, who can spend more and offer more services and conveniences. Also, she doesn’t say this directly, but the bigger banks definitely view their size as a competitive security advantage to their smaller peers.
The bigger banks can spend more and can offer services and conveniences that a smaller bank can’t afford. That is the very reason smaller banks engage with a company like Fiserv or Jack Henry or FIS, or CSI. These larger ‘co-ops’ put us all on a level playing field. Fiserv’s 1700 + financial institutions can rival the largest US banking institution in offerings and online security.
It’s a weak argument to say that a larger bank or budget is a panacea for how secure your data is. Frankly some would argue that smaller institutions have the ability be more nimble in the security and risk management perspective, since the effort and resources required to protect a smaller number of informational systems could be an advantage?
I don’t think people understand the amount of functionality and code that goes into a full fledged online banking platform.
If you can afford to be behind your competition and start simple and expand it over time, then sure go for it, but I think banks would rather use an existing product and be able to compete from day one.
Additionally as you add more code to your platform you structural complexity goes up, your code gets less clean and you reach a point where the product requires quite a bit of maintenance to the point where your forward development of the product slows down and in some cases even grinds to a halt.
Steve Holt!
I was really waiting for an obligatory “we take security of our customers very seriously” deadbeat statement from their PR email. Thanks for including a version of it Brian.
What amazes me though is that they still don’t seem to get it. They said that now they are using random strings instead of sequential ids. Well, how about a good old authentication for those pages, you dumb a$&es?
Those who make rules they also brake the rules
Fiserv is much smaller than you indicate. It did not have $5.7 billion in earnings. According to their 2017 10-K Annual Report, they had $5.7 billion in revenue, not earnings.
Their net income (‘earnings’) were only $1.2 billion. Nice business, but no giant.
If you use an online portal that Fiserv developed/wrote than I would suspect it was exposed to that breach. Ask your vendor (Verizon in your case) and press for an answer.
I wonder how simple that number in the URL’s meaning was. If it was something like a SQL row index or anything similar that developer/team should be terminated. Harsh but something like that is inexcusable in the Financial Services biz.
Good article. From what I can tell this is only affecting their “Bank Solutions” customers, is that correct? I don’t personally bank with any of their clients, but with their other services such as Biller solutions, card solutions, etc., I do use some of those clients. So for instance, will my Verizon account have the same vulnerability? Or was this just a colossal ball drop only in the one division?
Matt, the issue effected one particular (messaging) application in a suite of applications that Fiserv sells and hosts. In addition, not all institutions deploy this messaging app and it’s not required for Fiserv’s internet banking suite. Therefore not all of the banks utilizing Fiserv’s software are necessarily effected.
I work for a community bank that purchases and utilizes Fiserv software and we do NOT use the messaging module and therefore were not vulnerable/effected.
FiServ has multiple online banking platforms.
I suspect the culprit in this case is one of the smaller ones that service credit unions.
The impact may be much smaller than what is claimed.
Agree Awyn. Of the 3 FI’s I patronize of which one is small does use the core processing but not the online access from Fiserv. And after researching further it appears Fiservs division for credit unions pushes a platform call DNA.
I just checked the FISV stock and it was down today 1.6% on double the volume on an up day in the overall market. I’d say this news release had some eyes on it.
I used to architect one of their other online banking platforms and I am disappointed in their PR department.
The least they can say is, it affects x % (which I suspect is like <5%) of our online banking clients.
Now everybody reads hundreds of banks and the stock suffers.
Back 20 years ago when Dan Murphy was the Fiserv internal auditor VP and I was his customer at TruSecure and we certified his platform at Corvallis this never happened. Now, it’s hard to say, but standards definitely went down.
Did he say which Fiserv online banking platform had this issue?
We too are a Fiserv hosted Bank, using the Premier suite of products. Notification came late last night and was poorly crafted. Attempts to get additional information today were met by crickets.
As someone who administered a Fiserv online banking system for almost 7 years I hate to say that this doesn’t fully surprise me.
I will the article says 1,700 banks use Fiserv, but to be fair, Fiserv offers many different online banking systems, so not all 1700 banks could be on that system, just a sub-set. Then not all of that subset may offer online alerts, so the affected banks could be a sub-set of a sub-set.
Me thinks me smells a 1990’s style SQL injection flaw.
No, as there is no sql involved with event manager (the alerting platform) at all.
What pissed me off was the corporate talk that says they take security seriously when it was only when the press stepped in the company did something about it. They probably did the cheapest pseudo fix just to get rid of the bad press.
What no one has pointed out yet is that this is a common developer mistake that should have EASILY been caught and fixed had the Banks and Credit Unions been conducting their due diligence in performing quarterly vulnerability scans and Penetration Testing as required by many regulatory standards and industry standard security practices. The banks not doing their testing are just as much at fault as the Fiserv developers. Tsk Tsk.
As the safety net prevents an undertrained, inexperienced trapeze artist from death. Relying on quarterly reviews to “fix” shockingly inappropriate mechanisms = paddling.
Minor point. Fiserv does not have $5.7 billion in earnings. It has $5.7 billion in revenue and around a billion in net income or earnings. There is a significant difference.
It took me six months of constant emailing and calling my local credit union in Canada to allow longer than 8 character passwords for online access. Worse still, you could use as few as 4 until they finally relented and changed it.
Just another “ID in URL” vulnerability
So Classical
So Sad
Despite years of developers awareness programs
Nightmare of CISOs