What can banks learn from the TSB IT disaster?
Mark Hipperson, chief technology officer of Centtrip, explores where TSB went wrong in their recent technical venture and what he recommends as best practice for future organizations.
Mark Hipperson, chief technology officer of Centtrip, explores where TSB went wrong in their recent technical venture and what he recommends as best practice for future organizations.
Last month UK high-street bank TSB, endured a technical nightmare after a systems breakdown left thousands of customers unable to access their accounts for more than a week.
The blunder led to Paul Pester, TSB’s CEO, being brought before the UK Parliament to provide an explanation for why this IT crisis occurred. What was discovered was that TSB was attempting a transfer of 1.3 billion customer records – a gargantuan project by any standards and although the transfer was planned well in advance, it came with significant risks. A month later, it remains to be answered what the bank could have done differently to avoid the chaos that unfolded and what can other lenders learn from the episode.
TSB’s plan was to move customer records from its former parent company, Lloyds Banking Group, to Proteo4, a platform built by the banks’ Spanish owner, Banco Sabadell. The change-over, which started on Friday 20th April was supposed to be completed over the weekend by 6 pm on Sunday. However, by Monday morning, millions of customers were unable to use online or mobile banking with some even being given access to other people’s accounts.
Data migrations of this scale are of high risk. This is especially true with a banking platform that needs to be available 99.999%of the time – with only five minutes of downtime a year – with the aim of allowing people to have 24/7 access to their money. In fact, no UK bank has previously managed to carry out a transfer of this magnitude successfully and TSB’s attempt was no exception. Looking more closely at what happened and how the events evolved, it appears that some key IT best practices may have been omitted.
Initially, it seems that the developers who had access to the system were making live fixes to production. This is a big no in software development. When the first stage went wrong, there appeared to be no contingency plan or option to revert back to the original platform. TSB should have first validated each change to ensure it is successful before moving on to the next. The testing process is crucial and the bank should confirm that all changes had been implemented successfully and were in working order.
It’s also imperative that early live support is present, with sufficient high-skilled staff available immediately after such a challenging data transfer, in case anything goes wrong. This doesn’t appear to have been the case, with several TSB customers reporting that when they tried to call the bank’s helpline, they were left waiting before the line went blank.
The final stages are proof of concepts (PoCs), which would have revealed any tech and planning errors. TSB should have run PoCs on test accounts, or potentially staff accounts, before the full release.
What is clear is that with common project-management practices in place, this situation could have been prevented, or at least the chance of it occurring could have been reduced significantly.
Another cautionary tale of IT troubles is Williams & Glyn, a division of the Royal Bank of Scotland (RBS) and National Westminster Bank (NatWest) who tried to sell its old branches to numerous suitors. Having failed to do that, RBS reached an agreement with HM Treasury and the European Commission to retain the W&G assets in return for contributing £833m to help increase lending to small and medium-sized enterprises (SMEs).
RBS then set about the relaunch of W&G as a new challenger brand, which would have required migrating its customers to a separate platform. This resulted in high IT migration risks, along with Brexit concerns and regulators telling it to just focus on providing a good service. The bank, therefore, had to abandon the idea and close 162 branches that would have formed W&G in April.
Being meticulous in identifying and mitigating potential risks, planning for any complexities and eventualities and testing as you go along is all common sense, but more can be done.
One option available to banks is to collaborate with the fintech sector to harness its agility to build innovative, cost-effective technology mitigating potential risks and ensuring a smooth transition.
Modern fintech platforms are built from the ground up and are open source and cloud-based. Consequently, they can achieve the required level of resilience, performance and security at a fraction of the cost and complexity of legacy mainframe banking systems. This is a less risky approach than attempting the migration of a large number of customer data between legacy systems. Some banks are even taking the approach of starting from scratch themselves with a new brand, culture and customers.