I work for a global investment bank, in the upper reaches of a vast office tower just like this one. It's one of those Wall Street companies that people love to target during election campaigns. And you know what: conspiracy theories aside, there are good reasons to be concerned about these rulers of the financial markets.
This post will be the first in a series chronicling my adventures within the secretive & byzantine labyrinth of finance. I'll be referring to my employer as Big Bank throughout, obscuring certain identifying details, and speaking in generalities. This is mainly out of practicality; I need to keep paying the bills and have no desire to get on the wrong side of non-disclosure agreements (whoever says free speech isn't dead has obviously never worked for a large corporation).
In writing this I aim not to make my employer look bad, but to sound the alarm and raise awareness. The world's financial system is teetering on the edge of a cliff, and sanity must prevail before it falls off.
My duties for Big Bank
I'm a member of the front office IT team. That means I look after the software systems used by the bank's traders and clients. As opposed to the middle office or back office, which own different systems related to trade settlement and risk management. My job is split pretty evenly between tech support and software development duties. I'll start by giving you a brief overview of each, and then we'll go into more detail in future posts.
1. On the front lines of tech support
The role is referred to as front line support for good reason: at times it can feel like making a courageous last stand in the middle of a war zone, standing your ground while facing down hoards of angry traders, impatient salespeople, and stressed out clients.
My team is basically the first point of contact when IT systems malfunction (which happens with disturbing regularity). At least two or three people monitor the support mailbox at all times, 24 hours a day in rotating shifts. Response time is critical; if client facing trading systems go down and aren't brought back online quickly, the bank can lose millions of dollars from lost business opportunities, not to mention reputational impact with clients who depend on reliable service. System malfunctions are also carefully monitored by government regulators and in extreme cases can result in large fines levied against the bank.
During support shifts I don my detective hat. When issues occur, my goal is to solve the case and ensure as little business disruption as possible. A typical incident might go something like this:
- E-mail comes in from a disgruntled trader: "hey all the prices are stale in my liquidity GUI. What gives!?"
- I walk over to the trader's desk, get him to show me the problem.
- I go back to my desk and fruitlessly hunt around in log files looking for cryptic error messages that might be related.
- After pulling info from the logs, I read through a support wiki with sparse documentation that has nothing useful whatsoever about this mysterious problem I've never seen before.
- I open a chat window with someone on the pricing team: "do you have any idea what this error means? I've never seen it before".
- Response comes back: "hmm, no idea. Try asking my counterpart in London, he's the expert".
- Of course London guys won't be in the office for 5 more hours. Meanwhile trader e-mails me again: "what's taking so long? I've gotta be ready for a BOJ (Bank of Japan) announcement in 10 minutes!"
- Me: "umm... let me try restarting the server software". I proceed to restart the process in question, praying it won't break something else and / or get me yelled at.
- A few minutes later, trader happily calls me up: "it worked, you're a genius!"
- I feel good, even though I have no idea what the problem was or why restarting fixed it.
- I send a follow up mail to the development team that made the software: "we had this incident today, would you mind taking a look and letting me know what the root cause was?"
- Development team never mails me back.
This right here illustrates the whole problem with tech support in a nutshell. We have a bewilderingly vast array of systems to support, many of which require specialized business knowledge to operate. The support teams are understaffed and undertrained, with most knowledge concentrated amongst a few senior people scattered around the globe. If I was a client, I'd be scared witless if I knew just how little the people that are supposed to keep these multi-million dollar software systems running actually know about them.
The problems are two-fold:
- Entropy - as time goes on, systems get increasingly chaotic. Documentation that might have been perfectly well organized at product launch quickly grows outdated. New features introduce more complexity, and more bugs. Longstanding problems don't get fixed properly due to budget constraints, and hastily improvised patches keep things running but at the expense of maintainability.
- Knowledge transfer (or lack thereof) - in an age of reduced profits and increased regulation, layoffs are common. And every time someone gets axed, they take all their specialized systems knowledge, sometimes built up over many years, with them. Training is minimal, done in informal hour-long sessions that are barely sufficient to scratch the surface. Often the only help I get is "here's a system diagram and a wiki page that may or may not be accurate; good luck!".
2. Financial software development
My days off from support shifts are known as "dev days". This is the part of the job that is actually fun and rewarding. Big Bank has created a lot of custom in-house development tools & frameworks to make the lives of developers easier and provide a consistent strategic direction for the firm. There's always something new to learn and some interesting project to sink my teeth into.
The first thing to know about a bank's financial software is that it's not just a single computer program that runs on your PC, like Microsoft Word. It truly is a software system, made up of hundreds of individual programs (yes, hundreds) that all communicate with each other, passing data back & forth to accomplish some larger purpose.
For example, consider the simple action of a client using electronic trading software to buy some Japanese yen (JPY) with US dollars (USD). Behind the scenes, this triggers a hugely complicated data flow within Big Bank to "book" the trade and settle the cash flows it represents. In highly simplified form, the system diagram for this action might look something like this:
Note: this is a hypothetical example whipped up for illustrative purposes only. It is not a depiction of real trading systems at any company I have worked for.
Each box in that diagram is a separate computer program, most of which run on the bank's servers. As if this isn't complicated enough, different teams own different parts of the system, with their own programming styles, logging formats, and interfaces for the programs to talk to other programs. This is represented by the color coding in the diagram:
| The pricing system runs complicated mathematical algorithms on trader input to produce quotes for various financial products | |
| Human facing graphical interface programs | |
| Trade booking system (I work mostly on these components) | |
| The settlement system handles cash flows, trade confirmations, and client account management | |
| Databases for storing trade information & miscellaneous info needed to settle the trades |
But wait! There's more...
Legacy Software
Really old bits of the system that no longer work so well and have fallen into disrepair are known as "legacy software". Just hearing that term and thinking of all the trouble it causes is enough to make me shudder. It puts me in mind of a big ball of tangled, rusty piping, much patched over with duct tape and leaking water from a dozen places. Not something you want to go near if you can help it, for fear that the slightest touch will cause the whole thing to fall apart.
This is legacy software given physical form. The stuff of nightmares...
I once worked for a firm where the low-level plumbing in the back office settlement systems dated back to the early 1990s. And sections of it were even older, written in FORTRAN (an ancient programming language that nobody in their right mind would use for business software today). All the original developers were gone, and nobody really understood how the hundreds of thousands of lines of code worked. And this was a vital system at the heart of the bank's business, without which the firm could not function on a day-to-day basis.
God forbid it should ever malfunction, because nobody would know how to fix it. Before I left the firm for greener pastures, there was an attempt to start an initiative to replace this aging monster with something more modern. But it never got off the ground because it was estimated to require several years of effort, hiring a whole new team of expert developers, and carry a price tag in the tens of millions of dollars.
Sadly, this is not an uncommon problem for a bank to have. Pretty much every firm I've worked for has been saddled with legacy software of some form or another, and dealing with it consumes a significant portion of the IT budget. The finance industry as a whole is paying a high price as it struggles to modernize and keep up with the latest technology.
Beyond human comprehension
Now you can begin to understand why maintaining and providing tech support for Big Bank's software is such a challenge. By their very nature, financial systems are brittle and break easily, despite attempts to build in redundancy and safeguards (which in turn just makes the system even more complicated and difficult to understand, like a vicious feedback loop).
These days there is much talk at Big Bank about using machine learning algorithms and adaptive AI to tame the complexity, thereby taking away a lot of the support burden from humans. Imagine programs that can self-diagnose and heal themselves, or are able to judge the severity of issues, only turning to their human masters for help in the most extreme circumstances.
With vast, sprawling software systems on the verge of spiraling beyond the ability of humans to comprehend them, more and more duties will be handed over to machines. Already, Big Bank's systems are too big for any one person to understand in their entirety; efforts to trace data flow from one end of the system to the other often require coordination across multiple teams of engineers. Travel down this path far enough, and it's not impossible to imagine a day when my job gets completely replaced by an artificial intelligence.
Would you want this to be your new boss?
Anyone who's seen The Matrix, or Terminator, should fear that day. Skynet may not be that far-fetched after all.
What can we do to avert disaster?
Well, setting aside the issue of malevolent AI for the time being, it's clear that the present state of affairs cannot continue. Sooner or later, the world's financial system will implode in spectacular fashion unless drastic changes are made.
The answer may be blockchain technology, which has generated much excitement in financial circles. This radical new technology (upon which Steemit itself is based!) has the potential to cut clean through the rusted pipes of legacy software, replacing the whole cancerous system with something much cleaner and purer. But that's a topic for a future blog post.
I'll stop here, ending on a positive note with my blockchain teaser that all is not lost, that a solution could be at hand if we but have the courage to reach for it.
Next time, we will journey deeper into the financial mire and discuss one beast that bank employees fear above all others: the dreaded RIF (Reduction In Force). So stay tuned and grab your weed-whacker, because there's plenty more adventuring to come!
Links for further study
Good explanation on the differences between front office, middle office, and back office: http://www.investopedia.com/terms/f/frontoffice.asp
Discussion & examples of legacy software: https://en.wikipedia.org/wiki/Legacy_system
A starting point for researching the types of financial software systems I work on: https://en.wikipedia.org/wiki/Electronic_trading_platform
For more posts about cryptocurrency, finance, travels in Japan, and my journey to escape corporate slavery, please follow me: 
@cryptomancer
Image credits: the first image of the skyscraper is a photo I took on my iPhone. The trade flow system diagram is my own invention, made with Microsoft PowerPoint & incorporating some people pictures from a clip art package I purchased and own distribution rights to.
All other images in this post are taken from Pixabay and used under Creative Commons CC0 .