Managing an ETRM Crisis: What to Fix First and Why

You may have gotten a whiff of the fact that the war in Iran impacted energy markets. Oil and gas prices have been jumping up and down with volatility close to 2022 levels and I will dare a wild guess some energy and commodity traders either picked up smoking or got back to it.

So, for large organizations reliant on ETRMs for their risk and front office reporting, the last few weeks have been among the worst to get a delayed end-of-day, or a major incident blocking deals from timely entering the system (or – as it usually happens – all of the above). Having been at least twice on the receiving end of a statement along the lines of “you caused a once in 10 years mess in production” in the past 18 years of ETRM-ing – and having been many times on the sorting out side of these events, I figured it may be a better time than usual to share how ETRM crisis management should ideally play out, so that minimum damage to business is risked.

Question number one…

…should in my world by a wide margin be “do traders have a position to trade against?”. People doing operations often omit this question, because most experienced – or just healthily paranoid – traders have a backup. Either they back up a real time reporting tool with a scheduled extract in the early morning, or they track their trades manually in an Excel, or they just know the positions. Every now and again, however, things go wrong with the backup and this makes a massive difference: whether the ops team should be trying to identify and fix the root cause of the problem, or set 20 minutes aside for figuring out a basic position report for the affected trader(s).

The ETRM crisis management process

There is a very dangerous scenario that plays out in some companies: everybody is worried who will be blamed and becomes a pencil pusher trying to follow process and leave a very clear paper trail. While following process is key, in a scenario where the markets are volatile, speed of fixing the system becomes paramount. And this means, ideally, the key parties should be in the same room or call as soon as possible. This means at the minimum:

• The most senior ops person available (and their boss should more reinforcement be needed)
• The incident manager (if there is such a function – if not, the person who ensures all affected parties are informed what is happening on a regular basis, when the resolution is expected, and what steps are being taken to mitigate the problems)
• The product owner (or whoever decides what gets prioritized in the live system once such decisions are warranted – e.g. do we continue running the price fixing or the market risk reporting will survive without it for the day and we do it in the afternoon)
• Any severely affected party that has a reasonable understanding of the systems and whose contribution is needed for prioritization of activities or understanding the problem better
• Anybody else who has a role to play in getting the system landscape back to normal

Coordination between these key parties needs to be tight and efficient in such scenarios. E-mail chains may do the job, chats as well – but they are likely to cause time delays and misunderstandings. And you want to minimize those as much as possible.

Once all the relevant people have gathered, the task would be:

• Determine all known facts and establish shared problem understanding
• Formulate root cause hypotheses
• Distribute tasks very explicitly (typically they are more or less predetermined by the functional role of the person – but in complex cases, very likely cross-functional follow-ups will be needed to come to needed conclusions faster)
• Define who communicates what to stakeholders (ideally an incident management function if there is one, or directly one of the key experts working on the issue)
• Identify next check-in for the same group

Don’ts

The worst thing that could happen is panic (and in such situations it may go across the organization). Communication needs to be very fact based, impact on business considered (“servers are stuck” is less meaningful than “Conti Gas position reports are not generated yet”) and clear on next steps and ETA (and if no ETA of resolution yet, at the minimum an ETA for the next update). Coordination between key stakeholders needs to be very tight, so that a resolution is achieved in an efficient manner. Don’t go playing with the system without telling key stakeholders about what steps you are taking.

The final don’t for people involved: don’t go telling the business things are solved once you see a “Success” status of some kind – verify that there are outputs and to the best of your ability, verify the outputs are correct. If the systems are shaky, a fake success here and there tends to pop up and reporting falsely in such cases can do a lot of harm to trust between people, functions, or in systems.

Good luck and fewer PROD crises!

Ventsislav Topuzov

CEO