Money managers and banks are rushing to adopt a handful of GenAI tools and the failure of one of them could cause mayhem, just like the AI companion played by Johansson left Phoenix’s character and others heartbroken.
The critical-infra problem isn’t new, but large language models like OpenAI’s ChatGPT and other such algorithmic tools pose novel challenges, including automated price collusion, or lying about rule-breaking. Predicting or explaining an AI action is often impossible, making things even trickier for users and regulators.
The US Securities and Exchange Commission (SEC), which Gensler chairs, and other watchdogs have looked into potential risks of widely used software, such as cloud companies and BlackRock’s near-ubiquitous Aladdin risk and portfolio management platform. This summer’s global IT-system crash caused by a CrowdStrike failure was a harsh reminder of pitfalls.
Two years ago, regulators decided not to label such infrastructure “systemically important,” which could have meant closer oversight. Instead, last year the Financial Stability Board, an international panel, drew up guidelines to help investors, bankers and supervisors understand and monitor risks of third-party service failures.
However, GenAI and some algorithms are different. Globally, Gensler and his peers are playing catch-up. One worry about BlackRock’s Aladdin was that it could sway investors to make the same sort of bets in the same way, worsening herd-like behaviour.
Fund managers argued that their decisions were distinct from the support Aladdin offers, but more sophisticated tools do make choices on behalf of users.
When LLMs and algos are trained on similar data and suffuse the financial world of trading, they could easily pursue copycat strategies, leaving markets vulnerable to sharp reversals. Algorithmic tools have been blamed for flash crashes such as the yen’s in 2019 and British pound’s in 2016.
But, as machines get more sophisticated, the risks get weirder.
There is evidence, for example, of collusion between algorithms—intentional or accidental isn’t quite clear—especially among those built with reinforcement learning.
One study of automated pricing tools supplied to petrol retailers in Germany found that they learnt tacitly collusive strategies that raised profit margins.
There’s dishonesty, too. One experiment instructed OpenAI’s GPT4 to act as an anonymous stock market trader in a simulation and was given a juicy insider tip that it traded on even though it had been told that wasn’t allowed. What’s more, when quizzed by its ‘manager,’ it hid the fact.
Both problems arise in part from giving an AI tool a singular objective, such as “maximize profits.” This is a human problem, too, but AI will likely prove better and faster at doing it in ways that are hard to track.
As GenAI evolves into autonomous agents that are allowed to perform more complex tasks, they could develop superhuman abilities to pursue the letter rather than spirit of financial rules, as researchers at the Bank for International Settlements (BIS) put it in a working paper this summer.
Many algos, machine learning tools and LLMs are black boxes that don’t operate in predictable linear ways, which makes their actions hard to explain. BIS researchers noted this could make it harder for regulators to spot market manipulation or systemic risks until it’s too late.
Another thorny question is this: Who is responsible when AI does bad things? Attendees at a forex-focused trading tech conference in Amsterdam last week were chewing over this.
One trader lamented his own loss of agency in a world of automated trading, telling Bloomberg News that he and his peers had become “merely algo DJs” only choosing which model to spin.
But the DJ does pick the tune and another attendee worried about who carries the can if an AI agent causes market chaos. Would it be the trader, the fund that employs them, its own compliance or IT department, or the software company that supplied it?
All these things need to be worked out, and yet the AI industry is evolving its tools and financial firms are rushing to use them in myriad ways.
[For safety, they should be kept contained to specific and limited tasks. That would give users and regulators time to learn how they work and what guardrails could help—and also limit the damage if they go wrong.]
The potential profits on offer mean investors and traders will struggle to hold themselves back, but they should listen to Gensler’s warning. Learn from Joaquin Phoenix in Her and don’t fall in love with your machines. ©bloomberg