AI in Financial Markets, Part 1: Beyond the Market-Predicting Magic Box

About this blog series

This series of blog posts is based on a talk the author presented at ODSC Europe 2019 alongside Ayub Hanif, VP at JPMorgan’s Quantitative and Derivatives Strategy. We’ll look at how recent developments in automated machine learning and interpretability can help market participants build, test, and understand powerful AI models that support and enhance their investment processes. But first, some context is needed. 

Check out all of the blog series: part 2, part 3, part 4, part 5.

Part 1: Beyond the Market-Predicting Magic Box

In which we conclude that the existence of this article is strong evidence that we don’t know how to use AI to predict asset prices, either, and try to get real.

It Looks Like Magic

“Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke

There’s a popular conception of what AI can do in financial markets, and how to do it, which runs something like this: you hire a rocket scientist or two and a gaggle of Ph.Ds, and lock them in a room with a lot of expensive computer equipment. Occasionally, you unlock the door and throw some pizza in there, but you don’t let them out until they build you an amazing magic box¹.

It’s a monolithic black box — nobody really knows how it works except for the rocket scientists and one of the Ph.Ds — but the way it works is, you throw a load of data (images, share prices, tweets, news stories, videos, and other stuff) in the hopper, the rocket scientists crank the handle a few times, and the market-predicting magic box spits out portfolios, trades, predictions, analyses, insights, warnings and, ultimately, lots of money. Result: everyone’s happy, buys some really expensive cars, and retires to villas in the Caribbean bought with the proceeds from the magic box, where the peace and quiet is only occasionally broken by the sound of the magic box’s handle being cranked. Sounds pretty cool, right? 

Unfortunately but unsurprisingly, it’s not that simple, and predicting asset prices is actually extremely difficult, even for sophisticated machine learning models. Data science typically succeeds where there are complex behaviors which can be described in data and where consistent inputs lead to consistent “predictable” outcomes. All too often, outcomes in financial markets are anything but consistent or predictable. Nevertheless, there are many great use cases for machine learning in the financial markets — both in the front office and beyond.

It’s Got To Be Real

The pace of adopting machine learning techniques is accelerating in both sell-side investment banks and on the buy-side; there are many, many excellent high-value use cases for machine learning spanning a wide gamut of activities: for instance, using modern automated machine learning techniques,

• Execution traders can demonstrate best execution by using historical transaction cost analysis (TCA) data to build models to predict the market impact of trades, and carry out (and record) pre-trade scenario analyses to evaluate the wide variety of different venues, brokers, algo providers, algorithms and parameters that are available in today’s markets;
  
• Market makers can build sophisticated non-linear models for price discovery in OTC markets and tailor quotes to individual inquiries, as well as identifying the best strategies or counterparties to offset risk;
  
• Sales desks can predict which of their clients are likely to be interested in particular products and market calls, and better target who is sent which pieces of research;
  
• Capital markets (new issues) desks can efficiently build shortlists of investors to target for a given equity or bond issue, and build models to assist them with their pricing;
  
• Prime brokers and business developers can predict which new relationships will actually lead to revenue-generating activities and which new clients will be most profitable, as well as creating early warning signals to identify clients at risk of churning; 
  
• Investment analysts can model credit ratings actions, earnings downgrades and a host of other stock-specific events, as well as using sophisticated natural-language processing to extract value from free text;
  
• Economists can build time-series forecasts of economic variables and other factors that influence asset prices, taking advantage of modern, non-parametric techniques;
  
• Mutual fund managers can build models to better predict daily investor inflows and outflows, thus minimizing the need to drag returns by carrying frictional cash balances;
  
• Fund marketing and distribution professionals can build models predicting how likely individual investors are to churn, to react to particular communications, or to respond to targeted cross-sell offers;
  
• Institutional relationship managers can model how long a sales cycle will be, or how price sensitive a particular mandate will be in negotiations;
  
• Operations teams can reduce costs by predicting which trades are likely to fall out of straight-through processing, require extra hand-holding, or otherwise cause breaks, thereby catching problems before they occur;
  
• Compliance teams can substantially reduce false positive rates when carrying out anti-money laundering, trade surveillance or fraud checks while maintaining accuracy, and
  
• Technologists can enhance their infrastructure’s resilience against cyberattacks and predict the impact of planned releases on business as usual.

Using DataRobot’s automated machine learning, these use cases can all be addressed in a rigorous, powerful way, searching over many different candidate algorithms and models, providing standardized interpretability, automated compliance documentation, robust governance, straightforward deployment and powerful performance monitoring.

Order, Order

The eagle-eyed reader will have noticed that there is no mention of using machine learning for alpha generation in this list. These use cases are all well and good, I hear you say, but they’re all second-order, third-order, or fourth-order problems compared to the first-order problem of generating “alpha” in the financial markets. So, is machine learning another “new new thing” that holds great promise for investors, but ultimately fails to deliver? Or does the promise hold true? How can you use machine learning to improve your effectiveness as an investor?  

Machine learning techniques do indeed offer substantial advantages over “traditional” quant techniques, such as rule mining and generalized linear models; in upcoming blog posts in this series,we’ll explore this in more detail, and look at why automated machine learning, in particular, is set to become an important tool particularly in the quant investor’s arsenal. 

¹ Which is fine with them — remember, they’re stereotypes, so they have no lives beyond their computers.

About the author

Peter Simon

Managing Director, Financial Markets Data Science

Peter leads DataRobot’s financial markets data science practice and works closely with fintech, banking, and asset management clients on numerous high-ROI use cases for the DataRobot AI Platform.  Prior to joining DataRobot, he gained twenty-five years’ experience in senior quantitative research, portfolio management, trading, risk management and data science roles at investment banks and asset managers including Morgan Stanley, Warburg Pincus, Goldman Sachs, Credit Suisse, Lansdowne Partners and Invesco, as well as spending several years as a partner at a start-up global equities hedge fund. Peter has an M.Sc. in Data Science from City, University of London, an MBA from Cranfield University School of Management, and a B.Sc. in Accounting and Financial Analysis from the University of Warwick.  His paper, “Hunting High and Low: Visualising Shifting Correlations in Financial Markets”, was published in the July 2018 issue of Computer Graphics Forum.

Meet Peter Simon