Is Artificial Intelligence coming after your job?

By David H Bailey, on January 30th, 2020, in the Mathematical Investor blog

Recent progress in artificial intelligence

It is no secret that artificial intelligence (AI) systems have made enormous strides in recent years, partly due to the adoption of Bayesian (probability-based) machine learning techniques rather than the rule-based techniques used until about 20 years ago. One highly publicized AI advance was the 2011 defeat of two champion contestants on the American quiz show “Jeopardy!” by an IBM-developed computer system named “Watson.” The Watson achievement was particularly impressive because it involved natural language understanding, i.e. the understanding of ordinary (and often tricky) English text. Legendary Jeopardy champ Ken Jennings conceded by writing on his tablet, “I for one welcome our new computer overlords.”

Then in 2016, a computer program named “AlphaGo,” developed by researchers at DeepMind, a subsidiary of Alphabet (Google’s parent company), defeated Lee Se-dol, a South Korean Go master, 4-1 in a 5-game tournament. Given the notoriously complicated nature of Go, with strategies that can only be described in vague, subjective terms, most observers had not expected Go-playing computer programs to beat the best human players for many years, if ever.

A year later, in October 2017, Deep Mind researchers announced results of a new program, called AlphaGo Zero, which was programmed only with the rules of Go and a simple reward function and then instructed to play games against itself. After just three days of training (4.9 million training games), the AlphaGo Zero program had advanced to the point that it defeated the earlier Alpha Go program 100 games to zero. After 40 days of training, AlphaGo Zero’s performance was as far ahead of champion human players as champion human players are ahead of amateurs. Additional details are available in an excellent New York Times analysis by mathematician Steven Strogatz.

Of course, AI systems are doing much more than defeating human opponents in games. Here are just a few of the current commercial developments:

• Apple’s Siri and Amazon’s Alexa smartphone-based voice recognition systems are now significantly improved over the earlier versions, and speaker systems incorporating them are rapidly becoming a household staple.
• Facial recognition has also come of age, for example with Apple’s 3-D facial recognition hardware and software, which is built into the latest iPhones and iPads as a security feature, eliminating the need to type passwords for many functions and websites.
• Self-driving cars are already on the road, and 3.5 million truck driving jobs, just in the U.S., are at risk within the next ten years.
• Numerous applications of AI have been fielded in the medical arena, including AI-powered surgical robots, AI-powered radiology (which now out-performs humans at some tasks), and AI-powered entry and analysis of medical data.
• Other occupations likely to be impacted include package delivery drivers, construction workers, legal workers, accountants, report writers and salespeople.

For other examples and additional details see this Math Scholar blog.

What will be the impact of AI on employment?

Throughout history, fears have been raised about the employment impact of automation and technology. In the early 1800s, workers later called Luddites started breaking machines in British textile mills that they viewed as threats to their jobs. In the early 1900s, John Philip Sousa worried that the invention of the record player would render obsolete “the ennobling discipline of learning music” and put many professional musicians out of work (his fears were not realized — today there are more professional musicians than in Sousa’s day). Similarly, horse wranglers and blacksmiths feared that the new-fangled horseless carriages would put them out of work (they did, although millions are now employed in manufacturing, selling and servicing automobiles).

Even today, many have expressed concern about the impact of rapidly advancing technology in the workplace. An Oxford University report, for instance, notes that smart industrial robot installations have more than doubled since 2010, and that cumulative job losses due to smart robots have also more than doubled. A separate Oxford University study predicts that roughly 47% of current U.S. employment is at risk to computerization and automation, with heretofore uncomputerized occupations at the highest risk. A Century Foundation report warns that the pace of these changes may overwhelm the ability of workers to find new jobs and the capacity of social institutions to help.

Many have responded to these developments by observing that in our era, as in earlier eras, technology advances have led to productivity increases, which have advanced standards of living worldwide, and have opened the doors to new and, in most cases, more creative and fulfilling work than before. As Alex Tabarrok observes, if it were true that technology destroys jobs without replacing them, then “we would all be out of work because productivity has been increasing for two centuries.”

Yet some still worry that today’s situation is somehow fundamentally different. Are computers and AI becoming too smart, too fast? Will the requisite dislocations in the economy be too painful? Can governmental, educational and cultural institutions change fast enough? Will there any meaningful work for humans to do in the future?

How will AI and computer technology affect finance?

Until recently, many in the finance may have considered these developments not really applicable to their field — smart robots, for instance, may be useful in auto manufacture and medicine, but not in finance… True, no one has yet proposed a finance application for a smart robot. But machine learning, AI and the larger realm of computer-intensive technology is already having a significant impact in the field, and all signs point to a much greater role in the future.

According to a Bloomberg report, some specific areas that are prime for automation include:

• Sell side credit markets: Natural-language processing, data collection and machine learning are being applied to automate subjective human decisions.
• Sell side foreign exchange: Big data and machine learning are being used to anticipate variations in client demand and the resulting price swings.
• Sell side commodities: Trader and salesperson conversations are being catalogued to create profiles of clients.
• Sell side equities: Artificial intelligence is being applied to order execution.
• Buy side equities: Predictive analytics is being applied to time stock purchases and assess risk based on market liquidity.
• Buy side credit: Computer programs are being trained to scan and understand bond covenants, legal documents and court rulings.
• Buy side macroeconomics: Natural-language processing is being used to analyze central bank commentary for clues on monetary policy. Other software is analyzing data such as oil-tanker shipments and satellite images (e.g., Chinese industrial sites, Walmart parking lots and more) to spot trends in the economy.

Other potential applications for machine learning, AI and big data in finance are highlighted in two previous Mathematical Investor blogs: Blog A and Blog B.

Overall, what are the prospects for employment in the finance field?

It is clear that numerous opportunities await professionals with solid research credentials in state-of-the-art machine learning and artificial intelligence techniques and their application to finance. But what about for others in the field? Do they have a future?

Here the picture is somewhat cloudy. In fact, this subject was the topic of a December 2019 hearing at the U.S. House of Representatives. Here are some comments from Marcos Lopez de Prado’s prepared statement:

Financial [machine learning] creates a number of challenges for the 6.14 million people employed in the finance and insurance industry, many of whom will lose their jobs, not necessarily because they are replaced by machines, but because they are not trained to work alongside algorithms. The retraining of these workers is an urgent and difficult task. But not everything is bad news. Minorities are currently underrepresented in finance. As technical skills become more important than personal connections or privileged upbringing, the wage gap between genders, ethnicities and other classifications should narrow. The key is to ensure equal access to technical education. In finance, too, math could be “the great equalizer.”

Retraining our existing workforce is of paramount importance, however it is not enough. We must make sure that America retains the talent it develops. The founders of the next Google, Amazon or Apple are this very morning attending an engineering or math course at one of our Universities. Unlike in the past, odds are that these future entrepreneurs are in our country on a student visa, and that they will have a very hard time remaining in the United States after their graduation. Unless we help them stay, they will return to their countries of origin with their fellow students, to compete against us in the near future, hindering our competitive advantage.

Numerous other trends in the industry point to storm clouds ahead in the field:

1. The chronic under-performance of actively managed mutual funds. It is embarrassing but true that few actively managed mutual funds out-perform well-known market indices over a long-term time horizon. For example, only 8.3% of U.S. actively managed large-cap value and large-cap growth mutual funds survived and out-performed their equivalent index-based funds for the 10-year period ending February 2019. These statistics indicate that many actively managed funds do not add value, and thus are subject to declines and closure as investors head elsewhere.
2. The chronic under-performance of actively managed hedge funds. The picture is somewhat better in the hedge fund world, although still arguably subpar. For example, the HFRI Fund Weighted Composite Index, scaled to 1.00 at 1990, by 2018 increased to 14.34. But the scaled S&P 500 index increased to 15.10 over the same time period. Some hedge funds consistently beat market averages — for example, Renaissance Technologies’ Medallion Fund has delivered annual returns averaging a whopping 39%, after fees, from 2011 to 2018, with similarly high returns extending back nearly 30 years. But successful funds such as the Medallion Fund are, almost exclusively, the realm of highly sophisticated, highly mathematical, highly big-data-intensive quantitative operations. Other hedge funds, by and large, do not do nearly as well, and are under heavy pressure from major investors to either streamline their operations (and reduce their high fees) or cease operations.
3. The steady increase in the share of passively managed assets. Another measure of these trends is the rise in the fraction of total market assets that are passively managed, as opposed to actively managed. By one measure, passive assets have increased to 45% of the U.S. market, up from only 25% in 2010. Needless to say, passively managed funds require many fewer trained staff than actively managed funds, as evidenced by their much lower fee structures — as low as 0.05%, compared with 1.00% or higher for most actively managed funds.
4. The failure of charting and technical analysis. For many years, a large sector of the investing community has relied on relatively unsophisticated approaches, such as charting and “technical analysis.” Tragically, these obsolete and statistically dubious techniques are even promoted by major financial news organizations and brokerages. Yet all available evidence indicates that these methods do not work in today’s market, which is dominated, as noted above, by mathematically sophisticated, machine-learning based, big-data intensive operations. As awareness of this fact increases, those sectors of the investment world that continue to rely on these outdated techniques are doomed to suffer declines in business and employment.
5. The persistence of backtest overfitting and other statistically erroneous practices. In spite of years of effort by knowledgeable scholars to educate practitioners in the field about the dangers of backtest overfitting and other statistical errors, these practices still persist. But as more investors, institutional and individual, become aware of these difficulties, those investment organizations that cannot cite solid, independent certification that their products and services are free from these errors are doomed to see declines.
6. A growing realization that only a big-data, machine-learning approach can hope to consistently achieve higher-than-market-average returns in today’s high-tech market. The consistent message of many indicators and trends in the field is that investors should Go with big data and machine learning, or leave finance to those who do.

What is the best training for finance professionals?

In short, a growing list of indicators and trends in the finance field suggest that major readjustments and realignments lie ahead. What can one do to ensure that one will be at the forefront of these developments, rather than be left in the dust?

One good technical reference here is the recently-published book Advances in Financial Machine Learning by Marcos Lopez de Prado. It explores commonly used data structures in finance, modeling techniques, backtesting techniques (and ways to avoid backtest overfitting), and other more advanced techniques based on a machine learning approach.

A solid graduate training program in the field would also help. But there is concern that university curricula are not keeping up with these developments. For example, in his Bloomberg column, Noah Smith wonders whether the current training for finance PhDs in particular is the best preparation for careers in the field. He suggests forming academic tracks that guide students to employment in the industry, possibly including apprentice-like research done in conjunction with advisers in the private sector, with dissertation research possibly done in team efforts rather than alone.

Marcos Lopez de Prado has also expressed concern about the typical preparation of researchers in finance careers. He notes, for example, that econometric models often employ statistical practices, such as multiple testing, that are not only considered ineffective but also downright unethical in other scientific research fields. And while most mathematical training for such persons is in areas such as linear algebra and calculus, topics such as graph theory, topology, discrete mathematics, information theory and signal processing are rising in importance.

In a Institutional Investor commentary, Lopez de Prado goes further, arguing that “The presence of financial academia is fading, something that was unthinkable 10 years ago.” He adds, “The [leading] edge is not yet another reincarnation of the capital asset pricing model,” but instead it is in analyzing heretofore untapped data sources. He adds that emerging technologies such as FinTech, machine learning and quantum computing are likely to render traditional academic education in finance even more irrelevant. Compounding the problem is that many academic journals in the finance field are mostly geared as “tenure-track vehicles” for aspiring professors, rather than venues for state-of-the-art research by practitioners. Similarly, books in the field are written by authors who, in many cases, have not actually attempted to field their techniques. As Lopez de Prado explains, “They contain extremely elegant mathematics that describe a world that does not exist.”

As David H. Bailey and Lopez de Prado further argued in a Forbes commentary, interviewed by Brett Steenbarger, rigorous training in statistics is typically not given its appropriate emphasis for prospective finance professionals, PhD or not. As a result, the finance field, as noted above, is replete with backtest overfitting and multiple-testing errors and, even more significantly, many in the field fail to appreciate how deeply these difficulties pervade modern finance, and the extent to which institutional customers and individual investors are potentially misled by inaccurate claims.

Some additional observations about training for finance professions are presented in a previous Mathematical Investor blog.

Nirvana or brave new world?

However these trends turns out, many worry about what the future holds for “average” workers, not only in finance but also in the broader economy. Will the future be a nirvana of creative, interesting and fulfilling work, or a brave new world where most humans are hopelessly relegated to secondary status by computational overlords?

Some of these questions were addressed by Yuval Noah Harari in his recent book Homo Deus: A Brief History of Tomorrow. He argues that future technology, and AI in particular, draws into question many of the bedrock systems that underpin modern society, and may eventually lead to a “post-human” world. But whatever happens in the far future, society faces the more immediate need to substantially improve the education new workers, to retrain many of those who are displaced, and to humanely deal others who find themselves in industries and occupations that are no longer economically valued.

So to a large extent, the future will be what we make it to be.

                 https://mathinvestor.org/2020/01/is-ai-coming-after-your-job/