How confident are we in forecasting the automation of work?

This week, the Office for National Statistics released their first statistics on which occupations they believe are at the highest risk of automation across England. They estimate about 7.4% of all jobs are at high risk of automation, i.e. there is a 70%+ probability that workers in that occupation will be replaced with some combination of software and robotics. In particular, they predict:

1. Over 10% of roles held by women are at high risk of automation, compared to 4% of men’s roles.
2. 15.7% of 20-24-year-olds are in roles that are at high risk of automation, more than any other age group.
3. The North-East, the East of England, Cornwall and Devon have the highest average risk of local jobs being automated, whereas the rest of the South and London have the lowest average risk of automation.

Forecasts like these are an important tool in making effective long-term policy decisions. Where do we need to invest in reskilling? What skills should form the basis of our education and training system to future-proof our economy? To what extent is unemployment systematic versus cyclical? All of these are important questions a robust forecast could help us answer. However, there are reasons to take this forecast with more than a pinch of salt.

First, there is no time horizon attached to the risk of automation. Different capabilities relevant to automation, like computer vision or emotional intelligence, may develop at radically different paces, so the extent to which an occupation is at high-risk of being automated is significantly dependent on the ‘when’ we are talking about. Further, if a job is at high risk of automation within the next five years, that has significantly different policy implications versus a high risk of automation in twenty or fifty years, e.g. investing in adult versus childhood education or planning welfare spending.

Further, when you take a closer look at the data, some of the specific predictions start to seem dubious. For example, the ONS predicts that waiters and waitresses are the most at risk occupation, with 72.8% of jobs at high risk. Yet progress in robotic manual dexterity, a seemingly key part of the role of a waiter, has been slow even with recent advances. Conversely, professionals are forecast as the group at the lowest risk of automation. However, in their recent book The Future of The Professions, Daniel and Richard Susskind lay out just how vulnerable traditional professional roles are to decomposition and replacement by increasingly capable information processing systems. Neither of these provide a concrete alternative prediction of risk and the ONS forecast may well pan out. However, these are a good sanity-check about being too confident in the results.

Finally, the models their analysis builds on use the judgement of artificial intelligence experts to determine whether a set of jobs or tasks could be automated and then infer those judgements across the whole range of occupations in the economy. On the face of it, relying on experts might seem like a reasonable suggestion and clearly there is a lot to be gained from the insight of those with subject-matter knowledge. However, when it comes to making concrete predictions about specific outcomes in the realm of artificial intelligence, expert predictions without an underlying quantitative model have been notoriously unreliable in the past, producing results with no significant difference to non-expert predictions. Therefore there are strong reasons to be sceptical about the ONS’s forecast.

We should try to develop automation forecasts based on more quantitative data, grounded in real-world data. For example, we could measure the total investment in automating technologies, the number of patents registered in this area, and the differential investment across different capabilities. This could then be extended using a dynamic model of firm competition and cooperation over time, based on records of existing AI and robotics firms, in combination with labour markets models to estimate how industries might actually react to labour-substituting or labour-augmenting technologies.

Further, the implementation and especially the training of AI systems requires a significant amount of computational power. Some predictions estimate the current rate of growth in compute used by AI systems can only continue for another 3.5-10 years, with cutting-edge AI experiments costing $200bn, putting it out of the reach of anyone but the US or Chinese governments. Therefore, there may be a significant fall-off in the rate of improvement in the performance of AI systems, and so their ability to substitute or augment human labour, even if there is significant potential for automation in the short-term. It’s possible the expert predictions may implicitly take this into account in their prediction of automation. However, mapping out these potential physical and economic constraints and integrating them into predictions of automation could increase their predictive power, improve their interpretability and make them easier to update in light of new information about computational capacity or requirements.

All models carry a degree of uncertainty, but any of these approaches would offer alternative insights in how automation is likely to progress and how it might affect employment in the UK going forward, with a more transparent and grounded basis than ‘subject-expert judgement’. Further, employing a range of independent modelling and forecasting techniques would help develop more robust conclusions, if similar results arise from different theories and data.

That the ONS has produced these figures at all is certainly an encouraging sign that the government is taking seriously the challenges from automation. There has been relatively little and relatively homogenous work in the area of quantitatively forecasting the risks and trajectories of automation, given the significant consequences that these trends are likely to have for the economy and society more widely.

The UK government should invest more resources into developing and exploring a range of robust potential models in this area, to inform its own policymaking, to help firms make better long-term decisions, and to take the lead in pushing forward the boundaries of understanding on a pivotal topic.