On May 30, 2017, five researchers from the Future of Humanity Institute, Oxford University; AI Impacts; and the Yale University department of political science published the results of their survey When Will AI Exceed Human Performance? Convinced that advances in AI (artificial intelligence) will have massive social consequences, and that “to prepare for these challenges, accurate forecasting of transformative AI would be invaluable,” the group created a survey that was delivered to a “larger and more representative sample of AI experts than any study to date.” They were looking for insights into the timing of AI advances to better measure both the social and ethical impacts of AI.
The group chosen to receive the survey was composed of researchers who had published at the 2015 Conference on Neural Information Processing Systems (NIPS) and the International Conference on Machine Learning (ICML), which were characterized as “two of the premier venues for peer-reviewed research in machine learning.” Out of the 1,634 researchers contacted, 352 responded, which is a little more than 21%. The survey questions concerned the timing of specific AI abilities, such as folding laundry, translating languages, and superiority over humans at specific occupations. Researchers hoped that the responses would explain when we can expect to see specific AI developments happen and what the social impacts of advanced AI might be.
More than 70 years ago, British mathematician Alan Turing created a test to measure a computer’s intelligence relative to humans, who at that point were still writing the programming for their computers. Essentially, it involved a human sitting at a keyboard communicating with another human and a computer, in other rooms. If, and when, the computer in the conversation could fool the others into believing it was human also, we would then have attained an odd equity with our machines. Today, that goal for machine intelligence is measured with something called high-level-machine intelligence (HLMI).
In the AI study, HLMI was the central marker for equity, and the questionnaire asked when machines will reach this goal. Specifically, the report explained, HLMI will be “achieved when unaided machines can accomplish every task better and more cheaply than human workers.”
The report took the individual responses for the probability of HLMI arriving in future years and then reported the mean for each category. The aggregate 2016 forecast from these 352 experts in machine learning “gave a 50% chance of HLMI occurring within 45 years (from the year 2016) and a 10% chance of it occurring within nine years.” Curiously, when the responding groups are broken down by region, there’s a wide discrepancy between Asian and North American respondents. Asian respondents expect HLMI in 30 years, and North Americans expect it to arrive in 74 years.
Dividing the responses into different categories by tasks and occupations, milestones for each were reported with dots (the mean) that show the number of years respondents think machines will need to meet or surpass humans and be cheaper to employ.
Timeline of Median Estimates for AI Achieving Human Performance
SOCIAL AND ECONOMIC CONSEQUENCES
The impending displacement of humans by AI-empowered smart machines introduces a number of important questions, and the report addresses those related to an intelligence explosion, the quality of the outcomes, and AI safety. More specifically, respondents were asked: Will AI progress explode once research and development become automated? In what ways will HLMI affect economic growth, and will these outcomes be positive or negative, or both? And finally, how do we ensure that AI progress will benefit mankind, not just shoulder us aside?
The answers were direct:
- Respondents believe the field of machine learning has accelerated in recent years, and explosive progress in AI after HLMI is seen as possible but improbable.
- HLMI is seen as likely to have positive outcomes, but catastrophic risks are possible.
- Society should prioritize research aimed at minimizing the risks of AI.
ESTIMATED AI MILESTONES
The most interesting appendix in the report details the survey content. It includes examples of the questions asked and presents a snapshot of what the authors used to assess machine intelligence. Here are just a few of the predicted milestones and median estimates pulled directly from the report:
Translate (vs. amateur human)—8 years
Perform translation about as good as a human who is fluent in both languages but unskilled in translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese, and Arabic).
Telephone Banking Operator—8.2 years
Provide phone-banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.
Transcribe Speech—7.8 years
Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.
Math Research—43.3 years
Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.
Quick Novice Play at Random Game—12.4 years
Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.
Fold Laundry—5.6 years
Fold laundry as well and as fast as the median human clothing store employee.
5Km Race in City (bipedal robot vs. human)—11.8 years
Beat the fastest human runners in a 5-kilometer race through city streets using a bipedal robot body.
Python Code for Simple Algorithms—8.2 years
Write concise, efficient, human-readable Python [programming] code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists. Suppose the system is given only:
- A specification of what counts as a sorted list
- Several examples of lists undergoing sorting by quicksort
Generate Top 40 Pop Song—11.4 years
Compose a song that is good enough to reach the U.S. Top 40. The system should output the complete song as an audio file.
Write New York Times Best Seller—33 years
Write a novel or short story good enough to make it to the New York Times best-seller list.
Explain Own Actions in Games—10.2 years
For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.
AN UNKNOWN HORIZON
Melvin Kranzberg, who was a professor of technology history at the Georgia Institute of Technology, once posited six laws of technology. The first has become the most famous: “Technology is neither good nor bad; nor is it neutral.” That doesn’t provide a great deal of consolation for humans who are facing some rather uncomfortable replacement scenarios in both the near and distant future.
When the survey asked about the probabilities of HLMI having “a positive or negative impact on humanity over the long run,” respondents were generally optimistic: “The median probability was 25% for a ‘good’ outcome and 20% for an ‘extremely good’ outcome. By contrast, the probability was 10% for a ‘bad outcome’ and 5% for an outcome described as ‘extremely bad (e.g., human extinction)’.” It should be noted that in the final summary, more than half of the respondents called for increased study of how we might protect ourselves from negative outcomes beyond the HLMI horizon.
For the complete 20-page report, visit www.arxiv.org/abs/1705.08807.