Computers

Eugene Goostman chatbot claimed to have passed Turing Test

Eugene Goostman chatbot claimed to have passed Turing Test
The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test
The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test
View 1 Image
The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test
1/1
The Eugene Goostman chatbot, which simulates a 13-year old boy, has passed the Turing Test

It might be time to start being nicer to your laptop, because researchers at the University of Reading are claiming that a supercomputer program has passed the Turing Test for the first time in history. On Saturday, at the Turing Test 2014 organized by the University of Reading’s School of Systems Engineering, the chatbot Eugene Goostman reportedly convinced the judges 33 percent of the time that it was a human being and not a computer.

Devised by the mathematician Alan Turing in 1950 in his paper "Computing Machinery and Intelligence,” the Turing Test is considered the gold standard for gauging how far we’ve come in the field of artificial intelligence. The test is named after Turing, but the roots of it go back to René Descartes in the 17th century. It strikes not only at questions of artificial intelligence, but also at the limits of automata in general, the question of how we know if other people possess consciousness, and even the philosophical basis of materialism.

The idea is that an interrogator communicates with two contestants, one human and the other a machine, solely by text. The interrogator asks questions of each for five minutes and then decides which one is human. If the machine wins more than 30 percent of the rounds, it’s regarded as having passed the test. So far, no computer program has ever achieved this, but on Saturday, Professor Kevin Warwick, a Visiting Professor at the University of Reading and Deputy Vice-Chancellor for Research at Coventry University, says that the chatbot Eugene Goostman scored 33 percent.

There are actually several versions of the test, each with its own rules and criteria of what constitutes success. "Some will claim that the Test has already been passed,” says Professor Warwick. "The words Turing Test have been applied to similar competitions around the world. However this event involved more simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted. A true Turing Test does not set the questions or topics prior to the conversations. We are therefore proud to declare that Alan Turing's Test was passed for the first time on Saturday.”

The test is controversial because of the tendency of interrogators to attribute human characteristics to what is often a very simple algorithm. This is unfortunate because chatbots are easy to trip up if the interrogator is even slightly suspicious. Chatbots have difficulty with follow up questions and are easily thrown by non-sequiturs that a human could either give a straight answer to or respond to by specifically asking what the heck you’re talking about, then replying in context to the answer.

One of five supercomputer programs participating in the event, Eugene Goostman was developed in 2001 by Russian-born Vladimir Veselov, Ukrainian-born Eugene Demchenko, and Russian-born Sergey Ulasen in Saint Petersburg, and is designed to simulate a 13-year old boy in Odessa with an abrasive adolescent personality to match.

"Eugene was 'born' in 2001,” says Veselov. “Our main idea was that he can claim that he knows anything, but his age also makes it perfectly reasonable that he doesn't know everything. We spent a lot of time developing a character with a believable personality. This year we improved the 'dialog controller' which makes the conversation far more human-like when compared to programs that just answer questions. Going forward we plan to make Eugene smarter and continue working on improving what we refer to as 'conversation logic'."

The Turing Test 2014 was held in partnership with RoboLaw, an organization that examines the regulation of robotic technologies, and the judges included Red Dwarf actor Robert Llewellyn and Lord Sharkey.

"Of course the Test has implications for society today," says Warwick. "Having a computer that can trick a human into thinking that someone, or even something, is a person we trust is a wake-up call to cybercrime. The Turing Test is a vital tool for combating that threat. It is important to understand more fully how online, real-time communication of this type can influence an individual human in such a way that they are fooled into believing something is true ... when in fact it is not."

Sources: University of Reading, Eugene Goostman

Update (June 10, 2014): There is some controversy surrounding the claim that Eugene passed the Turing Test. The language of the story has been modified to reflect this.

17 comments
17 comments
Rt1583
It's ironic that the test for gauging humanity in AI requires only 30% to be regarded as successful when so many of human endeavours which are tested for a pass/fail percentage require somewhere around 70%-75% to be deemed passing.
Nairda
And the reds have it !
A true test for this no disrespect is to let the general public at it.
If Eugene was offered the opportunity to talk to thousands where the website recorded random user's votes we would see something better then 33% (which pangs of " we talked three times, and he fooled us once") Hardly a large sample set.
MarylandUSA
It flunks if it relies on dodgy responses like "It's interesting that you say that," "Really? Tell me more" and "Why do you ask?" If I were pitted against it, the first thing I'd tell it would be that I won't accept such responses. I'll bet its reply would be precisely the kind of reply I told it I won't accept.
nutcase
My Aunty Pat would fail the Turing test I reckon.
Beaugrand_RTMC
The Turing Test should be administered to high school seniors as a requirement for graduation.
Also to politicians to qualify them for candidacy.
rocketride
My problem with the Turing test is not that computers would eventually pass it but that so many of the bipedal hominids already here can't even come close. And yet they're allowed to breed, vote, drive, etc.
bajessup
ELIZA was a pseudo-therapist interactive program demonstrated in 1966. It relied quite a bit on reflective sentences, like those Paul Stregevsky noted above, so could be fooled by nonsense comments. e.g. human: "I have a carrot in my ear." ELIZA: "How do you feel about the carrot in your ear?" Some clinical psychology professors used ELIZA to challenge students to think about the nature of "empathy" and psychotherapy. ELIZA is now noted as one of the earliest examples of a chatbot. http://en.wikipedia.org/wiki/ELIZA
Another somewhat different example is the computer "WATSON" which won the TV game show "Jeopardy" in 2011. WATSON has since been used in optimizing cancer treatment. http://en.wikipedia.org/wiki/Watson_(computer)
BlueGull
I think patterning an AI personality after a 13 year old boy is cheating - do you know any who could pass a turing test?
If you think about it, 33% would be pretty good for a real test - remember there is an actual human the bot is being compared to each time - 1/3 the time the judge gets it right, 1/3 it is a tossup (potentially), 1/3 the (human) judge actually gets wrong which the human contact is...
Jay_Wilson
Here's some tips on how to trip up a telemarketing robot which probably aren't on the Turing test:
http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarketer-who-denies-shes-a-robot/
Badger Watkins
No, it didn't. You should know better.
https://www.techdirt.com/articles/20140609/07284327524/no-computer-did-not-pass-turing-test-first-time-everyone-should-know-better.shtml
Load More