Fortune reports that Munster split the questions into two halves, asking 800 on a busy street and 800 in a quiet room. Munster then published his results in a note to clients last week.
When testing Google, the search engine understood 100 percent of the typed in questions and replied accurately 86 percent of the time, earning a B+ from the analyst.
Siri understood 83 percent of the queries in noisy conditions, and 89 percent in a quiet room. Siri gave an accurate reply 62 percent of the time on the busy streets and 68 percent in a quiet environment. This earned Siri a grade D for accuracy, according to Munster.
“In order to become a viable mobile search alternative, Siri must match or surpass Google’s accuracy of B+ and move from a grade D to a B or higher,” writes Munster.
The release of iOS 6 this spring and the decrease in reliance on Google could help Siri catch up, said Munster, who believes that the iPhone’s assistant is two years behind Google in its learning curve.
But Google is fighting back with its own smart assistant, Google Now, which was announced at the company’s annual I/O developer conference last week.
“Breaking down Siri’s reliance further, Google provides 100 percent of navigation results, 61 percent of information results, 48 percent of commerce results and 42 percent of local results,” wrote Munster. “Among other result aggregators, Yelp provided the most local results (51 percent) and commerce results (51 percent), while WolframAlpha provided 34 percent of information results.”
With the release of iOS 6, Apple’s new maps service will provide all of the navigation results, Yahoo Sports will be added for sports information, Open Table will provide detail about restaurants, Rotten Tomatoes will give movie show times and Fandango will allow ticket purchases. How these changes will affect Siri’s accuracy is not yet known.
Munster highlighted several types of mistakes made by Siri in his test, the most common error being a response with the answer to the previous query.
When asked where Elvis was buried, Siri said “I can’t answer that for you,” mistaking ‘Elvis buried’ for the name of a person.
The pin would sometimes drop in the wrong place when Munster asked Siri “Where am I?”
Several other examples showed how Siri can pick up on particular words in a query, providing related information that does not answer the question asked.
Apple co-founder Steve Wozniak recently said that he thinks Apple ruined Siri, calling the voice recognition assistant “poo-poo”. Charming.
How have you found your experience with Siri? Let us know in the comments section below.