This actually worked really well for me for factual information, for example, I asked "When was the World Trade Center Built?" and it gave me the correct date. For a lot of factual stuff, it worked well.
There are some catches with word similarities, for example, "What is the lifespan of a turtle?" worked fine, but "How long can a turtle live?" and various re-phrasings result in very different answers, most of them correct within the context of the content they were scraped from, but some resulted in no answers at all.
Separately, the system doesn't handle preposterous questions very well, like "Who won the World Cup in 2009?" it says Italy won it, but I think it was some other thing referred to as a World Cup.
There are a lot of questions on Quora about personal experience like what it's like to have a family member die of cancer or what it was like to be in the WTC during the 9/11 attacks, and questions about these kinds of experience return nothing for the most part, despite being available via other resources (it's not all locked into Quora).
Very great first go, I'm really interested in seeing how this progresses. Like you say, if you could scrape a lot more data, this would produce a very interesting spectrum of information.
It'd probably be good to scrape some of the "How to" sites too, I asked "How does a gasoline engine work?" and the responses were... funny? :)
I think crowd-based approaches and machine-based approaches will always have very different strengths and weaknesses. The quality of algorithmic answers will generally (always?) be well below that of human-written ones, especially at first.
However, in the long term I can see machine-based approaches "winning", simply because of their near-infinite scalability (and the sort of insight that emerges from enormous scale). Take web search: if started out with hand-curation in the portal era, until that got wiped out by algos and scrapers that could better manage the growing scale of the web (and exploit patterns only visible at scale, like PageRank).
Human knowledge, including the sum of personal experiences of the people who express themselves on the web, is now at a size where it is no longer human-manageable. I see enormous potential here to extract, understand and navigate that knowledge using algorithms.
Yep, I agree on the whole. The problem that I've seen in a lot of cases is that the resources that collect knowledge and display it on the web don't necessarily incentivize the knowledgeable to contribute or reward them for doing so in the best possible way.
What you end up with is a high signal to noise for quality, factual content, and for things where facts are not even expected (politics, etc...) you often end up with a polarity that doesn't acknowledge the other side exists. Beyond all that there is a huge amount of sarcasm, humor and nuance to content that will be hard for a machine to figure out on its own, but hopefully you can find a way to separate the wheat from the chaff :)
In the longer term, I'd like to read the entirety of the web and build a complete semantic bayesian map matching a maximum of knowledge items. Also, it would be nice to have access to a visualization tool for the different answers available and their frequency across sectors of opinion, thus solving the problem of subjectivity (which Wikipedia deals with by completely ignoring it).
If you haven't already, check out Massimo Marchiori's ideas for his (now defunct) Volunia search engine [1]. You are definitely onto something potentially revolutionary here.
My favorite test for this sort of thing is asking questions about "President Bush." Any successful engine is going to have to ask for clarification: Do you mean senior or junior? This chokes on questions about Bush.
I am working on handling ambiguity, as well as questions that don't have one definite answer (eg. "how should I invest 10k at a 5 year term?"). Possibly through an interface that shows a spectrum of answers and their sources.
Asking for clarification would also be much needed, of course.
Keep in mind this engine is still a very early prototype.
It got the reference, but didn't answer my question:
> "what is the airspeed velocity of an unladen swallow?"
> "Monty Python and the Holy Grail is a 1975 British comedy film written and performed by the comedy group of Monty Python (Graham Chapman, John Cleese, Terry Gilliam, Eric Idle, Terry Jones and Michael Palin), as directed by Gilliam and Jones. It was conceived during the hiatus between the third and fourth series of their popular BBC television programme Monty Python's Flying Circus. "
just commenting that an earlier draft of your post mentioned something specific about [the space that systems like this attempt to map] that we both know is overreaching our knowledge of what knowledge is.
Oh, I get it now. Yup, I went overboard and then edited it back.
These things always suck so badly that I don't think I've ever gotten a good answer out of one. This, astonishingly, gave me two right off the bat. But yes, the bar could hardly be lower.
My question to the machine: Who is the happiest man in the world, the answer of the engine: Forbes Magazine named Carlos Slim Helu, a Mexican businessman, the richest man in the world as of March 2010. So the engine assimilate happiness with beying the richest man.
Perhaps a bit different, but Id love to see this implemented for specific subsets of information. Even better, a DIY version where I could enter my own QAs (or get them crowdsourced). There could be a lot of use cases for such an app.
There are some problems with answers form SO, example "What is monad?", but nice overall. Some answers could be combined into one like "Where is New Zealand?"
On the space of all valid questions, there are many it can't answer. Judging from my logs, it's still doing pretty well on most questions. Even though people are trying hard to trick it with impossible questions.
Maybe the engine could answer something like this:
"If There's a God, God invented reality, therefore God is not real, because He transcends reality"
It would not answer what you expect, cause theres no objective answer to that, but the engine could play a game with the words you used showing some "sense of humor"
There are some catches with word similarities, for example, "What is the lifespan of a turtle?" worked fine, but "How long can a turtle live?" and various re-phrasings result in very different answers, most of them correct within the context of the content they were scraped from, but some resulted in no answers at all.
Separately, the system doesn't handle preposterous questions very well, like "Who won the World Cup in 2009?" it says Italy won it, but I think it was some other thing referred to as a World Cup.
There are a lot of questions on Quora about personal experience like what it's like to have a family member die of cancer or what it was like to be in the WTC during the 9/11 attacks, and questions about these kinds of experience return nothing for the most part, despite being available via other resources (it's not all locked into Quora).
Very great first go, I'm really interested in seeing how this progresses. Like you say, if you could scrape a lot more data, this would produce a very interesting spectrum of information.
It'd probably be good to scrape some of the "How to" sites too, I asked "How does a gasoline engine work?" and the responses were... funny? :)