End-to-End Web Navigation

Traditional search engines have three main processing phases: crawling, inverted ­index construction, and candidate documents retrieval. Our retrieval method, instead, has only one: Navigation. We built a neural net based agent that navigates through a website, such as Wikipedia, to find a web page that contains the answer to the question. During training, it learns to assign high probabilities to the hyperlinks that will lead to the correct page and it stops when it is confident that the current page contains the answer to the question.