Human Expertise and Artificial Intelligence in Vertical Search

Peter Jackson (Thomson Reuters)
Location: Mission City B4
Average rating: ***..
(3.57, 7 ratings)

Our talk summarizes some recent thinking in the field of data-driven vertical search and illustrates it in the context of a new version of Westlaw, called WestlawNext. In particular, we examine the paradox that it is hard to gain user acceptance for more sophisticated search algorithms, even if the results are demonstrably better than keyword searching. There needs to be an explanatory model of how relevant documents are retrieved and ranked that is well grounded in the actual performance of the search engine. Our approach is based upon a relatively simple story about how our algorithms leverage the expertise of both editors and users to generate meta-data that improves search. Improved recall depends upon editorial enhancements interpreted by natural language processing to find the long tail, while voluminous online user behavior feeds machine learning programs that improve precision in the top ranks. We argue that getting the right allocation of function between person and machine is the key to making specialist content more findable and search results more understandable.

Photo of Peter Jackson

Peter Jackson

Thomson Reuters

Peter Jackson holds a Ph.D. in Artificial Intelligence from the University of Leeds. He then joined the Department of Artificial Intelligence at Edinburgh University, earning tenure in 1987 and publishing “Introduction to Expert Systems” with Pearson, a textbook that has run to three editions and been translated into seven languages.

In 1988, he moved to the US as a principal scientist at McDonnell Douglas Research Laboratories, bringing out a book of papers with MIT Press entitled “Logic Based Knowledge Representation” in 1989. He has also taught courses at University of Washington in Saint Louis, Clarkson University in New York, and Singapore Polytechnic.

In 1995, he joined the Thomson Corporation, and he is now chief scientist and vice president of R&D at Thomson Reuters Professional. He runs a group of 40 researchers who work with business units in the legal, financial, science and healthcare sectors to deliver custom information solutions. His latest book, “Natural Language Processing for Online Applications”, came out in a 2nd edition with John Benjamins in 2007.

Comments on this page are now closed.

Comments

Anthony Cassandra
02/02/2011 10:23pm PST

Very well organized and presented case-study comparing horizontal and vertical search with good meta-points about vertical search in general. Having some deep experience in vertical search, I found their meta-points to be either right on the mark or important aspects of the problem space that I had not previously considered.

Sponsors

  • Thomson Reuters
  • EMC Data Computing Division
  • EnterpriseDB
  • Microsoft
  • Gnip
  • Rackspace Hosting
  • IBM
  • Windows Azure MarketPlace DataMarket
  • Amazon Mechanical Turk
  • Amazon Web Services
  • Aster Data
  • Cloudera
  • Clustrix
  • DataStax, Inc. (formerly Riptano, Inc.)
  • Digital Reasoning Systems
  • Heritage Provider Network
  • Impetus
  • Jaspersoft
  • Karmasphere
  • LinkedIn
  • MarkLogic
  • Pentaho
  • Pervasive
  • Revolution Analytics
  • Splunk
  • Urban Mapping
  • Wolfram|Alpha
  • Esri
  • ParAccel
  • Tableau Software

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Susan Young at syoung@oreilly.com

Download the Strata Sponsor/Exhibitor Prospectus

Contact Us

View a complete list of Strata Contacts