Search and Real-time Analytics on Big Data

In this hands-on tutorial, you will learn the importance of distributed search by our industry experience and a specific example. In particular, we’ll introduce the architecture that incorporates distributed search techniques, share pain points experienced and lessons learned. Building atop, we’ll depict the landscape of distributed search tools and their future directions. For the hands-on part of the tutorial, you will learn how to install and use Apache Solr for real-time Big Data analytics, search, and reporting. You’ll also learn some tricks of the trade and how to handle known issues.

• Distributed Search – applications in industry
  • Search applications for structured data
  • Search applications for unstructured data
  • Geo-indexed search
  • Why distributed search? What happens as index size grows with data?

• Use Case: Log Data
  • Requirements
    • Petabytes of semi-structured log data
    • Billions of parsed Solr documents
  • Apache Solr
    • Schema
    • Backups
    • Disaster Recovery
    • Hbase splitting/region allocation
    • Duplicates
    • Joins

• Technology Landscape
  • Solr/Lucene out of the box
  • Integrated Solr Solutions
    • Datastax Enterprise (DSE) 2.0
    • Lily
    • Lucid Imagination
    • Kitenga
    • Katta
  • Non-Solr Solutions
    • Riak
    • MongoDB
    • Amazon cloud search
    • Google BigQuery

• Using the Big Data Search Tutorial Tools

We’ll email instructions to you before the tutorial so you can come prepared with the necessary tools installed and ready to go. This prior preparation will let us use the whole tutorial time to learn some of the fundamentals of the Lucene query language and other important topics. At the beginning of the tutorial we’ll show you how to use these tools.

• Amazon AWS instance pre set up or user setup
• Setup Solr cores
  • Install
  • Load
• Setup HBase
  • Install
  • Load

• Writing Lucene Queries

We’ll spend most of the tutorial using a series of hands-on exercises with actual Lucene queries, so you can learn by doing. We’ll go over all the main features of Lucene’s query language, and how Lucene works with data in Hadoop.

• Advanced Techniques

This section will cover advanced topics such as relevance ranking, facets, group by, sort by, and other important features for Big Data search projects. Lucene enables many types of customizations of the underlying technology.

• Lucene / Solr in the Hadoop Ecosystem

We’ll conclude with a discussion of Lucene’s place in the Hadoop ecosystem, such as how it compares to other available tools. We’ll discuss installation and configuration issues that ensure the best performance and ease of use in a real production cluster. In particular, we’ll discuss how to create an efficient Lucene secondary index on data stored in HBase, Cassandra, and other NoSQL databases.