High Availability for the HDFS NameNode: Phase 2

Aaron Myers (Cloudera, Inc.), Todd Lipcon (Cloudera, Inc.)
Hadoop: Tools & Technology, Grand East (NY Hilton)
Average rating: ***..
(3.67, 3 ratings)

The initial implementation of a highly-available NameNode was completed and merged to Apache Hadoop trunk in February 2012. The design featured an Active NameNode with a hot Standby NameNode.

In this initial implementation, failure of the Active NameNode was not detected automatically, requiring an operator to initiate a failover from Active to Standby. This provided the Hadoop operator the ability to dramatically reduce the frequency of planned HDFS downtime due to configuration changes, software upgrades, hardware maintenance, etc. However, requiring operator intervention is clearly insufficient for preventing unplanned HDFS downtime. Since the initial implementation of HDFS HA, we have developed a system for monitoring the health of the Active NameNode and automatically triggering a failover to the Standby NameNode when the Active is no longer able to provide service.

In order to share pertinent file system state between the Active and Standby NameNodes, the initial implementation of HDFS HA was reliant upon a highly-available NFS filer. While many organizations are willing to purchase and administer such a service, removing this dependency can improve lower the administrative overhead and potentially improve the reliability of operating HDFS.

This session will discuss the design and implementation of these features, as well as give an overview of how to deploy these new features.

Photo of Aaron Myers

Aaron Myers

Cloudera, Inc.

Aaron T. Myers is a Software Engineer at Cloudera and an Apache Hadoop Committer. Aaron’s work is primarily focused on HDFS. Prior to joining Cloudera, Aaron was a Software Engineer and VP of Engineering at Amie Street, where he worked on all components of the software stack, including operations, infrastructure, and customer-facing feature development. Aaron holds both an Sc.B. and Sc.M. in Computer Science from Brown University.

Todd Lipcon

Cloudera, Inc.

Todd holds a Sc.B in Computer Science from Brown University, where he completed an honors thesis developing a new collaborative filtering algorithm for the Netflix Prize Competition. Todd interned at Google developing machine learning methods to detect credit card fraud on AdWords and Google Checkout. Currently Todd works at Cloudera, Inc on bringing Map/Reduce technology to enterprises.

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities, contact Susan Stewart at sstewart@oreilly.com.

Media Partner Opportunities

For information on trade opportunities contact Kathy Yu at mediapartners
@oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Strata contacts.