Facebook’s Large Scale Monitoring System Built on HBase

Liyin Tang (Facebook), Vinod Venkataraman (Facebook), Charles Thayer (Facebook)
Hadoop: Case Studies Hadoop: Tools & Technology, Gramercy Suite (NY Hilton)
Average rating: ***..
(3.54, 13 ratings)

ODS is Facebook’s internal monitoring system, which collects a variety of system and application level metrics from every server for real-time monitoring, anomaly detection, alerting and analysis.

In this talk, we’ll start by giving an overview of ODS, and go over the manageability and scalability challenges with the previous MySQL based setup. We will then discuss our motivation to choose HBase for this workload, and share a series of lessons learnt from building ODS on top of HBase, including takeaways from our attempts to scale the HBase cluster separately from the HDFS cluster, a dual-HBase architecture for high availability, migration data from MySQL to HBase, running MapReduce jobs for time-based rollup of metrics.

Photo of Liyin Tang

Liyin Tang

Facebook

Liyin Tang is a software engineer at Facebook and a HBase Committer at the Apache Software Foundation. At Facebook, he works on building the data storage system based on HBase for various applications. Liyin holds a bachelor degree in Software Engineering from Shanghai Jiao Tong University, China and a master degree in Computer Science from University of Southern California, US.

Photo of Vinod Venkataraman

Vinod Venkataraman

Facebook

Vinod Venkataraman is a Software Engineer at Facebook, where he focuses on developing the in-house monitoring systems. Vinod holds a Master’s degree from the University of Texas at Austin, and a Bachelor’s degree from the National Institute of Technology, Trichy, India, both in Computer Science.

Photo of Charles Thayer

Charles Thayer

Facebook

Charles Thayer is a Software Engineer at Facebook, where he works on the Monitoring Systems. Before Facebook, he worked at Yahoo on search technology including the Web Crawler and Hosted Vertical Crawler. His focus has been scaling both storage and compute resources across thousands of nodes and tens of thousands of disks. He’s been involved with many startups in NYC including Metrobeat/Citysearch and CityRealty. He graduated with a BS EE from Columbia University’s School of Engineering and Applied Science, before becoming CEO of his first startup, Mediabridge Infosystems.

Comments on this page are now closed.

Comments

Picture of Liyin Tang
Liyin Tang
10/29/2012 1:32pm EDT

FYI: I have uploaded the slides here.

Leitao Guo
10/26/2012 2:07am EDT

could you please share the ppt?

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities, contact Susan Stewart at sstewart@oreilly.com.

Media Partner Opportunities

For information on trade opportunities contact Kathy Yu at mediapartners
@oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Strata contacts.