Schedule: Order of Magnitude Shifts sessions

When it comes to big data insights, how do you know you’re asking the right questions? Hiring data scientists is a good start – we’re seeing their growth both on LinkedIn and at LinkedIn. But even data scientists are not immune from the myriad of hidden pitfalls that keep your key insights out of sight.

Drawing from a deceptively simple exercise that I’ve used to haze dozens of data scientists on their first day, I will discuss the good, the bad and the ugly lessons we’ve learned about asking the right questions, denominators and being a data skeptic.

By using social media metrics, TV networks can now get ratings for their programs in real time.

As the world of data expands, the challenge of understanding it expands as well. Narrative Science is addressing this challenge with a software platform that uses data to drive the generation of compelling narratives that tell the stories contained within it. The technology tells the stories that are hidden in the numbers.

Disruptive technology shapes the world, defining political, military, financial, and commercial opportunities and threats. Whether originating in academic research, in National Labs, or in privately held or public companies, these technologies can emerge with explosive impact, creating and destroying value. Yet there are few tools to track these innovations—at a global scale and at a pace that keeps up with the rate of change.

What if corporate strategists could literally draw a map to find growth opportunities? A technique called semantic clustering analysis makes this possible. When applied to technology entities worldwide, this analysis can reveal not only which innovation areas are thick with competition, but also where in the market there are opportunities, or “white spaces,” ripe for innovation. The result is a data-driven visual tool that can be used to drive corporate innovation strategy.

Digital media publishers sit astride torrents of data about their content, audiences and advertisers. Every day, the world’s two billion internet users visit millions of pages, and are exposed to more than one hundred billion online advertising impressions. Publishers of digital content need to crunch this data to predict who will visit tomorrow, and how to price the ads those visitors see.

In the past year, Metamarkets has developed a Big Data stack for the media markets, comprised of three pieces:

1. a stream-based aggregation and processing service, built on top of the Kafka messaging system
2. a distributed, in-memory OLAP store, hosted on Amazon’s web services, that can scan and aggregate over one billion rows per second
3. a real-time web console, served by node.js, with dynamic data visualizations powered by D3, the successor to Protovis.

   Our platform is in currently in production with three of the largest media publishers in Europe and United States.

Even the biggest commercial datasets can’t rival those produced at CERN, home of the Large Hadron Collider. As scientists research the conditions that created our universe, the volume of data captured is truly staggering.

We all need clean, useful, validated data—but most of us don’t have an easy way to gather it and the data we do have is messy. Over the last year I’ve spent my time hanging with crowds—crowds of tech savvy, mobile-device enabled modern day distributed workers who are changing the way we should think about acquiring and managing data. These workers can convert a Masterpiece like Moby Dick into an artful expression by translating it into emoji—or they can walk across the nation validating in-store beer pricing, product placement and restaurant times. Hear some of the numbers behind these crowds and how startups like Gigwalk are changing the way we work.