Distilling Data Exhaust: How to Surface Insights and Build Products

As the world becomes more heavily instrumented, we are collecting massive amounts of raw data which often sits unused in log files or data warehouses. At the same time, statistical techniques, cloud computing, and software frameworks have matured to a point where a small team or even a single person can rapidly extract insights and build products on top of this data.

As a case study, this talk will combine datasets from Twitter, LinkedIn, Wikipedia, Mechanical Turk and other sources to extract insights about the O’Reilly Strata Conference and its attendees.

During the process, we will walk you through the nuts & bolts of building data products using these tools. We will cover coming up with ideas, tracking down or creating data, using visualizations during development, wiring together a prototype, and show some algorithmic tricks that can get you out of a jam.