The Great Railway Caper: Big Data in 1955

John Graham-Cumming (CloudFlare)
Location: King's Suite
Average rating: ****.
(4.60, 10 ratings)

It’s 1951 and you’ve got the world’s first business computer and you’ve just been handed a Big Data problem. Go! With 2K of memory it was powerful enough to run the then massive Lyons business. But it wasn’t long, in 1955, before Big Data came calling in the form of a request from British Rail to calculate the shortest distance between every one of their 5,000 railway stations.

The British Rail network was, of course, a graph with edges labelled with the distance between the stations, but graph algorithms hadn’t been invented yet. Dijkstra’s Shortest Path algorithm which is widely used today wouldn’t be described until 1959. And with 5,000 stations there were 12.5 million distance pairs to calculate exceeding the memory capacity of LEO by four orders of magnitude (equivalent today of a single large machine being faced with a petabyte to calculate).

What could be more modern? Data that exceeds available memory by orders of magnitude; a graph structure; the need for new algorithms.

In this talk I’ll tell the story of Roger Coleman who worked out how to partition the problem into manageable parts, discovered Dijkstra’s algorithm, and hand coded the solution in assembly language.

Photo of John Graham-Cumming

John Graham-Cumming

CloudFlare

John Graham-Cumming is computer programmer and author. He studied mathematics and computation at Oxford and stayed for a doctorate in computer security. As a programmer he has worked in Silicon Valley and New York, the UK, Germany and France. His open source POPFile program won a Jolt Productivity Award in 2004.

He is the author of a travel book for scientists published in 2009 called The Geek Atlas and has written articles for The Times, The Guardian, The Sunday Times, The San Francisco Chronicle, New Scientist and other publications.

He can be found on the web at jgc.org and on Twitter as @jgrahamc.

If you’ve heard of him at all, it’s likely because in 2009 he successfully petitioned the British Government to apologize for the mistreatment of British mathematician Alan Turing.

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities, contact Susan Stewart at sstewart@oreilly.com or +1 (707) 827-7148

Media Partner Opportunities

For information on trade opportunities contact Kathy Yu at mediapartners
@oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Strata contacts.