Map Reduce begat Hadoop begat Big Data. NoSQL moved us away from the stricture of monolithic storage architectures to fit-for-purpose designs. But, Houston, we still have a problem – architects are still designing systems like they did in the ‘70s. Yet most systems are still designed for store-then-compute rather than to observe, orient, decide and act on in-flight data. Rather than just taking a spoonful of stream processing from large web companies with well intended open source engines, such as the Lambda Architecture, let’s look at the crazy architectures in high frequency trading such as Complex Event Processing (CEP). They got the techniques right, but the solution wrong – it didn’t scale. This talk isn’t about big, slow data; this talk isn’t about small, fast data. This talk takes a nuanced path through most of the rest which sits somewhere in between, with tradeoffs aplenty.
