CEP/Storm and Big Data Analytics
Not only have data sets gotten larger, but the need to gain insights from it more quickly have increased. These challenges have driven new breeds of engines categorized as Complex Events Processing (CEP) or more broadly known as Distributed Computation Engines. These engines perform very similar functions to ETL platforms however they more often interact with message queues than databases and they process continuously rather than in batch. Storm, ESPER, and Akka are the leading platforms in this space. All three are open source, highly scalable and perform extremely well.
In a sophisticated application architecture, there is no need to land to a target system, so we can batch process it later. The analytics system now becomes a more integrated part of the strategic application architecture more than just a downstream consumer.
Storm – born from big data challenges
When Twitter user and tweet count began to grow there was a need for a processing scale that didn’t exist. To deal with this, Storm was born to provide real-time insight, and process and route their extreme payload of messages. With the ability to process thousands to millions of messages per second with full message guarantees, it provides an amazing engine for real time analytics. Now, this technology has been adopted by a wide range of industries ranging from eCommerce to Financial institutions with great success.
Caserta Concepts has the real world experience in implementing high volume/low latency analytic solutions on these cutting edge platforms. Contact us to learn how we can help you bring your big data solution into real-time using CEP/Storm.