Sample project on real time ingestion of more than 1Billion events at a time

[Deepak Sharma <de...@gmail.com>]

Hi All
Can anyone point me to sample code that can scale upto real time ingestion
of events in millions or billion ?
Also i need reference to which analytic engine to store the transformed
events for real time querying .

Thanks a lot in advance.

--Deepak

Re: Sample project on real time ingestion of more than 1Billion events at a time

[Timo Walther <tw...@apache.org>]

Hi Deepak,

actually, every Flink example program can scale up to millions of events 
and more. The Flink APIs are designed to abstract the business logic 
from the parallelism. You just need to implement the interfaces that 
Flink provides.

If you are interesting in some example program, I can recommend this 
blog post [1]. A typical storage after processing, can be Kafka, 
EleasticSearch or you skip the additional system and use Flink's 
queryable state feature [2].

Hope that helps.

Regards,
Timo

[1] https://data-artisans.com/blog/extending-the-yahoo-streaming-benchmark
[2] https://data-artisans.com/blog/queryable-state-use-case-demo



Am 10/23/17 um 4:42 PM schrieb Deepak Sharma:
> Hi All
> Can anyone point me to sample code that can scale upto real time 
> ingestion of events in millions or billion ?
> Also i need reference to which analytic engine to store the 
> transformed events for real time querying .
>
> Thanks a lot in advance.
>
> --Deepak