You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Gaurav Sehgal <gs...@gmail.com> on 2018/07/05 12:34:43 UTC
Batch processing.
Hello,
Is there another framework like Apache Storm, which does batch
processing of Data. I have been looking at Apache Spark, but the use cases
it addresses are more of Map Reduce nature. The use case we are looking at
is to read data from data source such as Mongo in batches and upload in
ElasticSearch.
Regards,
Gaurav
Re: Batch processing.
Posted by Gaurav Sehgal <gs...@gmail.com>.
Thanks Nicolas
On Thu, Jul 5, 2018 at 6:06 PM Nicolas Maillard <nm...@hortonworks.com>
wrote:
> Hello Gaurav
>
>
> I think you may find what you are looking for in Apache Nifi
>
> https://nifi.apache.org/
>
>
> *Nicolas Maillard*
>
> *Manager Solution Engineer – SEMEA*
>
>
> *Mobile*: +33 6 68 17 66 05
>
> Website:http://www.hortonworks.com
>
> Blog:http://www.hortonworks.com/blog
>
>
>
> ------------------------------
> *From:* Gaurav Sehgal <gs...@gmail.com>
> *Sent:* Thursday, July 5, 2018 2:34 PM
> *To:* user@storm.apache.org
> *Subject:* Batch processing.
>
> Hello,
> Is there another framework like Apache Storm, which does batch
> processing of Data. I have been looking at Apache Spark, but the use cases
> it addresses are more of Map Reduce nature. The use case we are looking at
> is to read data from data source such as Mongo in batches and upload in
> ElasticSearch.
>
> Regards,
> Gaurav
>
>
Re: Batch processing.
Posted by Nicolas Maillard <nm...@hortonworks.com>.
Hello Gaurav
I think you may find what you are looking for in Apache Nifi
https://nifi.apache.org/?
Nicolas Maillard
Manager Solution Engineer - SEMEA
Mobile: +33 6 68 17 66 05
Website:http://www.hortonworks.com<http://www.hortonworks.com/>
Blog:http://www.hortonworks.com/blog
________________________________
From: Gaurav Sehgal <gs...@gmail.com>
Sent: Thursday, July 5, 2018 2:34 PM
To: user@storm.apache.org
Subject: Batch processing.
Hello,
Is there another framework like Apache Storm, which does batch processing of Data. I have been looking at Apache Spark, but the use cases it addresses are more of Map Reduce nature. The use case we are looking at is to read data from data source such as Mongo in batches and upload in ElasticSearch.
Regards,
Gaurav
Re: Batch processing.
Posted by Arun Mahadevan <ar...@apache.org>.
> I'm aware of some ideas so far: one simple idea is killing topology when spout gets all acknowledge messages and data source has no more data.
There would be more work needed to optimize the shuffle and scheduling (batch), but I agree with minor modifications to the Spout API (like having a batch Spout that returns some End-of-data marker) and waiting till all messages are ack-ed, Storm can address most of the simple bounded use cases (batch processing).
From: Jungtaek Lim <ka...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, July 5, 2018 at 5:49 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Batch processing.
Hi Gaurav,
While streaming of Apache Spark lacks some features like branch because it is bound to MR nature, I guess most of cases it doesn't matter when you are doing batch.
If you are looking for batch solution which is streaming-oriented nature (so ingesting finite source to streaming engine), Apache Flink would be the thing to consider.
If you don't mind to define spout working with finite data source you can still consider Apache Storm. I'm aware of some ideas so far: one simple idea is killing topology when spout gets all acknowledge messages and data source has no more data.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 7월 5일 (목) 오후 9:35, Gaurav Sehgal <gs...@gmail.com>님이 작성:
Hello,
Is there another framework like Apache Storm, which does batch processing of Data. I have been looking at Apache Spark, but the use cases it addresses are more of Map Reduce nature. The use case we are looking at is to read data from data source such as Mongo in batches and upload in ElasticSearch.
Regards,
Gaurav
Re: Batch processing.
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Gaurav,
While streaming of Apache Spark lacks some features like branch because it
is bound to MR nature, I guess most of cases it doesn't matter when you are
doing batch.
If you are looking for batch solution which is streaming-oriented nature
(so ingesting finite source to streaming engine), Apache Flink would be the
thing to consider.
If you don't mind to define spout working with finite data source you can
still consider Apache Storm. I'm aware of some ideas so far: one simple
idea is killing topology when spout gets all acknowledge messages and data
source has no more data.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 7월 5일 (목) 오후 9:35, Gaurav Sehgal <gs...@gmail.com>님이 작성:
> Hello,
> Is there another framework like Apache Storm, which does batch
> processing of Data. I have been looking at Apache Spark, but the use cases
> it addresses are more of Map Reduce nature. The use case we are looking at
> is to read data from data source such as Mongo in batches and upload in
> ElasticSearch.
>
> Regards,
> Gaurav
>
>
Re: Batch processing.
Posted by Arun Mahadevan <ar...@apache.org>.
Your use case seems to be a simple ETL (read from a data source and write to a Sink), which is very well addressed by Storm. With Storm you don’t necessarily need to split the data into batches, but can continuously load the data into ES. If your data set is bounded, you can just kill the topology after all the data has been processed.
Thanks,
Arun
From: Gaurav Sehgal <gs...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, July 5, 2018 at 5:34 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Batch processing.
Hello,
Is there another framework like Apache Storm, which does batch processing of Data. I have been looking at Apache Spark, but the use cases it addresses are more of Map Reduce nature. The use case we are looking at is to read data from data source such as Mongo in batches and upload in ElasticSearch.
Regards,
Gaurav