You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ashish rawat <dc...@gmail.com> on 2017/12/21 20:32:08 UTC
Spark Streaming to REST API
Hi,
We are working on a streaming solution where multiple out of order streams
are flowing in the system and we need to join the streams based on a unique
id. We are planning to use redis for this, where for every tuple, we will
lookup if the id exists, we join if it does or else put the tuple into
redis. Also, we need to write the final out to a system through REST API
(the system doesn't provide any other mechanism to write).
Is it a common pattern to read/write to db per tuple? Also, are there any
connectors to write to REST endpoints.
Regards,
Ashish
Re: Spark Streaming to REST API
Posted by ashish rawat <dc...@gmail.com>.
Sorry, for not making it explicit. We are using Spark Streaming as the
streaming solution and I was wondering if it is a common pattern to do per
tuple redis read/write and write to a REST API through Spark Streaming.
Regards,
Ashish
On Fri, Dec 22, 2017 at 4:00 AM, Gourav Sengupta <go...@gmail.com>
wrote:
> hi Ashish,
>
> I was just wondering if there is any particular reason why you are posting
> this to a SPARK group?
>
> Regards,
> Gourav
>
> On Thu, Dec 21, 2017 at 8:32 PM, ashish rawat <dc...@gmail.com> wrote:
>
>> Hi,
>>
>> We are working on a streaming solution where multiple out of order
>> streams are flowing in the system and we need to join the streams based on
>> a unique id. We are planning to use redis for this, where for every tuple,
>> we will lookup if the id exists, we join if it does or else put the tuple
>> into redis. Also, we need to write the final out to a system through REST
>> API (the system doesn't provide any other mechanism to write).
>>
>> Is it a common pattern to read/write to db per tuple? Also, are there any
>> connectors to write to REST endpoints.
>>
>> Regards,
>> Ashish
>>
>
>
Re: Spark Streaming to REST API
Posted by Gourav Sengupta <go...@gmail.com>.
hi Ashish,
I was just wondering if there is any particular reason why you are posting
this to a SPARK group?
Regards,
Gourav
On Thu, Dec 21, 2017 at 8:32 PM, ashish rawat <dc...@gmail.com> wrote:
> Hi,
>
> We are working on a streaming solution where multiple out of order streams
> are flowing in the system and we need to join the streams based on a unique
> id. We are planning to use redis for this, where for every tuple, we will
> lookup if the id exists, we join if it does or else put the tuple into
> redis. Also, we need to write the final out to a system through REST API
> (the system doesn't provide any other mechanism to write).
>
> Is it a common pattern to read/write to db per tuple? Also, are there any
> connectors to write to REST endpoints.
>
> Regards,
> Ashish
>