You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shyla deshpande <de...@gmail.com> on 2016/11/22 20:55:08 UTC
How do I persist the data after I process the data with Structured streaming...
Hi,
Structured streaming works great with Kafka source but I need to persist
the data after processing in some database like Cassandra or at least
Postgres.
Any suggestions, help please.
Thanks
Re: How do I persist the data after I process the data with
Structured streaming...
Posted by shyla deshpande <de...@gmail.com>.
Has anyone written a custom sink to persist data to Cassandra from
structured streaming.
Please provide me any link or reference. Thanks
On Tue, Nov 22, 2016 at 2:40 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> Forgot the link: https://spark.apache.org/docs/latest/structured-
> streaming-programming-guide.html#using-foreach
>
> On Tue, Nov 22, 2016 at 2:40 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> We are looking to add a native JDBC sink in Spark 2.2. Until then you
>> can write your own connector using df.writeStream.foreach.
>>
>> On Tue, Nov 22, 2016 at 12:55 PM, shyla deshpande <
>> deshpandeshyla@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Structured streaming works great with Kafka source but I need to persist
>>> the data after processing in some database like Cassandra or at least
>>> Postgres.
>>>
>>> Any suggestions, help please.
>>>
>>> Thanks
>>>
>>
>>
>
Re: How do I persist the data after I process the data with
Structured streaming...
Posted by Michael Armbrust <mi...@databricks.com>.
Forgot the link:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach
On Tue, Nov 22, 2016 at 2:40 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> We are looking to add a native JDBC sink in Spark 2.2. Until then you can
> write your own connector using df.writeStream.foreach.
>
> On Tue, Nov 22, 2016 at 12:55 PM, shyla deshpande <
> deshpandeshyla@gmail.com> wrote:
>
>> Hi,
>>
>> Structured streaming works great with Kafka source but I need to persist
>> the data after processing in some database like Cassandra or at least
>> Postgres.
>>
>> Any suggestions, help please.
>>
>> Thanks
>>
>
>
Re: How do I persist the data after I process the data with
Structured streaming...
Posted by Michael Armbrust <mi...@databricks.com>.
We are looking to add a native JDBC sink in Spark 2.2. Until then you can
write your own connector using df.writeStream.foreach.
On Tue, Nov 22, 2016 at 12:55 PM, shyla deshpande <de...@gmail.com>
wrote:
> Hi,
>
> Structured streaming works great with Kafka source but I need to persist
> the data after processing in some database like Cassandra or at least
> Postgres.
>
> Any suggestions, help please.
>
> Thanks
>