How to bulk insert using spark streaming job

[amit kumar singh <am...@gmail.com>]

How to bulk insert using spark streaming job 

Sent from my iPhone

Re: How to bulk insert using spark streaming job

[scorpio <sa...@gmail.com>]

You need to insert per partition per batch. Normally database drivers meant
for spark have bulk update feature built in. They take a RDD and do a bulk
insert per partition.
In case db driver you are using doesn't provide this feature, you can
aggregate records per partition and then send out to db by writing your own
code.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: How to bulk insert using spark streaming job

[ayan guha <gu...@gmail.com>]

by writing code, I suppose :) Jokes apart, I think you need to articulate
the problem with more details for others to help.

Do you mean you want to batch up data in memory and then write as a chunk?
Where do want to insert? Etc etc...

On Fri, Apr 20, 2018 at 1:08 PM, amit kumar singh <am...@gmail.com>
wrote:

> How to bulk insert using spark streaming job
>
> Sent from my iPhone
>



-- 
Best Regards,
Ayan Guha