You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by david w <df...@gmail.com> on 2015/09/17 23:56:43 UTC
Spark streaming to database exception handling
I am using spark stream to receive data from kafka, and then write result rdd
to external database inside foreachPartition(). All thing works fine, my
question is how can we handle no data loss if there is database connection
failure, or other exception happened during write data to external storage.
Is there any way we can notify spark streaming to replay that RDD or some
ack mechanism?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-to-database-exception-handling-tp24728.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark streaming to database exception handling
Posted by chyeers <99...@qq.com>.
I have the same problem. I had use spark streaming to save data to hbase with
using JDBC of phoenix .It seems to be ok sometimes,but missing data while
the exception happened during write data to external
storagorg.apache.phoenix.exception.BatchUpdateExecution: ERROR 1106 (XCL06):
Exception while executing batch
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-to-database-exception-handling-tp24728p24741.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark streaming to database exception handling
Posted by Cody Koeninger <co...@koeninger.org>.
If you fail the task (throw an exception) it will be retried
On Thu, Sep 17, 2015 at 4:56 PM, david w <df...@gmail.com> wrote:
> I am using spark stream to receive data from kafka, and then write result
> rdd
> to external database inside foreachPartition(). All thing works fine, my
> question is how can we handle no data loss if there is database connection
> failure, or other exception happened during write data to external storage.
> Is there any way we can notify spark streaming to replay that RDD or some
> ack mechanism?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-to-database-exception-handling-tp24728.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>