You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Soumitra Kumar <ku...@gmail.com> on 2014/09/23 22:28:21 UTC

SPARK-3660 : Initial RDD for updateStateByKey transformation

Hello fellow developers,

Thanks TD for relevant pointers.

I have created an issue :
https://issues.apache.org/jira/browse/SPARK-3660

Copying the description from JIRA:
"
How to initialize state tranformation updateStateByKey?

I have word counts from previous spark-submit run, and want to load that in next spark-submit job to start counting over that.

One proposal is to add following argument to updateStateByKey methods.
initial : Option [RDD [(K, S)]] = None

This will maintain the backward compatibility as well.

I have a working code as well.

This thread started on spark-user list at:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-initialize-updateStateByKey-operation-td14772.html
"

Please let me know if I shall add a parameter "initial : Option [RDD [(K, S)]] = None" to all updateStateByKey methods or create new ones?

Thanks,
-Soumitra.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: SPARK-3660 : Initial RDD for updateStateByKey transformation

Posted by Soumitra Kumar <ku...@gmail.com>.
Hello,

I have submitted a pull request (Adding support of initial value for state update. #2665), please review and let me know.

Excited to submit my first pull request.

-Soumitra.

----- Original Message -----
From: "Soumitra Kumar" <ku...@gmail.com>
To: dev@spark.apache.org
Sent: Tuesday, September 23, 2014 1:28:21 PM
Subject: SPARK-3660 : Initial RDD for updateStateByKey transformation

Hello fellow developers,

Thanks TD for relevant pointers.

I have created an issue :
https://issues.apache.org/jira/browse/SPARK-3660

Copying the description from JIRA:
"
How to initialize state tranformation updateStateByKey?

I have word counts from previous spark-submit run, and want to load that in next spark-submit job to start counting over that.

One proposal is to add following argument to updateStateByKey methods.
initial : Option [RDD [(K, S)]] = None

This will maintain the backward compatibility as well.

I have a working code as well.

This thread started on spark-user list at:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-initialize-updateStateByKey-operation-td14772.html
"

Please let me know if I shall add a parameter "initial : Option [RDD [(K, S)]] = None" to all updateStateByKey methods or create new ones?

Thanks,
-Soumitra.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org