You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sancheng <sa...@gmail.com> on 2016/12/06 01:35:19 UTC

[Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

Hello,

we are trying to use Spark streaming to do some billing related application.
so  our case is 
that we need to correlate two different messages, and calculate the time
invterval between two
messages, the two message should be in same partition but probabaly not in
the same RDD,
it seems spark DStream.join can only join between current RDDs. 

We want to avoid store the erailer message to some datastore but prefer to
use natvie solution if
these is any in Spark.

pls advice, thanks!





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-How-to-do-join-two-messages-in-spark-streaming-Probabaly-messasges-are-in-differnet--tp28161.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

Posted by Tathagata Das <ta...@gmail.com>.
This sounds like something you can solve by a stateful operator. check out
mapWithState. If both the message can be keyed with a common key, then you
can define a keyed-state. the state will have a field for the first
message.When you see the first message for a key, fill the first field with
timestamp, etc. Then when the second message of the same key arrives, Spark
Streaming will ensure that it calls your state update function with old
state (i.e. first message filled up) and you can take the time difference.

Check out my blog -
https://databricks.com/blog/2016/02/01/faster-stateful-stream-processing-in-apache-spark-streaming.html

On Tue, Dec 6, 2016 at 5:50 PM, sancheng <sa...@gmail.com> wrote:

> any valuable feedback is appreciated!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Spark-Streaming-How-to-do-join-two-
> messages-in-spark-streaming-Probabaly-messasges-are-in-
> differnet--tp28161p28163.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

Posted by sancheng <sa...@gmail.com>.
any valuable feedback is appreciated!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-How-to-do-join-two-messages-in-spark-streaming-Probabaly-messasges-are-in-differnet--tp28161p28163.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org