You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Something Something <ma...@gmail.com> on 2015/10/12 17:47:57 UTC

Creating Custom Receiver for Spark Streaming

Is it safe to assume that Spark will always create a single instance of
Custom Receiver? Or would it create multiple instances on each node in a
cluster? Wondering if I need to worry about receiving the same message on
different nodes etc.

Please help. Thanks.

Re: Creating Custom Receiver for Spark Streaming

Posted by Shixiong Zhu <zs...@gmail.com>.

Each ReceiverInputDStream will create one Receiver. If you only use
one ReceiverInputDStream, there will be only one Receiver in the cluster.
But if you create multiple ReceiverInputDStreams, there will be multiple
Receivers.

Best Regards,
Shixiong Zhu

2015-10-12 23:47 GMT+08:00 Something Something <ma...@gmail.com>:

> Is it safe to assume that Spark will always create a single instance of
> Custom Receiver? Or would it create multiple instances on each node in a
> cluster? Wondering if I need to worry about receiving the same message on
> different nodes etc.
>
> Please help. Thanks.
>