You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "bit1129@163.com" <bi...@163.com> on 2015/05/25 04:15:40 UTC

回复: How to use zookeeper in Spark Streaming

Can someone please help me on this? 



bit1129@163.com
 
发件人： bit1129@163.com
发送时间： 2015-05-24 13:53
收件人： user
主题： How to use zookeeper in Spark Streaming

Hi，
In my spark streaming application, when the application starts and get running, the Tasks running on the Worker nodes need to be notified that some configurations have been changed from time to time, these configurations reside on the Zookeeper.

My question is, where should I put the code that works with Zookeeper for the configuration change, in the Driver code or in the Task code? Is there some guide on this? Thanks.







bit1129@163.com

Re: How to use zookeeper in Spark Streaming

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
If you want to notify after every batch is completed, then you can simply
implement the StreamingListener
<https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.streaming.scheduler.StreamingListener>
interface, which has methods like onBatchCompleted, onBatchStarted etc in
which you can push the configuration changes to zookeeper. This piece of
code runs on the driver of course. And in the task code, you can pull these
configuration changes from zookeeper.

Thanks
Best Regards

On Mon, May 25, 2015 at 7:45 AM, bit1129@163.com <bi...@163.com> wrote:

> Can someone please help me on this?
>
> ------------------------------
> bit1129@163.com
>
>
> *发件人：* bit1129@163.com
> *发送时间：* 2015-05-24 13:53
> *收件人：* user <us...@spark.apache.org>
> *主题：* How to use zookeeper in Spark Streaming
>
> Hi，
> In my spark streaming application, when the application starts and get
> running, the Tasks running on the Worker nodes need to be notified that
> some configurations have been changed from time to time, these
> configurations reside on the Zookeeper.
>
> My question is, where should I put the code that works with Zookeeper for
> the configuration change, in the Driver code or in the Task code? Is there
> some guide on this? Thanks.
>
>
>
>
>
> ------------------------------
> bit1129@163.com
>
>

Re: How to use zookeeper in Spark Streaming

Posted by Ted Yu <yu...@gmail.com>.
I think the Zookeeper watcher code should reside in task code.

Haven't found guide on this subject so far.

Cheers

On Sun, May 24, 2015 at 7:15 PM, bit1129@163.com <bi...@163.com> wrote:

> Can someone please help me on this?
>
> ------------------------------
> bit1129@163.com
>
>
> *发件人：* bit1129@163.com
> *发送时间：* 2015-05-24 13:53
> *收件人：* user <us...@spark.apache.org>
> *主题：* How to use zookeeper in Spark Streaming
>
> Hi，
> In my spark streaming application, when the application starts and get
> running, the Tasks running on the Worker nodes need to be notified that
> some configurations have been changed from time to time, these
> configurations reside on the Zookeeper.
>
> My question is, where should I put the code that works with Zookeeper for
> the configuration change, in the Driver code or in the Task code? Is there
> some guide on this? Thanks.
>
>
>
>
>
> ------------------------------
> bit1129@163.com
>
>