You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Rabin Banerjee <de...@gmail.com> on 2017/06/13 11:54:25 UTC

Assign Custom receiver to a scheduler pool

HI All,

  I have a spark streaming job, which is running fine. And in that job I am
using FAIR scheduler with multiple pools. One pool for main streaming job.
And another for some parallel batch processing . All is working as
expected, except ..

 "The custom receiver that I used is not assigned to the pool used for
streaming."

I am using stream.foreachRDD.. { rdd =>

rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming")

}

What I can understand is , I am setting the Thread local after Spark Custom
Receiver is initialized.

Receiver is running on default pool.

Setting Receiver using : ssc.receiverStream(MyCustomReceiver())

Any Help ?

Regards,
Rabin Banerjee

Fwd: Assign Custom receiver to a scheduler pool

Posted by Rabin Banerjee <de...@gmail.com>.

HI All,

  I have a spark streaming job, which is running fine. And in that job I am
using FAIR scheduler with multiple pools. One pool for main streaming job.
And another for some parallel batch processing . All is working as
expected, except ..

 "The custom receiver that I used is not assigned to the pool used for
streaming."

I am using stream.foreachRDD.. { rdd =>

rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming")

}

What I can understand is , I am setting the Thread local after Spark Custom
Receiver is initialized.

Receiver is running on default pool.

Setting Receiver using : ssc.receiverStream(MyCustomReceiver())

Any Help ?

Regards,
Rabin Banerjee

Fwd: Assign Custom receiver to a scheduler pool

Posted by Rabin Banerjee <de...@gmail.com>.

HI All,

  I have a spark streaming job, which is running fine. And in that job I am
using FAIR scheduler with multiple pools. One pool for main streaming job.
And another for some parallel batch processing . All is working as
expected, except ..

 "The custom receiver that I used is not assigned to the pool used for
streaming."

I am using stream.foreachRDD.. { rdd =>

rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming")

}

What I can understand is , I am setting the Thread local after Spark Custom
Receiver is initialized.

Receiver is running on default pool.

Setting Receiver using : ssc.receiverStream(MyCustomReceiver())

Any Help ?

Regards,
Rabin Banerjee