You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jacob Abraham <ab...@gmail.com> on 2014/11/14 02:24:50 UTC
Spark Custom Receiver
Hi Folks,
I have written a custom Spark receiver and in my testing I have found that
its doing its job properly.
However, I am wondering if someone could shed some light on how the
"driver" could query the "receiver" for some information. In other words,
how can I make the driver talk to the receivers to get some information
from it ? This information could be something like the state of the
receiver or some statistics about the receiver itself...
Regards,
-Abe
Re: Spark Custom Receiver
Posted by Tathagata Das <ta...@gmail.com>.
That is not yet in the roadmap. If there is sufficient demand for it,
we can think about it. But exposing internal communication details in
a API stable manner is usually a little tricky.
TD
On Mon, Nov 17, 2014 at 5:50 PM, Jacob Abraham <ab...@gmail.com> wrote:
> Thanks TD for your reply. Hacking is fine, but I am not a fan of hacking
> into something that I don't fully understand (law of unintended
> consequences). So, this option is out for me... :) I could setup my own
> actor (I suppose you mean akka actors) and here the problem is that I don't
> understand the actor framework very well. However this is some I could gain
> some knowledge in. No better time to start....
>
> Anyway, is there any plans in the roadmnap for spark streaming where the
> existing actor based messaging will be exposed to the public API?
>
> -abe
>
>
> On Thu, Nov 13, 2014 at 5:30 PM, Tathagata Das <ta...@gmail.com>
> wrote:
>>
>> Haha, there is a actor-based messaging path that exists between the
>> driver (see ReceiverTracker) and the Receiver (see
>> ReceiverSupervisorImpl). But none of that is exposed to the public
>> API. So either you could hack in Spark Streaming code to expose that,
>> or it might be better to setup you own actor in the custom receiver,
>> and communicate with another actor in the application driver.
>>
>> Does that make sense?
>>
>> On Thu, Nov 13, 2014 at 5:24 PM, Jacob Abraham <ab...@gmail.com>
>> wrote:
>> > Hi Folks,
>> >
>> > I have written a custom Spark receiver and in my testing I have found
>> > that
>> > its doing its job properly.
>> >
>> > However, I am wondering if someone could shed some light on how the
>> > "driver"
>> > could query the "receiver" for some information. In other words, how can
>> > I
>> > make the driver talk to the receivers to get some information from it ?
>> > This
>> > information could be something like the state of the receiver or some
>> > statistics about the receiver itself...
>> >
>> > Regards,
>> > -Abe
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Custom Receiver
Posted by Jacob Abraham <ab...@gmail.com>.
Thanks TD for your reply. Hacking is fine, but I am not a fan of hacking
into something that I don't fully understand (law of unintended
consequences). So, this option is out for me... :) I could setup my own
actor (I suppose you mean akka actors) and here the problem is that I don't
understand the actor framework very well. However this is some I could gain
some knowledge in. No better time to start....
Anyway, is there any plans in the roadmnap for spark streaming where the
existing actor based messaging will be exposed to the public API?
-abe
On Thu, Nov 13, 2014 at 5:30 PM, Tathagata Das <ta...@gmail.com>
wrote:
> Haha, there is a actor-based messaging path that exists between the
> driver (see ReceiverTracker) and the Receiver (see
> ReceiverSupervisorImpl). But none of that is exposed to the public
> API. So either you could hack in Spark Streaming code to expose that,
> or it might be better to setup you own actor in the custom receiver,
> and communicate with another actor in the application driver.
>
> Does that make sense?
>
> On Thu, Nov 13, 2014 at 5:24 PM, Jacob Abraham <ab...@gmail.com>
> wrote:
> > Hi Folks,
> >
> > I have written a custom Spark receiver and in my testing I have found
> that
> > its doing its job properly.
> >
> > However, I am wondering if someone could shed some light on how the
> "driver"
> > could query the "receiver" for some information. In other words, how can
> I
> > make the driver talk to the receivers to get some information from it ?
> This
> > information could be something like the state of the receiver or some
> > statistics about the receiver itself...
> >
> > Regards,
> > -Abe
>
Re: Spark Custom Receiver
Posted by Tathagata Das <ta...@gmail.com>.
Haha, there is a actor-based messaging path that exists between the
driver (see ReceiverTracker) and the Receiver (see
ReceiverSupervisorImpl). But none of that is exposed to the public
API. So either you could hack in Spark Streaming code to expose that,
or it might be better to setup you own actor in the custom receiver,
and communicate with another actor in the application driver.
Does that make sense?
On Thu, Nov 13, 2014 at 5:24 PM, Jacob Abraham <ab...@gmail.com> wrote:
> Hi Folks,
>
> I have written a custom Spark receiver and in my testing I have found that
> its doing its job properly.
>
> However, I am wondering if someone could shed some light on how the "driver"
> could query the "receiver" for some information. In other words, how can I
> make the driver talk to the receivers to get some information from it ? This
> information could be something like the state of the receiver or some
> statistics about the receiver itself...
>
> Regards,
> -Abe
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org