You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Imran Rashid <ir...@cloudera.com.INVALID> on 2018/08/03 16:59:46 UTC
SPIP: Executor Plugin (SPARK-24918)
I'd like to propose adding a plugin api for Executors, primarily for
instrumentation and debugging (
https://issues.apache.org/jira/browse/SPARK-24918). The changes are small,
but as its adding a new api, it might be spip-worthy. I mentioned it as
well in a recent email I sent about memory monitoring
The spip proposal is here (and attached to the jira as well):
https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
There are already some comments on the jira and pr, and I hope to get more
thoughts and opinions on it.
thanks,
Imran
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Reynold Xin <rx...@databricks.com>.
Both ahead of time, or just in time. Just like a normal Spark closure.
On Fri, Aug 31, 2018 at 10:18 AM Nihar Sheth <ns...@cloudera.com> wrote:
> Hi @rxin,
>
> Just to make sure I understand your desired use case, are you suggesting a
> way (for the driver) to call, at any given time, a general method that can
> be defined ahead of time on the executors?
>
> On Thu, Aug 30, 2018 at 11:11 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> I actually had a similar use case a while ago, but not entirely the same.
>> In my use case, Spark is already up, but I want to make sure all existing
>> (and new) executors run some specific code. Can we update the API to
>> support that? I think that's doable if we split the design into two: one is
>> the ability to do what I just mentioned, and second is the ability to
>> register via config class when Spark starts to run the code.
>>
>>
>> On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>>> +1
>>> ------------------------------
>>> *From:* Mridul Muralidharan <mr...@gmail.com>
>>> *Sent:* Wednesday, August 29, 2018 1:27:27 PM
>>> *To:* dev@spark.apache.org
>>> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918)
>>>
>>> +1
>>> I left a couple of comments in NiharS's PR, but this is very useful to
>>> have in spark !
>>>
>>> Regards,
>>> Mridul
>>> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
>>> <ir...@cloudera.com.invalid> wrote:
>>> >
>>> > I'd like to propose adding a plugin api for Executors, primarily for
>>> instrumentation and debugging (
>>> https://issues.apache.org/jira/browse/SPARK-24918). The changes are
>>> small, but as its adding a new api, it might be spip-worthy. I mentioned
>>> it as well in a recent email I sent about memory monitoring
>>> >
>>> > The spip proposal is here (and attached to the jira as well):
>>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>>> >
>>> > There are already some comments on the jira and pr, and I hope to get
>>> more thoughts and opinions on it.
>>> >
>>> > thanks,
>>> > Imran
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Nihar Sheth <ns...@cloudera.com.INVALID>.
Hi @rxin,
Just to make sure I understand your desired use case, are you suggesting a
way (for the driver) to call, at any given time, a general method that can
be defined ahead of time on the executors?
On Thu, Aug 30, 2018 at 11:11 PM, Reynold Xin <rx...@databricks.com> wrote:
> I actually had a similar use case a while ago, but not entirely the same.
> In my use case, Spark is already up, but I want to make sure all existing
> (and new) executors run some specific code. Can we update the API to
> support that? I think that's doable if we split the design into two: one is
> the ability to do what I just mentioned, and second is the ability to
> register via config class when Spark starts to run the code.
>
>
> On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> +1
>> ------------------------------
>> *From:* Mridul Muralidharan <mr...@gmail.com>
>> *Sent:* Wednesday, August 29, 2018 1:27:27 PM
>> *To:* dev@spark.apache.org
>> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918)
>>
>> +1
>> I left a couple of comments in NiharS's PR, but this is very useful to
>> have in spark !
>>
>> Regards,
>> Mridul
>> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
>> <ir...@cloudera.com.invalid> wrote:
>> >
>> > I'd like to propose adding a plugin api for Executors, primarily for
>> instrumentation and debugging (https://issues.apache.org/
>> jira/browse/SPARK-24918). The changes are small, but as its adding a
>> new api, it might be spip-worthy. I mentioned it as well in a recent email
>> I sent about memory monitoring
>> >
>> > The spip proposal is here (and attached to the jira as well):
>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5c
>> bHBQtyqIA2hgtc/edit?usp=sharing
>> >
>> > There are already some comments on the jira and pr, and I hope to get
>> more thoughts and opinions on it.
>> >
>> > thanks,
>> > Imran
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Lars Francke <la...@gmail.com>.
+1
On Fri, Aug 31, 2018 at 8:11 AM, Reynold Xin <rx...@databricks.com> wrote:
> I actually had a similar use case a while ago, but not entirely the same.
> In my use case, Spark is already up, but I want to make sure all existing
> (and new) executors run some specific code. Can we update the API to
> support that? I think that's doable if we split the design into two: one is
> the ability to do what I just mentioned, and second is the ability to
> register via config class when Spark starts to run the code.
>
>
> On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> +1
>> ------------------------------
>> *From:* Mridul Muralidharan <mr...@gmail.com>
>> *Sent:* Wednesday, August 29, 2018 1:27:27 PM
>> *To:* dev@spark.apache.org
>> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918)
>>
>> +1
>> I left a couple of comments in NiharS's PR, but this is very useful to
>> have in spark !
>>
>> Regards,
>> Mridul
>> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
>> <ir...@cloudera.com.invalid> wrote:
>> >
>> > I'd like to propose adding a plugin api for Executors, primarily for
>> instrumentation and debugging (https://issues.apache.org/
>> jira/browse/SPARK-24918). The changes are small, but as its adding a
>> new api, it might be spip-worthy. I mentioned it as well in a recent email
>> I sent about memory monitoring
>> >
>> > The spip proposal is here (and attached to the jira as well):
>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5c
>> bHBQtyqIA2hgtc/edit?usp=sharing
>> >
>> > There are already some comments on the jira and pr, and I hope to get
>> more thoughts and opinions on it.
>> >
>> > thanks,
>> > Imran
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Ted Yu <yu...@gmail.com>.
+1
-------- Original message --------From: Reynold Xin <rx...@databricks.com> Date: 8/30/18 11:11 PM (GMT-08:00) To: Felix Cheung <fe...@hotmail.com> Cc: dev <de...@spark.apache.org> Subject: Re: SPIP: Executor Plugin (SPARK-24918)
I actually had a similar use case a while ago, but not entirely the same. In my use case, Spark is already up, but I want to make sure all existing (and new) executors run some specific code. Can we update the API to support that? I think that's doable if we split the design into two: one is the ability to do what I just mentioned, and second is the ability to register via config class when Spark starts to run the code.
On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <fe...@hotmail.com> wrote:
+1
From: Mridul Muralidharan <mr...@gmail.com>
Sent: Wednesday, August 29, 2018 1:27:27 PM
To: dev@spark.apache.org
Subject: Re: SPIP: Executor Plugin (SPARK-24918)
+1
I left a couple of comments in NiharS's PR, but this is very useful to
have in spark !
Regards,
Mridul
On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
<ir...@cloudera.com.invalid> wrote:
>
> I'd like to propose adding a plugin api for Executors, primarily for instrumentation and debugging (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, but as its adding
a new api, it might be spip-worthy. I mentioned it as well in a recent email I sent about memory monitoring
>
> The spip proposal is here (and attached to the jira as well):
https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>
> There are already some comments on the jira and pr, and I hope to get more thoughts and opinions on it.
>
> thanks,
> Imran
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Reynold Xin <rx...@databricks.com>.
I actually had a similar use case a while ago, but not entirely the same.
In my use case, Spark is already up, but I want to make sure all existing
(and new) executors run some specific code. Can we update the API to
support that? I think that's doable if we split the design into two: one is
the ability to do what I just mentioned, and second is the ability to
register via config class when Spark starts to run the code.
On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <fe...@hotmail.com>
wrote:
> +1
> ------------------------------
> *From:* Mridul Muralidharan <mr...@gmail.com>
> *Sent:* Wednesday, August 29, 2018 1:27:27 PM
> *To:* dev@spark.apache.org
> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918)
>
> +1
> I left a couple of comments in NiharS's PR, but this is very useful to
> have in spark !
>
> Regards,
> Mridul
> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
> <ir...@cloudera.com.invalid> wrote:
> >
> > I'd like to propose adding a plugin api for Executors, primarily for
> instrumentation and debugging (
> https://issues.apache.org/jira/browse/SPARK-24918). The changes are
> small, but as its adding a new api, it might be spip-worthy. I mentioned
> it as well in a recent email I sent about memory monitoring
> >
> > The spip proposal is here (and attached to the jira as well):
> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
> >
> > There are already some comments on the jira and pr, and I hope to get
> more thoughts and opinions on it.
> >
> > thanks,
> > Imran
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Felix Cheung <fe...@hotmail.com>.
+1
________________________________
From: Mridul Muralidharan <mr...@gmail.com>
Sent: Wednesday, August 29, 2018 1:27:27 PM
To: dev@spark.apache.org
Subject: Re: SPIP: Executor Plugin (SPARK-24918)
+1
I left a couple of comments in NiharS's PR, but this is very useful to
have in spark !
Regards,
Mridul
On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
<ir...@cloudera.com.invalid> wrote:
>
> I'd like to propose adding a plugin api for Executors, primarily for instrumentation and debugging (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, but as its adding a new api, it might be spip-worthy. I mentioned it as well in a recent email I sent about memory monitoring
>
> The spip proposal is here (and attached to the jira as well): https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>
> There are already some comments on the jira and pr, and I hope to get more thoughts and opinions on it.
>
> thanks,
> Imran
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: SPIP: Executor Plugin (SPARK-24918)
Posted by Mridul Muralidharan <mr...@gmail.com>.
+1
I left a couple of comments in NiharS's PR, but this is very useful to
have in spark !
Regards,
Mridul
On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
<ir...@cloudera.com.invalid> wrote:
>
> I'd like to propose adding a plugin api for Executors, primarily for instrumentation and debugging (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, but as its adding a new api, it might be spip-worthy. I mentioned it as well in a recent email I sent about memory monitoring
>
> The spip proposal is here (and attached to the jira as well): https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>
> There are already some comments on the jira and pr, and I hope to get more thoughts and opinions on it.
>
> thanks,
> Imran
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Thomas Graves <tg...@gmail.com>.
+1
Tom
On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid <ir...@cloudera.com.invalid>
wrote:
> There has been discussion on jira & the PR, all generally positive, so I'd
> like to call a vote for this spip.
>
> I'll start with own +1.
>
> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com> wrote:
>
>> I'd like to propose adding a plugin api for Executors, primarily for
>> instrumentation and debugging (
>> https://issues.apache.org/jira/browse/SPARK-24918). The changes are
>> small, but as its adding a new api, it might be spip-worthy. I mentioned
>> it as well in a recent email I sent about memory monitoring
>>
>> The spip proposal is here (and attached to the jira as well):
>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>>
>> There are already some comments on the jira and pr, and I hope to get
>> more thoughts and opinions on it.
>>
>> thanks,
>> Imran
>>
>
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Sean Owen <sr...@apache.org>.
I get it, the use case here sounds like adding some logic that's not
specific to an app. Adding to every job of every app would be painful then.
Any task can make sure the init has happened, so doesn't matter if later
tasks execute elsewhere. But if the desired init is really for all tasks,
or all apps, that instrumentation would be painful to add even if it works.
That sounds like a valid use case.
The original use cases on the very first JIRA were about things like
initializing some HBase connections for a particular call, or prefetching
some data for logging. This didn't seem so global or non-app-specific in
comparison.
On Tue, Aug 28, 2018 at 11:45 AM Marcelo Vanzin <va...@cloudera.com> wrote:
> The issue is: where do you name the class you want to initialize? In your
> job?
>
> What if the plugin has nothing to do with your job, e.g. it's some
> monitoring tool? Are you going to modify all your jobs so you include
> initialization of that particular plugin? Forcing everybody to change
> their code, builds, re-deploy stuff, etc?
>
> What if a task in an early stage of your job, the one that initializes
> the plugin, runs in executor A, and a task in a further stage runs on
> executor B?
>
> It's just way more complicated, if possible at all, to write this
> feature by asking everybody to change their application code.
>
>
> On Tue, Aug 28, 2018 at 9:39 AM, Sean Owen <sr...@apache.org> wrote:
> > I should be able to force a class to init by just naming it. The issue is
> > that this means doing that in any job that needs init? not that it
> doesn't
> > work, right? I concede that could be annoying if just about all your code
> > needs this init; I had understood the use cases to be more like
> "establish
> > some local config and init for this particular thing I'm doing for this
> > legacy system".
> >
> > On Tue, Aug 28, 2018 at 11:35 AM Marcelo Vanzin <va...@cloudera.com>
> wrote:
> >>
> >> +1
> >>
> >> Class init is not enough because there is nowhere for you to force a
> >> random class to be initialized. This is basically adding that
> >> mechanism, instead of forcing people to add hacks using e.g.
> >> mapPartitions which don't even cover all scenarios.
> >>
> >> On Tue, Aug 28, 2018 at 7:09 AM, Sean Owen <sr...@apache.org> wrote:
> >> > Still +0 on the idea, as I am still not sure it does much over simple
> >> > JVM
> >> > mechanisms like a class init. More comments on the JIRA. I can't say
> >> > it's a
> >> > bad idea though, so would not object to it.
> >> >
> >> > On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid
> >> > <ir...@cloudera.com.invalid>
> >> > wrote:
> >> >>
> >> >> There has been discussion on jira & the PR, all generally positive,
> so
> >> >> I'd
> >> >> like to call a vote for this spip.
> >> >>
> >> >> I'll start with own +1.
> >> >>
> >> >> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com>
> >> >> wrote:
> >> >>>
> >> >>> I'd like to propose adding a plugin api for Executors, primarily for
> >> >>> instrumentation and debugging
> >> >>> (https://issues.apache.org/jira/browse/SPARK-24918). The changes
> are
> >> >>> small,
> >> >>> but as its adding a new api, it might be spip-worthy. I mentioned
> it
> >> >>> as
> >> >>> well in a recent email I sent about memory monitoring
> >> >>>
> >> >>> The spip proposal is here (and attached to the jira as well):
> >> >>>
> >> >>>
> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
> >> >>>
> >> >>> There are already some comments on the jira and pr, and I hope to
> get
> >> >>> more thoughts and opinions on it.
> >> >>>
> >> >>> thanks,
> >> >>> Imran
> >>
> >>
> >>
> >> --
> >> Marcelo
>
>
>
> --
> Marcelo
>
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Marcelo Vanzin <va...@cloudera.com.INVALID>.
The issue is: where do you name the class you want to initialize? In your job?
What if the plugin has nothing to do with your job, e.g. it's some
monitoring tool? Are you going to modify all your jobs so you include
initialization of that particular plugin? Forcing everybody to change
their code, builds, re-deploy stuff, etc?
What if a task in an early stage of your job, the one that initializes
the plugin, runs in executor A, and a task in a further stage runs on
executor B?
It's just way more complicated, if possible at all, to write this
feature by asking everybody to change their application code.
On Tue, Aug 28, 2018 at 9:39 AM, Sean Owen <sr...@apache.org> wrote:
> I should be able to force a class to init by just naming it. The issue is
> that this means doing that in any job that needs init? not that it doesn't
> work, right? I concede that could be annoying if just about all your code
> needs this init; I had understood the use cases to be more like "establish
> some local config and init for this particular thing I'm doing for this
> legacy system".
>
> On Tue, Aug 28, 2018 at 11:35 AM Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> +1
>>
>> Class init is not enough because there is nowhere for you to force a
>> random class to be initialized. This is basically adding that
>> mechanism, instead of forcing people to add hacks using e.g.
>> mapPartitions which don't even cover all scenarios.
>>
>> On Tue, Aug 28, 2018 at 7:09 AM, Sean Owen <sr...@apache.org> wrote:
>> > Still +0 on the idea, as I am still not sure it does much over simple
>> > JVM
>> > mechanisms like a class init. More comments on the JIRA. I can't say
>> > it's a
>> > bad idea though, so would not object to it.
>> >
>> > On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid
>> > <ir...@cloudera.com.invalid>
>> > wrote:
>> >>
>> >> There has been discussion on jira & the PR, all generally positive, so
>> >> I'd
>> >> like to call a vote for this spip.
>> >>
>> >> I'll start with own +1.
>> >>
>> >> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com>
>> >> wrote:
>> >>>
>> >>> I'd like to propose adding a plugin api for Executors, primarily for
>> >>> instrumentation and debugging
>> >>> (https://issues.apache.org/jira/browse/SPARK-24918). The changes are
>> >>> small,
>> >>> but as its adding a new api, it might be spip-worthy. I mentioned it
>> >>> as
>> >>> well in a recent email I sent about memory monitoring
>> >>>
>> >>> The spip proposal is here (and attached to the jira as well):
>> >>>
>> >>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>> >>>
>> >>> There are already some comments on the jira and pr, and I hope to get
>> >>> more thoughts and opinions on it.
>> >>>
>> >>> thanks,
>> >>> Imran
>>
>>
>>
>> --
>> Marcelo
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Sean Owen <sr...@apache.org>.
I should be able to force a class to init by just naming it. The issue is
that this means doing that in any job that needs init? not that it doesn't
work, right? I concede that could be annoying if just about all your code
needs this init; I had understood the use cases to be more like "establish
some local config and init for this particular thing I'm doing for this
legacy system".
On Tue, Aug 28, 2018 at 11:35 AM Marcelo Vanzin <va...@cloudera.com> wrote:
> +1
>
> Class init is not enough because there is nowhere for you to force a
> random class to be initialized. This is basically adding that
> mechanism, instead of forcing people to add hacks using e.g.
> mapPartitions which don't even cover all scenarios.
>
> On Tue, Aug 28, 2018 at 7:09 AM, Sean Owen <sr...@apache.org> wrote:
> > Still +0 on the idea, as I am still not sure it does much over simple JVM
> > mechanisms like a class init. More comments on the JIRA. I can't say
> it's a
> > bad idea though, so would not object to it.
> >
> > On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid
> <ir...@cloudera.com.invalid>
> > wrote:
> >>
> >> There has been discussion on jira & the PR, all generally positive, so
> I'd
> >> like to call a vote for this spip.
> >>
> >> I'll start with own +1.
> >>
> >> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com>
> wrote:
> >>>
> >>> I'd like to propose adding a plugin api for Executors, primarily for
> >>> instrumentation and debugging
> >>> (https://issues.apache.org/jira/browse/SPARK-24918). The changes are
> small,
> >>> but as its adding a new api, it might be spip-worthy. I mentioned it
> as
> >>> well in a recent email I sent about memory monitoring
> >>>
> >>> The spip proposal is here (and attached to the jira as well):
> >>>
> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
> >>>
> >>> There are already some comments on the jira and pr, and I hope to get
> >>> more thoughts and opinions on it.
> >>>
> >>> thanks,
> >>> Imran
>
>
>
> --
> Marcelo
>
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Marcelo Vanzin <va...@cloudera.com.INVALID>.
+1
Class init is not enough because there is nowhere for you to force a
random class to be initialized. This is basically adding that
mechanism, instead of forcing people to add hacks using e.g.
mapPartitions which don't even cover all scenarios.
On Tue, Aug 28, 2018 at 7:09 AM, Sean Owen <sr...@apache.org> wrote:
> Still +0 on the idea, as I am still not sure it does much over simple JVM
> mechanisms like a class init. More comments on the JIRA. I can't say it's a
> bad idea though, so would not object to it.
>
> On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid <ir...@cloudera.com.invalid>
> wrote:
>>
>> There has been discussion on jira & the PR, all generally positive, so I'd
>> like to call a vote for this spip.
>>
>> I'll start with own +1.
>>
>> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com> wrote:
>>>
>>> I'd like to propose adding a plugin api for Executors, primarily for
>>> instrumentation and debugging
>>> (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small,
>>> but as its adding a new api, it might be spip-worthy. I mentioned it as
>>> well in a recent email I sent about memory monitoring
>>>
>>> The spip proposal is here (and attached to the jira as well):
>>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>>>
>>> There are already some comments on the jira and pr, and I hope to get
>>> more thoughts and opinions on it.
>>>
>>> thanks,
>>> Imran
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Sean Owen <sr...@apache.org>.
Still +0 on the idea, as I am still not sure it does much over simple JVM
mechanisms like a class init. More comments on the JIRA. I can't say it's a
bad idea though, so would not object to it.
On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid <ir...@cloudera.com.invalid>
wrote:
> There has been discussion on jira & the PR, all generally positive, so I'd
> like to call a vote for this spip.
>
> I'll start with own +1.
>
> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com> wrote:
>
>> I'd like to propose adding a plugin api for Executors, primarily for
>> instrumentation and debugging (
>> https://issues.apache.org/jira/browse/SPARK-24918). The changes are
>> small, but as its adding a new api, it might be spip-worthy. I mentioned
>> it as well in a recent email I sent about memory monitoring
>>
>> The spip proposal is here (and attached to the jira as well):
>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>>
>> There are already some comments on the jira and pr, and I hope to get
>> more thoughts and opinions on it.
>>
>> thanks,
>> Imran
>>
>
[VOTE] SPIP: Executor Plugin (SPARK-24918)
Posted by Imran Rashid <ir...@cloudera.com.INVALID>.
There has been discussion on jira & the PR, all generally positive, so I'd
like to call a vote for this spip.
I'll start with own +1.
On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <ir...@cloudera.com> wrote:
> I'd like to propose adding a plugin api for Executors, primarily for
> instrumentation and debugging (
> https://issues.apache.org/jira/browse/SPARK-24918). The changes are
> small, but as its adding a new api, it might be spip-worthy. I mentioned
> it as well in a recent email I sent about memory monitoring
>
> The spip proposal is here (and attached to the jira as well):
> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
>
> There are already some comments on the jira and pr, and I hope to get more
> thoughts and opinions on it.
>
> thanks,
> Imran
>