You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <sr...@gmail.com> on 2019/01/21 17:55:11 UTC
Make proactive check for closure serializability optional?
The ClosureCleaner proactively checks that closures passed to
transformations like RDD.map() are serializable, before they're
executed. It does this by just serializing it with the JavaSerializer.
That's a nice feature, although there's overhead in always trying to
serialize the closure ahead of time, especially if the closure is
large. It shouldn't be large, usually. But I noticed it when coming up
with this fix: https://github.com/apache/spark/pull/23600
It made me wonder, should this be optional, or even not the default?
Closures that don't serialize still fail, just later when an action is
invoked. I don't feel strongly about it, just checking if anyone had
pondered this before.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Make proactive check for closure serializability optional?
Posted by Sean Owen <sr...@gmail.com>.
Agree, I'm not pushing for it unless there's other evidence. The closure
check does entail serialization, not just checking serializability, note.
I don't like flags either but this one sounded like it could actually be
something a user wanted to vary, globally, for runs of the same code.
On Tue, Jan 22, 2019 at 11:25 AM Reynold Xin <rx...@databricks.com> wrote:
> Typically very large closures include some array, and the serialization
> itself should be much more expensive than the closure check. Does anybody
> have actual data on this could be a problem? We don't want to add a config
> flag if for virtually any case it doesn't make sense to change.
>
>
Re: Make proactive check for closure serializability optional?
Posted by Reynold Xin <rx...@databricks.com>.
Typically very large closures include some array, and the serialization itself should be much more expensive than the closure check. Does anybody have actual data on this could be a problem? We don't want to add a config flag if for virtually any case it doesn't make sense to change.
On Mon, Jan 21, 2019 at 12:37 PM, Felix Cheung < felixcheung_m@hotmail.com > wrote:
>
> Agreed on the pros / cons, esp driver could be the data science notebook.
> Is it worthwhile making it configurable?
>
>
>
>
> *From:* Sean Owen < srowen@ gmail. com ( srowen@gmail.com ) >
> *Sent:* Monday, January 21, 2019 10:42 AM
> *To:* Reynold Xin
> *Cc:* dev
> *Subject:* Re: Make proactive check for closure serializability optional?
>
> None except the bug / PR I linked to, which is really just a bug in
> the RowMatrix implementation; a 2GB closure isn't reasonable.
> I doubt it's much overhead in the common case, because closures are
> small and this extra check happens once per execution of the closure.
>
> I can also imagine middle-ground cases where people are dragging along
> largeish 10MB closures (like, a model or some data) and this could add
> non-trivial memory pressure on the driver. They should be broadcasting
> those things, sure.
>
> Given just that I'd leave it alone, but was wondering if anyone had
> ever had the same thought or more arguments that it should be
> disable-able. In 'production' one would imagine all the closures do
> serialize correctly and so this is just a bit overhead that could be
> skipped.
>
> On Mon, Jan 21, 2019 at 12:17 PM Reynold Xin < rxin@ databricks. com (
> rxin@databricks.com ) > wrote:
> >
> > Did you actually observe a perf issue?
> >
> > On Mon, Jan 21, 2019 at 10:04 AM Sean Owen < srowen@ gmail. com (
> srowen@gmail.com ) > wrote:
> >>
> >> The ClosureCleaner proactively checks that closures passed to
> >> transformations like RDD.map() are serializable, before they're
> >> executed. It does this by just serializing it with the JavaSerializer.
> >>
> >> That's a nice feature, although there's overhead in always trying to
> >> serialize the closure ahead of time, especially if the closure is
> >> large. It shouldn't be large, usually. But I noticed it when coming up
> >> with this fix: https:/ / github. com/ apache/ spark/ pull/ 23600 (
> https://github.com/apache/spark/pull/23600 )
> >>
> >> It made me wonder, should this be optional, or even not the default?
> >> Closures that don't serialize still fail, just later when an action is
> >> invoked. I don't feel strongly about it, just checking if anyone had
> >> pondered this before.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@ spark. apache. org (
> dev-unsubscribe@spark.apache.org )
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@ spark. apache. org (
> dev-unsubscribe@spark.apache.org )
>
Re: Make proactive check for closure serializability optional?
Posted by Felix Cheung <fe...@hotmail.com>.
Agreed on the pros / cons, esp driver could be the data science notebook.
Is it worthwhile making it configurable?
________________________________
From: Sean Owen <sr...@gmail.com>
Sent: Monday, January 21, 2019 10:42 AM
To: Reynold Xin
Cc: dev
Subject: Re: Make proactive check for closure serializability optional?
None except the bug / PR I linked to, which is really just a bug in
the RowMatrix implementation; a 2GB closure isn't reasonable.
I doubt it's much overhead in the common case, because closures are
small and this extra check happens once per execution of the closure.
I can also imagine middle-ground cases where people are dragging along
largeish 10MB closures (like, a model or some data) and this could add
non-trivial memory pressure on the driver. They should be broadcasting
those things, sure.
Given just that I'd leave it alone, but was wondering if anyone had
ever had the same thought or more arguments that it should be
disable-able. In 'production' one would imagine all the closures do
serialize correctly and so this is just a bit overhead that could be
skipped.
On Mon, Jan 21, 2019 at 12:17 PM Reynold Xin <rx...@databricks.com> wrote:
>
> Did you actually observe a perf issue?
>
> On Mon, Jan 21, 2019 at 10:04 AM Sean Owen <sr...@gmail.com> wrote:
>>
>> The ClosureCleaner proactively checks that closures passed to
>> transformations like RDD.map() are serializable, before they're
>> executed. It does this by just serializing it with the JavaSerializer.
>>
>> That's a nice feature, although there's overhead in always trying to
>> serialize the closure ahead of time, especially if the closure is
>> large. It shouldn't be large, usually. But I noticed it when coming up
>> with this fix: https://github.com/apache/spark/pull/23600
>>
>> It made me wonder, should this be optional, or even not the default?
>> Closures that don't serialize still fail, just later when an action is
>> invoked. I don't feel strongly about it, just checking if anyone had
>> pondered this before.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Make proactive check for closure serializability optional?
Posted by Sean Owen <sr...@gmail.com>.
None except the bug / PR I linked to, which is really just a bug in
the RowMatrix implementation; a 2GB closure isn't reasonable.
I doubt it's much overhead in the common case, because closures are
small and this extra check happens once per execution of the closure.
I can also imagine middle-ground cases where people are dragging along
largeish 10MB closures (like, a model or some data) and this could add
non-trivial memory pressure on the driver. They should be broadcasting
those things, sure.
Given just that I'd leave it alone, but was wondering if anyone had
ever had the same thought or more arguments that it should be
disable-able. In 'production' one would imagine all the closures do
serialize correctly and so this is just a bit overhead that could be
skipped.
On Mon, Jan 21, 2019 at 12:17 PM Reynold Xin <rx...@databricks.com> wrote:
>
> Did you actually observe a perf issue?
>
> On Mon, Jan 21, 2019 at 10:04 AM Sean Owen <sr...@gmail.com> wrote:
>>
>> The ClosureCleaner proactively checks that closures passed to
>> transformations like RDD.map() are serializable, before they're
>> executed. It does this by just serializing it with the JavaSerializer.
>>
>> That's a nice feature, although there's overhead in always trying to
>> serialize the closure ahead of time, especially if the closure is
>> large. It shouldn't be large, usually. But I noticed it when coming up
>> with this fix: https://github.com/apache/spark/pull/23600
>>
>> It made me wonder, should this be optional, or even not the default?
>> Closures that don't serialize still fail, just later when an action is
>> invoked. I don't feel strongly about it, just checking if anyone had
>> pondered this before.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Make proactive check for closure serializability optional?
Posted by Reynold Xin <rx...@databricks.com>.
Did you actually observe a perf issue?
On Mon, Jan 21, 2019 at 10:04 AM Sean Owen <sr...@gmail.com> wrote:
> The ClosureCleaner proactively checks that closures passed to
> transformations like RDD.map() are serializable, before they're
> executed. It does this by just serializing it with the JavaSerializer.
>
> That's a nice feature, although there's overhead in always trying to
> serialize the closure ahead of time, especially if the closure is
> large. It shouldn't be large, usually. But I noticed it when coming up
> with this fix: https://github.com/apache/spark/pull/23600
>
> It made me wonder, should this be optional, or even not the default?
> Closures that don't serialize still fail, just later when an action is
> invoked. I don't feel strongly about it, just checking if anyone had
> pondered this before.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>