You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by on 2019/05/14 12:01:43 UTC

Is there a way to decide what RDDs get cached in the Spark Runner?

Hi,

I guess the title says it all, right now it seems like BEAM caches all the intermediate RDD results for my pipeline when using the Spark runner, this leads to a very inefficient usage of memory. Any way to control this?

Best regards,
Augusto

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by David Morávek <da...@gmail.com>.
Hello Augusto,

This is a long standing problem that is really hard to fix properly, unless
the runner has better understanding of what is actually happening in the
pipeline itself (that's what Jan already pointed out). I still think the
best approach, that would require least "knobs" for user to tune, would be
sampling of previous pipeline runs and reusing the gathered information to
decide what should be cached and how.

I think we can probably eliminate some of the major cases that could cause
trouble, eg. when DAG split is right after shuffle, it's usually not worth
caching as Spark can reuse shuffle files, unless computation is really
expensive (random access db joins) ...

D.

On Thu, May 16, 2019 at 9:38 AM Augusto Ribeiro <au...@gmail.com>
wrote:

> Hi Robert,
>
> If Spark could decide then it would be ok, but probably not a trivial
> problem like Jan pointed out.
>
> In my case, the decision had to be made on the basis of the machines in
> the cluster, in principle if you had enough machines (or larger) to hold
> everything in memory it wouldn't be a problem but in reality we all have
> constrains on how many and what kind of machines to use.
>
> Best regards,
> Augusto
>
> > On 15 May 2019, at 10:39, Robert Bradshaw <ro...@google.com> wrote:
> >
> > Just to clarify, do you need direct control over what to cache, or
> > would it be OK to let Spark decide the minimal set of RDDs to cache as
> > long as we didn't cache all intermediates?
> >
> > From: Augusto Ribeiro <au...@gmail.com>
> > Date: Wed, May 15, 2019 at 8:37 AM
> > To: <us...@beam.apache.org>
> >
> >> Hi Kyle,
> >>
> >> Thanks for the help. It seems like I have no other choice than using
> Spark directly, since my job causes immense memory pressure if I can't
> decide what to cache.
> >>
> >> Best regards,
> >> Augusto
> >>
> >> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
> >>
> >> Minor correction: Slack channel is actually #beam-spark
> >>
> >> Kyle Weaver | Software Engineer | github.com/ibzib |
> kcweaver@google.com | +16502035555
> >>
> >>
> >> From: Kyle Weaver <kc...@google.com>
> >> Date: Tue, May 14, 2019 at 9:38 AM
> >> To: <us...@beam.apache.org>
> >>
> >>> Hi Augusto,
> >>>
> >>> Right now the default behavior is to cache all intermediate RDDs that
> are consumed more than once by the pipeline. This can be disabled with
> `options.setCacheDisabled(true)` [1], but there is currently no way for the
> user to specify to the runner that it should cache certain RDDs, but not
> others.
> >>>
> >>> There has recently been some discussion on the Slack (#spark-beam)
> about implementing such a feature, but no concrete plans as of yet.
> >>>
> >>> [1]
> https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
> >>>
> >>> Thanks
> >>>
> >>> Kyle Weaver | Software Engineer | github.com/ibzib |
> kcweaver@google.com | +16502035555
> >>>
> >>>
> >>> From: augusto.mcc@gmail.com <au...@gmail.com>
> >>> Date: Tue, May 14, 2019 at 5:01 AM
> >>> To: <us...@beam.apache.org>
> >>>
> >>>> Hi,
> >>>>
> >>>> I guess the title says it all, right now it seems like BEAM caches
> all the intermediate RDD results for my pipeline when using the Spark
> runner, this leads to a very inefficient usage of memory. Any way to
> control this?
> >>>>
> >>>> Best regards,
> >>>> Augusto
> >>
> >>
>
>

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Augusto Ribeiro <au...@gmail.com>.
Hi Robert,

If Spark could decide then it would be ok, but probably not a trivial problem like Jan pointed out. 

In my case, the decision had to be made on the basis of the machines in the cluster, in principle if you had enough machines (or larger) to hold everything in memory it wouldn't be a problem but in reality we all have constrains on how many and what kind of machines to use.

Best regards,
Augusto

> On 15 May 2019, at 10:39, Robert Bradshaw <ro...@google.com> wrote:
> 
> Just to clarify, do you need direct control over what to cache, or
> would it be OK to let Spark decide the minimal set of RDDs to cache as
> long as we didn't cache all intermediates?
> 
> From: Augusto Ribeiro <au...@gmail.com>
> Date: Wed, May 15, 2019 at 8:37 AM
> To: <us...@beam.apache.org>
> 
>> Hi Kyle,
>> 
>> Thanks for the help. It seems like I have no other choice than using Spark directly, since my job causes immense memory pressure if I can't decide what to cache.
>> 
>> Best regards,
>> Augusto
>> 
>> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
>> 
>> Minor correction: Slack channel is actually #beam-spark
>> 
>> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>> 
>> 
>> From: Kyle Weaver <kc...@google.com>
>> Date: Tue, May 14, 2019 at 9:38 AM
>> To: <us...@beam.apache.org>
>> 
>>> Hi Augusto,
>>> 
>>> Right now the default behavior is to cache all intermediate RDDs that are consumed more than once by the pipeline. This can be disabled with `options.setCacheDisabled(true)` [1], but there is currently no way for the user to specify to the runner that it should cache certain RDDs, but not others.
>>> 
>>> There has recently been some discussion on the Slack (#spark-beam) about implementing such a feature, but no concrete plans as of yet.
>>> 
>>> [1] https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
>>> 
>>> Thanks
>>> 
>>> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>>> 
>>> 
>>> From: augusto.mcc@gmail.com <au...@gmail.com>
>>> Date: Tue, May 14, 2019 at 5:01 AM
>>> To: <us...@beam.apache.org>
>>> 
>>>> Hi,
>>>> 
>>>> I guess the title says it all, right now it seems like BEAM caches all the intermediate RDD results for my pipeline when using the Spark runner, this leads to a very inefficient usage of memory. Any way to control this?
>>>> 
>>>> Best regards,
>>>> Augusto
>> 
>> 


Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Jan Lukavský <je...@seznam.cz>.
The DAG formatted strangely, the intended formatting was

PCollectionA -> PCollectionB -> PCollectionC

                             \-> PCollectionD

Jan

On 5/15/19 11:19 AM, Jan Lukavský wrote:
> Hi,
>
> I think this thread is another manifestation of a problem discussed 
> recently in [1]. Long story short - users in certain situations might 
> legitimately need finer control over how is their pipeline translated 
> into runner's operators. The case of caching is another one, where 
> looking at the pipeline itself doesn't contain enough information to 
> perform correct optimization - consider for example this DAG of 
> operations:
>
>  PCollectionA -> PCollectionB -> PCollectionC
>
>                                               \-> PCollectionD
>
> That is PCollectionB is consumed by two operators - PCollectionC and 
> PCollectionD. Logical conclusion would be that we need to cache 
> PCollectionB, but what if the transform that produced PCollectionB is 
> of type "cheap explosion" - where PCollectionB is significantly bigger 
> than PCollectionA and at the same time can be produced very cheaply 
> from elements of PCollectionA? Than it would make sense to cache 
> PCollectionA instead. But you cannot know that just from the DAG. 
> There would be many more examples of this. Maybe we could think about 
> how to support this, which might help widen the user base.
>
> Jan
>
> [1] https://www.mail-archive.com/user@beam.apache.org/msg03809.html
>
> On 5/15/19 10:39 AM, Robert Bradshaw wrote:
>> Just to clarify, do you need direct control over what to cache, or
>> would it be OK to let Spark decide the minimal set of RDDs to cache as
>> long as we didn't cache all intermediates?
>>
>> From: Augusto Ribeiro <au...@gmail.com>
>> Date: Wed, May 15, 2019 at 8:37 AM
>> To: <us...@beam.apache.org>
>>
>>> Hi Kyle,
>>>
>>> Thanks for the help. It seems like I have no other choice than using 
>>> Spark directly, since my job causes immense memory pressure if I 
>>> can't decide what to cache.
>>>
>>> Best regards,
>>> Augusto
>>>
>>> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
>>>
>>> Minor correction: Slack channel is actually #beam-spark
>>>
>>> Kyle Weaver | Software Engineer | github.com/ibzib | 
>>> kcweaver@google.com | +16502035555
>>>
>>>
>>> From: Kyle Weaver <kc...@google.com>
>>> Date: Tue, May 14, 2019 at 9:38 AM
>>> To: <us...@beam.apache.org>
>>>
>>>> Hi Augusto,
>>>>
>>>> Right now the default behavior is to cache all intermediate RDDs 
>>>> that are consumed more than once by the pipeline. This can be 
>>>> disabled with `options.setCacheDisabled(true)` [1], but there is 
>>>> currently no way for the user to specify to the runner that it 
>>>> should cache certain RDDs, but not others.
>>>>
>>>> There has recently been some discussion on the Slack (#spark-beam) 
>>>> about implementing such a feature, but no concrete plans as of yet.
>>>>
>>>> [1] 
>>>> https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
>>>>
>>>> Thanks
>>>>
>>>> Kyle Weaver | Software Engineer | github.com/ibzib | 
>>>> kcweaver@google.com | +16502035555
>>>>
>>>>
>>>> From: augusto.mcc@gmail.com <au...@gmail.com>
>>>> Date: Tue, May 14, 2019 at 5:01 AM
>>>> To: <us...@beam.apache.org>
>>>>
>>>>> Hi,
>>>>>
>>>>> I guess the title says it all, right now it seems like BEAM caches 
>>>>> all the intermediate RDD results for my pipeline when using the 
>>>>> Spark runner, this leads to a very inefficient usage of memory. 
>>>>> Any way to control this?
>>>>>
>>>>> Best regards,
>>>>> Augusto
>>>

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Jan Lukavský <je...@seznam.cz>.
Hi,

I think this thread is another manifestation of a problem discussed 
recently in [1]. Long story short - users in certain situations might 
legitimately need finer control over how is their pipeline translated 
into runner's operators. The case of caching is another one, where 
looking at the pipeline itself doesn't contain enough information to 
perform correct optimization - consider for example this DAG of operations:

  PCollectionA -> PCollectionB -> PCollectionC

                                               \-> PCollectionD

That is PCollectionB is consumed by two operators - PCollectionC and 
PCollectionD. Logical conclusion would be that we need to cache 
PCollectionB, but what if the transform that produced PCollectionB is of 
type "cheap explosion" - where PCollectionB is significantly bigger than 
PCollectionA and at the same time can be produced very cheaply from 
elements of PCollectionA? Than it would make sense to cache PCollectionA 
instead. But you cannot know that just from the DAG. There would be many 
more examples of this. Maybe we could think about how to support this, 
which might help widen the user base.

Jan

[1] https://www.mail-archive.com/user@beam.apache.org/msg03809.html

On 5/15/19 10:39 AM, Robert Bradshaw wrote:
> Just to clarify, do you need direct control over what to cache, or
> would it be OK to let Spark decide the minimal set of RDDs to cache as
> long as we didn't cache all intermediates?
>
> From: Augusto Ribeiro <au...@gmail.com>
> Date: Wed, May 15, 2019 at 8:37 AM
> To: <us...@beam.apache.org>
>
>> Hi Kyle,
>>
>> Thanks for the help. It seems like I have no other choice than using Spark directly, since my job causes immense memory pressure if I can't decide what to cache.
>>
>> Best regards,
>> Augusto
>>
>> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
>>
>> Minor correction: Slack channel is actually #beam-spark
>>
>> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>>
>>
>> From: Kyle Weaver <kc...@google.com>
>> Date: Tue, May 14, 2019 at 9:38 AM
>> To: <us...@beam.apache.org>
>>
>>> Hi Augusto,
>>>
>>> Right now the default behavior is to cache all intermediate RDDs that are consumed more than once by the pipeline. This can be disabled with `options.setCacheDisabled(true)` [1], but there is currently no way for the user to specify to the runner that it should cache certain RDDs, but not others.
>>>
>>> There has recently been some discussion on the Slack (#spark-beam) about implementing such a feature, but no concrete plans as of yet.
>>>
>>> [1] https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
>>>
>>> Thanks
>>>
>>> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>>>
>>>
>>> From: augusto.mcc@gmail.com <au...@gmail.com>
>>> Date: Tue, May 14, 2019 at 5:01 AM
>>> To: <us...@beam.apache.org>
>>>
>>>> Hi,
>>>>
>>>> I guess the title says it all, right now it seems like BEAM caches all the intermediate RDD results for my pipeline when using the Spark runner, this leads to a very inefficient usage of memory. Any way to control this?
>>>>
>>>> Best regards,
>>>> Augusto
>>

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Robert Bradshaw <ro...@google.com>.
Just to clarify, do you need direct control over what to cache, or
would it be OK to let Spark decide the minimal set of RDDs to cache as
long as we didn't cache all intermediates?

From: Augusto Ribeiro <au...@gmail.com>
Date: Wed, May 15, 2019 at 8:37 AM
To: <us...@beam.apache.org>

> Hi Kyle,
>
> Thanks for the help. It seems like I have no other choice than using Spark directly, since my job causes immense memory pressure if I can't decide what to cache.
>
> Best regards,
> Augusto
>
> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
>
> Minor correction: Slack channel is actually #beam-spark
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>
>
> From: Kyle Weaver <kc...@google.com>
> Date: Tue, May 14, 2019 at 9:38 AM
> To: <us...@beam.apache.org>
>
>> Hi Augusto,
>>
>> Right now the default behavior is to cache all intermediate RDDs that are consumed more than once by the pipeline. This can be disabled with `options.setCacheDisabled(true)` [1], but there is currently no way for the user to specify to the runner that it should cache certain RDDs, but not others.
>>
>> There has recently been some discussion on the Slack (#spark-beam) about implementing such a feature, but no concrete plans as of yet.
>>
>> [1] https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
>>
>> Thanks
>>
>> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com | +16502035555
>>
>>
>> From: augusto.mcc@gmail.com <au...@gmail.com>
>> Date: Tue, May 14, 2019 at 5:01 AM
>> To: <us...@beam.apache.org>
>>
>>> Hi,
>>>
>>> I guess the title says it all, right now it seems like BEAM caches all the intermediate RDD results for my pipeline when using the Spark runner, this leads to a very inefficient usage of memory. Any way to control this?
>>>
>>> Best regards,
>>> Augusto
>
>

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Augusto Ribeiro <au...@gmail.com>.
Hi Kyle,

Thanks for the help. It seems like I have no other choice than using Spark directly, since my job causes immense memory pressure if I can't decide what to cache.

Best regards,
Augusto

> On 14 May 2019, at 18:40, Kyle Weaver <kc...@google.com> wrote:
> 
> Minor correction: Slack channel is actually #beam-spark
> 
> Kyle Weaver | Software Engineer | github.com/ibzib <http://github.com/ibzib> | kcweaver@google.com <ma...@google.com> | +16502035555
> 
> 
> From: Kyle Weaver <kcweaver@google.com <ma...@google.com>>
> Date: Tue, May 14, 2019 at 9:38 AM
> To: <user@beam.apache.org <ma...@beam.apache.org>>
> 
> Hi Augusto,
> 
> Right now the default behavior is to cache all intermediate RDDs that are consumed more than once by the pipeline. This can be disabled with `options.setCacheDisabled(true)` [1], but there is currently no way for the user to specify to the runner that it should cache certain RDDs, but not others.
> 
> There has recently been some discussion on the Slack (#spark-beam) about implementing such a feature, but no concrete plans as of yet.
> 
> [1] https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150 <https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150>
> 
> Thanks
> 
> Kyle Weaver | Software Engineer | github.com/ibzib <http://github.com/ibzib> | kcweaver@google.com <ma...@google.com> | +16502035555
> 
> 
> From: augusto.mcc@gmail.com <ma...@gmail.com> <augusto.mcc@gmail.com <ma...@gmail.com>>
> Date: Tue, May 14, 2019 at 5:01 AM
> To: <user@beam.apache.org <ma...@beam.apache.org>>
> 
> Hi,
> 
> I guess the title says it all, right now it seems like BEAM caches all the intermediate RDD results for my pipeline when using the Spark runner, this leads to a very inefficient usage of memory. Any way to control this?
> 
> Best regards,
> Augusto


Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Kyle Weaver <kc...@google.com>.
Minor correction: Slack channel is actually #beam-spark

Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com |
+16502035555


*From: *Kyle Weaver <kc...@google.com>
*Date: *Tue, May 14, 2019 at 9:38 AM
*To: * <us...@beam.apache.org>

Hi Augusto,
>
> Right now the default behavior is to cache all intermediate RDDs that are
> consumed more than once by the pipeline. This can be disabled with
> `options.setCacheDisabled(true)` [1], but there is currently no way for the
> user to specify to the runner that it should cache certain RDDs, but not
> others.
>
> There has recently been some discussion on the Slack (#spark-beam) about
> implementing such a feature, but no concrete plans as of yet.
>
> [1]
> https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150
>
> Thanks
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
> | +16502035555
>
>
> *From: *augusto.mcc@gmail.com <au...@gmail.com>
> *Date: *Tue, May 14, 2019 at 5:01 AM
> *To: * <us...@beam.apache.org>
>
> Hi,
>>
>> I guess the title says it all, right now it seems like BEAM caches all
>> the intermediate RDD results for my pipeline when using the Spark runner,
>> this leads to a very inefficient usage of memory. Any way to control this?
>>
>> Best regards,
>> Augusto
>>
>

Re: Is there a way to decide what RDDs get cached in the Spark Runner?

Posted by Kyle Weaver <kc...@google.com>.
Hi Augusto,

Right now the default behavior is to cache all intermediate RDDs that are
consumed more than once by the pipeline. This can be disabled with
`options.setCacheDisabled(true)` [1], but there is currently no way for the
user to specify to the runner that it should cache certain RDDs, but not
others.

There has recently been some discussion on the Slack (#spark-beam) about
implementing such a feature, but no concrete plans as of yet.

[1]
https://github.com/apache/beam/blob/81faf35c8a42493317eba9fa1e7b06fb42d54662/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java#L150

Thanks

Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com |
+16502035555


*From: *augusto.mcc@gmail.com <au...@gmail.com>
*Date: *Tue, May 14, 2019 at 5:01 AM
*To: * <us...@beam.apache.org>

Hi,
>
> I guess the title says it all, right now it seems like BEAM caches all the
> intermediate RDD results for my pipeline when using the Spark runner, this
> leads to a very inefficient usage of memory. Any way to control this?
>
> Best regards,
> Augusto
>