You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alan Braithwaite <al...@cloudflare.com> on 2015/11/12 01:48:54 UTC

Window's Operations on Spark Partitioned RDD

Hey All,

I'm having an issue where I think doing a window operation is using a
UnionRDD when it should be using a PartitionAware RDD.  Can someone point
me in the right direction? (First by telling me what kind of information
you need.  I can provide a ton, but wasn't sure how to make it concise.)

The result of this is that there's too much network I/O during the shuffle
from filter->window.

Screenshots and details follow.

Thanks,
- Alan

Spark 1.5.1
76 Partition Kafka topic
Kafka 0.8.2

Here's the DAG of the job:

[image: Inline image 2]

Here's the relevant code that I think should be running:

[image: Inline image 3]

Thanks again.

Re: Window's Operations on Spark Partitioned RDD

Posted by Cody Koeninger <co...@koeninger.org>.
I basically never use interactive debuggers.  I'd just put more logging in.

Is there a reason you expect the rdds to have a partitioner?  The default
for partitioner is None

On Fri, Nov 13, 2015 at 5:57 PM, Alan Braithwaite <al...@cloudflare.com>
wrote:

> Here we are.  Ran into some other issues yesterday so I only got to it
> today.  Going to look into debugging this locally, any suggestions for
> debugging this issue would be appreciated (e.g. intellij interactive
> debugging hints/docs and whatnot).  I haven't worked extensively with scala
> yet but pointers are all I need.
>
> Relevant logs follow.  I'll need to sanitize these logs if I want to
> upload the full thing, otherwise we'll get more info after I've reproduced
> it locally.
>
> alan@spark-driver:~$ grep 'Using partition aware union' spark.log
> alan@spark-driver:~$ grep 'Using normal union for' spark.log
> 15/11/13 23:36:54 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457800000 ms
> 15/11/13 23:37:56 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457860000 ms
> 15/11/13 23:38:54 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457920000 ms
> 15/11/13 23:39:56 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457980000 ms
>
> Thanks,
> - Alan
>
> On Thu, Nov 12, 2015 at 11:15 AM, Alan Braithwaite <al...@cloudflare.com>
> wrote:
>
>> Yes of course, working on that now.  That should appear in the driver log
>> correct?
>>
>> Thanks,
>> - Alan
>>
>> On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org>
>> wrote:
>>
>>> So, starting with the most basic question - which branch of that if
>>> statement are you hitting?  I.e. which logDebug statement is being logged?
>>>
>>> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
>>> wrote:
>>>
>>>> Hey All,
>>>>
>>>> I'm having an issue where I think doing a window operation is using a
>>>> UnionRDD when it should be using a PartitionAware RDD.  Can someone point
>>>> me in the right direction? (First by telling me what kind of information
>>>> you need.  I can provide a ton, but wasn't sure how to make it concise.)
>>>>
>>>> The result of this is that there's too much network I/O during the
>>>> shuffle from filter->window.
>>>>
>>>> Screenshots and details follow.
>>>>
>>>> Thanks,
>>>> - Alan
>>>>
>>>> Spark 1.5.1
>>>> 76 Partition Kafka topic
>>>> Kafka 0.8.2
>>>>
>>>> Here's the DAG of the job:
>>>>
>>>> [image: Inline image 2]
>>>>
>>>> Here's the relevant code that I think should be running:
>>>>
>>>> [image: Inline image 3]
>>>>
>>>> Thanks again.
>>>>
>>>
>>>
>>
>

Re: Window's Operations on Spark Partitioned RDD

Posted by Alan Braithwaite <al...@cloudflare.com>.
Here we are.  Ran into some other issues yesterday so I only got to it
today.  Going to look into debugging this locally, any suggestions for
debugging this issue would be appreciated (e.g. intellij interactive
debugging hints/docs and whatnot).  I haven't worked extensively with scala
yet but pointers are all I need.

Relevant logs follow.  I'll need to sanitize these logs if I want to upload
the full thing, otherwise we'll get more info after I've reproduced it
locally.

alan@spark-driver:~$ grep 'Using partition aware union' spark.log
alan@spark-driver:~$ grep 'Using normal union for' spark.log
15/11/13 23:36:54 DEBUG WindowedDStream: Using normal union for windowing
at 1447457800000 ms
15/11/13 23:37:56 DEBUG WindowedDStream: Using normal union for windowing
at 1447457860000 ms
15/11/13 23:38:54 DEBUG WindowedDStream: Using normal union for windowing
at 1447457920000 ms
15/11/13 23:39:56 DEBUG WindowedDStream: Using normal union for windowing
at 1447457980000 ms

Thanks,
- Alan

On Thu, Nov 12, 2015 at 11:15 AM, Alan Braithwaite <al...@cloudflare.com>
wrote:

> Yes of course, working on that now.  That should appear in the driver log
> correct?
>
> Thanks,
> - Alan
>
> On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org>
> wrote:
>
>> So, starting with the most basic question - which branch of that if
>> statement are you hitting?  I.e. which logDebug statement is being logged?
>>
>> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> I'm having an issue where I think doing a window operation is using a
>>> UnionRDD when it should be using a PartitionAware RDD.  Can someone point
>>> me in the right direction? (First by telling me what kind of information
>>> you need.  I can provide a ton, but wasn't sure how to make it concise.)
>>>
>>> The result of this is that there's too much network I/O during the
>>> shuffle from filter->window.
>>>
>>> Screenshots and details follow.
>>>
>>> Thanks,
>>> - Alan
>>>
>>> Spark 1.5.1
>>> 76 Partition Kafka topic
>>> Kafka 0.8.2
>>>
>>> Here's the DAG of the job:
>>>
>>> [image: Inline image 2]
>>>
>>> Here's the relevant code that I think should be running:
>>>
>>> [image: Inline image 3]
>>>
>>> Thanks again.
>>>
>>
>>
>

Re: Window's Operations on Spark Partitioned RDD

Posted by Alan Braithwaite <al...@cloudflare.com>.
Yes of course, working on that now.  That should appear in the driver log
correct?

Thanks,
- Alan

On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org> wrote:

> So, starting with the most basic question - which branch of that if
> statement are you hitting?  I.e. which logDebug statement is being logged?
>
> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
> wrote:
>
>> Hey All,
>>
>> I'm having an issue where I think doing a window operation is using a
>> UnionRDD when it should be using a PartitionAware RDD.  Can someone point
>> me in the right direction? (First by telling me what kind of information
>> you need.  I can provide a ton, but wasn't sure how to make it concise.)
>>
>> The result of this is that there's too much network I/O during the
>> shuffle from filter->window.
>>
>> Screenshots and details follow.
>>
>> Thanks,
>> - Alan
>>
>> Spark 1.5.1
>> 76 Partition Kafka topic
>> Kafka 0.8.2
>>
>> Here's the DAG of the job:
>>
>> [image: Inline image 2]
>>
>> Here's the relevant code that I think should be running:
>>
>> [image: Inline image 3]
>>
>> Thanks again.
>>
>
>

Re: Window's Operations on Spark Partitioned RDD

Posted by Cody Koeninger <co...@koeninger.org>.
So, starting with the most basic question - which branch of that if
statement are you hitting?  I.e. which logDebug statement is being logged?

On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
wrote:

> Hey All,
>
> I'm having an issue where I think doing a window operation is using a
> UnionRDD when it should be using a PartitionAware RDD.  Can someone point
> me in the right direction? (First by telling me what kind of information
> you need.  I can provide a ton, but wasn't sure how to make it concise.)
>
> The result of this is that there's too much network I/O during the shuffle
> from filter->window.
>
> Screenshots and details follow.
>
> Thanks,
> - Alan
>
> Spark 1.5.1
> 76 Partition Kafka topic
> Kafka 0.8.2
>
> Here's the DAG of the job:
>
> [image: Inline image 2]
>
> Here's the relevant code that I think should be running:
>
> [image: Inline image 3]
>
> Thanks again.
>