You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alan Braithwaite <al...@cloudflare.com> on 2015/11/12 01:48:54 UTC
Window's Operations on Spark Partitioned RDD
Hey All,
I'm having an issue where I think doing a window operation is using a
UnionRDD when it should be using a PartitionAware RDD. Can someone point
me in the right direction? (First by telling me what kind of information
you need. I can provide a ton, but wasn't sure how to make it concise.)
The result of this is that there's too much network I/O during the shuffle
from filter->window.
Screenshots and details follow.
Thanks,
- Alan
Spark 1.5.1
76 Partition Kafka topic
Kafka 0.8.2
Here's the DAG of the job:
[image: Inline image 2]
Here's the relevant code that I think should be running:
[image: Inline image 3]
Thanks again.
Re: Window's Operations on Spark Partitioned RDD
Posted by Cody Koeninger <co...@koeninger.org>.
I basically never use interactive debuggers. I'd just put more logging in.
Is there a reason you expect the rdds to have a partitioner? The default
for partitioner is None
On Fri, Nov 13, 2015 at 5:57 PM, Alan Braithwaite <al...@cloudflare.com>
wrote:
> Here we are. Ran into some other issues yesterday so I only got to it
> today. Going to look into debugging this locally, any suggestions for
> debugging this issue would be appreciated (e.g. intellij interactive
> debugging hints/docs and whatnot). I haven't worked extensively with scala
> yet but pointers are all I need.
>
> Relevant logs follow. I'll need to sanitize these logs if I want to
> upload the full thing, otherwise we'll get more info after I've reproduced
> it locally.
>
> alan@spark-driver:~$ grep 'Using partition aware union' spark.log
> alan@spark-driver:~$ grep 'Using normal union for' spark.log
> 15/11/13 23:36:54 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457800000 ms
> 15/11/13 23:37:56 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457860000 ms
> 15/11/13 23:38:54 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457920000 ms
> 15/11/13 23:39:56 DEBUG WindowedDStream: Using normal union for windowing
> at 1447457980000 ms
>
> Thanks,
> - Alan
>
> On Thu, Nov 12, 2015 at 11:15 AM, Alan Braithwaite <al...@cloudflare.com>
> wrote:
>
>> Yes of course, working on that now. That should appear in the driver log
>> correct?
>>
>> Thanks,
>> - Alan
>>
>> On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org>
>> wrote:
>>
>>> So, starting with the most basic question - which branch of that if
>>> statement are you hitting? I.e. which logDebug statement is being logged?
>>>
>>> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
>>> wrote:
>>>
>>>> Hey All,
>>>>
>>>> I'm having an issue where I think doing a window operation is using a
>>>> UnionRDD when it should be using a PartitionAware RDD. Can someone point
>>>> me in the right direction? (First by telling me what kind of information
>>>> you need. I can provide a ton, but wasn't sure how to make it concise.)
>>>>
>>>> The result of this is that there's too much network I/O during the
>>>> shuffle from filter->window.
>>>>
>>>> Screenshots and details follow.
>>>>
>>>> Thanks,
>>>> - Alan
>>>>
>>>> Spark 1.5.1
>>>> 76 Partition Kafka topic
>>>> Kafka 0.8.2
>>>>
>>>> Here's the DAG of the job:
>>>>
>>>> [image: Inline image 2]
>>>>
>>>> Here's the relevant code that I think should be running:
>>>>
>>>> [image: Inline image 3]
>>>>
>>>> Thanks again.
>>>>
>>>
>>>
>>
>
Re: Window's Operations on Spark Partitioned RDD
Posted by Alan Braithwaite <al...@cloudflare.com>.
Here we are. Ran into some other issues yesterday so I only got to it
today. Going to look into debugging this locally, any suggestions for
debugging this issue would be appreciated (e.g. intellij interactive
debugging hints/docs and whatnot). I haven't worked extensively with scala
yet but pointers are all I need.
Relevant logs follow. I'll need to sanitize these logs if I want to upload
the full thing, otherwise we'll get more info after I've reproduced it
locally.
alan@spark-driver:~$ grep 'Using partition aware union' spark.log
alan@spark-driver:~$ grep 'Using normal union for' spark.log
15/11/13 23:36:54 DEBUG WindowedDStream: Using normal union for windowing
at 1447457800000 ms
15/11/13 23:37:56 DEBUG WindowedDStream: Using normal union for windowing
at 1447457860000 ms
15/11/13 23:38:54 DEBUG WindowedDStream: Using normal union for windowing
at 1447457920000 ms
15/11/13 23:39:56 DEBUG WindowedDStream: Using normal union for windowing
at 1447457980000 ms
Thanks,
- Alan
On Thu, Nov 12, 2015 at 11:15 AM, Alan Braithwaite <al...@cloudflare.com>
wrote:
> Yes of course, working on that now. That should appear in the driver log
> correct?
>
> Thanks,
> - Alan
>
> On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org>
> wrote:
>
>> So, starting with the most basic question - which branch of that if
>> statement are you hitting? I.e. which logDebug statement is being logged?
>>
>> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> I'm having an issue where I think doing a window operation is using a
>>> UnionRDD when it should be using a PartitionAware RDD. Can someone point
>>> me in the right direction? (First by telling me what kind of information
>>> you need. I can provide a ton, but wasn't sure how to make it concise.)
>>>
>>> The result of this is that there's too much network I/O during the
>>> shuffle from filter->window.
>>>
>>> Screenshots and details follow.
>>>
>>> Thanks,
>>> - Alan
>>>
>>> Spark 1.5.1
>>> 76 Partition Kafka topic
>>> Kafka 0.8.2
>>>
>>> Here's the DAG of the job:
>>>
>>> [image: Inline image 2]
>>>
>>> Here's the relevant code that I think should be running:
>>>
>>> [image: Inline image 3]
>>>
>>> Thanks again.
>>>
>>
>>
>
Re: Window's Operations on Spark Partitioned RDD
Posted by Alan Braithwaite <al...@cloudflare.com>.
Yes of course, working on that now. That should appear in the driver log
correct?
Thanks,
- Alan
On Thu, Nov 12, 2015 at 9:11 AM, Cody Koeninger <co...@koeninger.org> wrote:
> So, starting with the most basic question - which branch of that if
> statement are you hitting? I.e. which logDebug statement is being logged?
>
> On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
> wrote:
>
>> Hey All,
>>
>> I'm having an issue where I think doing a window operation is using a
>> UnionRDD when it should be using a PartitionAware RDD. Can someone point
>> me in the right direction? (First by telling me what kind of information
>> you need. I can provide a ton, but wasn't sure how to make it concise.)
>>
>> The result of this is that there's too much network I/O during the
>> shuffle from filter->window.
>>
>> Screenshots and details follow.
>>
>> Thanks,
>> - Alan
>>
>> Spark 1.5.1
>> 76 Partition Kafka topic
>> Kafka 0.8.2
>>
>> Here's the DAG of the job:
>>
>> [image: Inline image 2]
>>
>> Here's the relevant code that I think should be running:
>>
>> [image: Inline image 3]
>>
>> Thanks again.
>>
>
>
Re: Window's Operations on Spark Partitioned RDD
Posted by Cody Koeninger <co...@koeninger.org>.
So, starting with the most basic question - which branch of that if
statement are you hitting? I.e. which logDebug statement is being logged?
On Wed, Nov 11, 2015 at 6:48 PM, Alan Braithwaite <al...@cloudflare.com>
wrote:
> Hey All,
>
> I'm having an issue where I think doing a window operation is using a
> UnionRDD when it should be using a PartitionAware RDD. Can someone point
> me in the right direction? (First by telling me what kind of information
> you need. I can provide a ton, but wasn't sure how to make it concise.)
>
> The result of this is that there's too much network I/O during the shuffle
> from filter->window.
>
> Screenshots and details follow.
>
> Thanks,
> - Alan
>
> Spark 1.5.1
> 76 Partition Kafka topic
> Kafka 0.8.2
>
> Here's the DAG of the job:
>
> [image: Inline image 2]
>
> Here's the relevant code that I think should be running:
>
> [image: Inline image 3]
>
> Thanks again.
>