Initial tasks in job take time

[Anshul Singhle <an...@betaglide.com>]

Hi,

I'm running short spark jobs on rdds cached in memory. I'm also using a
long running job context. I want to be able to complete my jobs (on the
cached rdd) in under 1 sec.
I'm getting the following job times with about 15 GB of data distributed
across 6 nodes. Each executor has about 20GB of memory available. My
context has about 26 cores in total.

If number of partitions < no of cores -

Some jobs run in 3s others take about 6s -- the time difference can be
explained by GC time.

If number of partitions = no of cores -

All jobs run in 4s. The initial tasks of each stage on every executor take
about 1s.

If partitions > cores -

Jobs take more time. The initial tasks of each stage on every executor take
about 1s. The other tasks run in 45-50 ms each. However, since the initial
tasks again take about 1s each, the total time in this case is about 6s
which is more than the previous case.

Clearly the limiting factor here is the initial set of tasks. For every
case, these tasks take 1s to run, no matter the amount of partitions. Hence
best results are obtained with partitions = cores, because in that case.
every core gets 1 task which takes 1s to run.
In this case, I get 0 GC time. The only explanation is "scheduling delay"
which is about 0.2 - 0.3 seconds. I looked at my task size and result size
and that has no bearing on this delay. Also, I'm not getting the task size
warnings in the logs.

For what I can understand, the first time a task runs on a core, it takes
1s to run. Is this normal?

Is it possible to get sub-second latencies?

Can something be done about the scheduler delay?

What other things can I look at to reduce this time?

Regards,

Anshul

Re: Initial tasks in job take time

[Anshul Singhle <an...@betaglide.com>]

yes
On 29 Apr 2015 03:31, "ayan guha" <gu...@gmail.com> wrote:

> Are your driver running on the same m/c as master?
> On 29 Apr 2015 03:59, "Anshul Singhle" <an...@betaglide.com> wrote:
>
>> Hi,
>>
>> I'm running short spark jobs on rdds cached in memory. I'm also using a
>> long running job context. I want to be able to complete my jobs (on the
>> cached rdd) in under 1 sec.
>> I'm getting the following job times with about 15 GB of data distributed
>> across 6 nodes. Each executor has about 20GB of memory available. My
>> context has about 26 cores in total.
>>
>> If number of partitions < no of cores -
>>
>> Some jobs run in 3s others take about 6s -- the time difference can be
>> explained by GC time.
>>
>> If number of partitions = no of cores -
>>
>> All jobs run in 4s. The initial tasks of each stage on every executor
>> take about 1s.
>>
>> If partitions > cores -
>>
>> Jobs take more time. The initial tasks of each stage on every executor
>> take about 1s. The other tasks run in 45-50 ms each. However, since the
>> initial tasks again take about 1s each, the total time in this case is
>> about 6s which is more than the previous case.
>>
>> Clearly the limiting factor here is the initial set of tasks. For every
>> case, these tasks take 1s to run, no matter the amount of partitions. Hence
>> best results are obtained with partitions = cores, because in that case.
>> every core gets 1 task which takes 1s to run.
>> In this case, I get 0 GC time. The only explanation is "scheduling delay"
>> which is about 0.2 - 0.3 seconds. I looked at my task size and result size
>> and that has no bearing on this delay. Also, I'm not getting the task size
>> warnings in the logs.
>>
>> For what I can understand, the first time a task runs on a core, it takes
>> 1s to run. Is this normal?
>>
>> Is it possible to get sub-second latencies?
>>
>> Can something be done about the scheduler delay?
>>
>> What other things can I look at to reduce this time?
>>
>> Regards,
>>
>> Anshul
>>
>>
>>

Re: Initial tasks in job take time

[ayan guha <gu...@gmail.com>]

Are your driver running on the same m/c as master?
On 29 Apr 2015 03:59, "Anshul Singhle" <an...@betaglide.com> wrote:

> Hi,
>
> I'm running short spark jobs on rdds cached in memory. I'm also using a
> long running job context. I want to be able to complete my jobs (on the
> cached rdd) in under 1 sec.
> I'm getting the following job times with about 15 GB of data distributed
> across 6 nodes. Each executor has about 20GB of memory available. My
> context has about 26 cores in total.
>
> If number of partitions < no of cores -
>
> Some jobs run in 3s others take about 6s -- the time difference can be
> explained by GC time.
>
> If number of partitions = no of cores -
>
> All jobs run in 4s. The initial tasks of each stage on every executor take
> about 1s.
>
> If partitions > cores -
>
> Jobs take more time. The initial tasks of each stage on every executor
> take about 1s. The other tasks run in 45-50 ms each. However, since the
> initial tasks again take about 1s each, the total time in this case is
> about 6s which is more than the previous case.
>
> Clearly the limiting factor here is the initial set of tasks. For every
> case, these tasks take 1s to run, no matter the amount of partitions. Hence
> best results are obtained with partitions = cores, because in that case.
> every core gets 1 task which takes 1s to run.
> In this case, I get 0 GC time. The only explanation is "scheduling delay"
> which is about 0.2 - 0.3 seconds. I looked at my task size and result size
> and that has no bearing on this delay. Also, I'm not getting the task size
> warnings in the logs.
>
> For what I can understand, the first time a task runs on a core, it takes
> 1s to run. Is this normal?
>
> Is it possible to get sub-second latencies?
>
> Can something be done about the scheduler delay?
>
> What other things can I look at to reduce this time?
>
> Regards,
>
> Anshul
>
>
>