You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ravindra <ra...@gmail.com> on 2016/06/26 04:21:23 UTC
Spark Task is not created
Hi All,
May be I need to just set some property or its a known issue. My spark
application hangs in test environment whenever I see following message -
16/06/26 11:13:34 INFO DAGScheduler: *Submitting 2 missing tasks from
ShuffleMapStage* 145 (MapPartitionsRDD[590] at rdd at
WriteDataFramesDecorator.scala:61)
16/06/26 11:13:34 INFO TaskSchedulerImpl: Adding task set 145.0 with 2 tasks
16/06/26 11:13:34 INFO TaskSetManager: Starting task 0.0 in stage 145.0
(TID 186, localhost, PROCESS_LOCAL, 2389 bytes)
16/06/26 11:13:34 INFO Executor: Running task 0.0 in stage 145.0 (TID 186)
16/06/26 11:13:34 INFO BlockManager: Found block rdd_575_0 locally
16/06/26 11:13:34 INFO GenerateMutableProjection: Code generated in 3.796 ms
16/06/26 11:13:34 INFO Executor: Finished task 0.0 in stage 145.0 (TID
186). 2578 bytes result sent to driver
16/06/26 11:13:34 INFO TaskSetManager: Finished task 0.0 in stage 145.0
(TID 186) in 24 ms on localhost (1/2)
It happens with any action. The application works fine whenever I
notice "*Submitting
1 missing tasks from ShuffleMapStage". *For this I need to tweak the plan
like using repartition, coalesce etc but this also doesn't help always.
Some of the Spark properties are as given below -
Name Value
spark.app.id local-1466914377931
spark.app.name SparkTest
spark.cores.max 3
spark.default.parallelism 1
spark.driver.allowMultipleContexts true
spark.executor.id driver
spark.externalBlockStore.folderName
spark-050049bd-c058-4035-bc3d-2e73a08e8d0c
spark.master local[2]
spark.scheduler.mode FIFO
spark.ui.enabled true
Thanks,
Ravi.
Re: Spark Task is not created
Posted by Ravindra <ra...@gmail.com>.
I have a lot of spark tests. And the failure is not deterministic. It can
happen at any action that I do. Buy given below logs are common. And I
overcome that using the repartitioning, coalescing etc so that I don't get
that Submitting 2 missing tasks from ShuffleMapStage. Basically ensuring
that there is only one task.
I doubt if this has anything to do with some property.
In the Ui I don't see any failure. Just that few tasks have completed and
the last one is yet to be created
Thanks,
Ravi
On Sun, Jun 26, 2016, 11:33 Akhil Das <ak...@hacked.work> wrote:
> Would be good if you can paste the piece of code that you are executing.
>
> On Sun, Jun 26, 2016 at 11:21 AM, Ravindra <ra...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> May be I need to just set some property or its a known issue. My spark
>> application hangs in test environment whenever I see following message -
>>
>> 16/06/26 11:13:34 INFO DAGScheduler: *Submitting 2 missing tasks from
>> ShuffleMapStage* 145 (MapPartitionsRDD[590] at rdd at
>> WriteDataFramesDecorator.scala:61)
>> 16/06/26 11:13:34 INFO TaskSchedulerImpl: Adding task set 145.0 with 2
>> tasks
>> 16/06/26 11:13:34 INFO TaskSetManager: Starting task 0.0 in stage 145.0
>> (TID 186, localhost, PROCESS_LOCAL, 2389 bytes)
>> 16/06/26 11:13:34 INFO Executor: Running task 0.0 in stage 145.0 (TID 186)
>> 16/06/26 11:13:34 INFO BlockManager: Found block rdd_575_0 locally
>> 16/06/26 11:13:34 INFO GenerateMutableProjection: Code generated in 3.796
>> ms
>> 16/06/26 11:13:34 INFO Executor: Finished task 0.0 in stage 145.0 (TID
>> 186). 2578 bytes result sent to driver
>> 16/06/26 11:13:34 INFO TaskSetManager: Finished task 0.0 in stage 145.0
>> (TID 186) in 24 ms on localhost (1/2)
>>
>> It happens with any action. The application works fine whenever I notice "*Submitting
>> 1 missing tasks from ShuffleMapStage". *For this I need to tweak the
>> plan like using repartition, coalesce etc but this also doesn't help
>> always.
>>
>> Some of the Spark properties are as given below -
>>
>> Name Value
>> spark.app.id local-1466914377931
>> spark.app.name SparkTest
>> spark.cores.max 3
>> spark.default.parallelism 1
>> spark.driver.allowMultipleContexts true
>> spark.executor.id driver
>> spark.externalBlockStore.folderName
>> spark-050049bd-c058-4035-bc3d-2e73a08e8d0c
>> spark.master local[2]
>> spark.scheduler.mode FIFO
>> spark.ui.enabled true
>>
>>
>> Thanks,
>> Ravi.
>>
>>
>
>
> --
> Cheers!
>
>
Re: Spark Task is not created
Posted by Akhil Das <ak...@hacked.work>.
Would be good if you can paste the piece of code that you are executing.
On Sun, Jun 26, 2016 at 11:21 AM, Ravindra <ra...@gmail.com>
wrote:
> Hi All,
>
> May be I need to just set some property or its a known issue. My spark
> application hangs in test environment whenever I see following message -
>
> 16/06/26 11:13:34 INFO DAGScheduler: *Submitting 2 missing tasks from
> ShuffleMapStage* 145 (MapPartitionsRDD[590] at rdd at
> WriteDataFramesDecorator.scala:61)
> 16/06/26 11:13:34 INFO TaskSchedulerImpl: Adding task set 145.0 with 2
> tasks
> 16/06/26 11:13:34 INFO TaskSetManager: Starting task 0.0 in stage 145.0
> (TID 186, localhost, PROCESS_LOCAL, 2389 bytes)
> 16/06/26 11:13:34 INFO Executor: Running task 0.0 in stage 145.0 (TID 186)
> 16/06/26 11:13:34 INFO BlockManager: Found block rdd_575_0 locally
> 16/06/26 11:13:34 INFO GenerateMutableProjection: Code generated in 3.796
> ms
> 16/06/26 11:13:34 INFO Executor: Finished task 0.0 in stage 145.0 (TID
> 186). 2578 bytes result sent to driver
> 16/06/26 11:13:34 INFO TaskSetManager: Finished task 0.0 in stage 145.0
> (TID 186) in 24 ms on localhost (1/2)
>
> It happens with any action. The application works fine whenever I notice "*Submitting
> 1 missing tasks from ShuffleMapStage". *For this I need to tweak the plan
> like using repartition, coalesce etc but this also doesn't help always.
>
> Some of the Spark properties are as given below -
>
> Name Value
> spark.app.id local-1466914377931
> spark.app.name SparkTest
> spark.cores.max 3
> spark.default.parallelism 1
> spark.driver.allowMultipleContexts true
> spark.executor.id driver
> spark.externalBlockStore.folderName
> spark-050049bd-c058-4035-bc3d-2e73a08e8d0c
> spark.master local[2]
> spark.scheduler.mode FIFO
> spark.ui.enabled true
>
>
> Thanks,
> Ravi.
>
>
--
Cheers!