You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michael Campbell <mi...@gmail.com> on 2014/10/16 23:22:40 UTC
Spark Bug? job fails to run when given options on spark-submit (but
starts and fails without)
TL;DR - a spark SQL job fails with an OOM (Out of heap space) error. If
given "--executor-memory" values, it won't even start. Even (!) if the
values given ARE THE SAME AS THE DEFAULT.
Without --executor-memory:
14/10/16 17:14:58 INFO TaskSetManager: Serialized task 1.0:64 as 14710
bytes in 1 ms
14/10/16 17:14:58 WARN TaskSetManager: Lost TID 26 (task 1.0:25)
14/10/16 17:14:58 WARN TaskSetManager: Loss was due to
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
at
parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:609)
at
parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
...
USING --executor-memory (WITH ANY VALUE), even "1G" which is the default:
Parsed arguments:
master spark://<redacted>:7077
deployMode null
executorMemory 1G
...
System properties:
spark.executor.memory -> 1G
spark.eventLog.enabled -> true
...
14/10/16 17:14:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 678 tasks
14/10/16 17:14:38 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
Spark 1.0.0. Is this a bug?
Re: Spark Bug? job fails to run when given options on spark-submit
(but starts and fails without)
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can open the application UI (that runs on 4040) and see how much memory
is being allocated to the executor tabs and from the environments tab.
Thanks
Best Regards
On Wed, Oct 22, 2014 at 9:55 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Hi Michael Campbell,
>
> Are you deploying against yarn or standalone mode? In yarn try setting the
> shell variables SPARK_EXECUTOR_MEMORY=2G in standalone try and
> set SPARK_WORKER_MEMORY=2G.
>
>
> Cheers,
>
> Holden :)
>
> On Thu, Oct 16, 2014 at 2:22 PM, Michael Campbell <
> michael.campbell@gmail.com> wrote:
>
>> TL;DR - a spark SQL job fails with an OOM (Out of heap space) error. If
>> given "--executor-memory" values, it won't even start. Even (!) if the
>> values given ARE THE SAME AS THE DEFAULT.
>>
>>
>>
>> Without --executor-memory:
>>
>> 14/10/16 17:14:58 INFO TaskSetManager: Serialized task 1.0:64 as 14710
>> bytes in 1 ms
>> 14/10/16 17:14:58 WARN TaskSetManager: Lost TID 26 (task 1.0:25)
>> 14/10/16 17:14:58 WARN TaskSetManager: Loss was due to
>> java.lang.OutOfMemoryError
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:609)
>> at
>> parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
>> ...
>>
>>
>> USING --executor-memory (WITH ANY VALUE), even "1G" which is the default:
>>
>> Parsed arguments:
>> master spark://<redacted>:7077
>> deployMode null
>> executorMemory 1G
>> ...
>>
>> System properties:
>> spark.executor.memory -> 1G
>> spark.eventLog.enabled -> true
>> ...
>>
>> 14/10/16 17:14:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 678
>> tasks
>> 14/10/16 17:14:38 WARN TaskSchedulerImpl: Initial job has not accepted
>> any resources; check your cluster UI to ensure that workers are registered
>> and have sufficient memory
>>
>>
>>
>> Spark 1.0.0. Is this a bug?
>>
>>
>
>
> --
> Cell : 425-233-8271
>
Re: Spark Bug? job fails to run when given options on spark-submit
(but starts and fails without)
Posted by Holden Karau <ho...@pigscanfly.ca>.
Hi Michael Campbell,
Are you deploying against yarn or standalone mode? In yarn try setting the
shell variables SPARK_EXECUTOR_MEMORY=2G in standalone try and
set SPARK_WORKER_MEMORY=2G.
Cheers,
Holden :)
On Thu, Oct 16, 2014 at 2:22 PM, Michael Campbell <
michael.campbell@gmail.com> wrote:
> TL;DR - a spark SQL job fails with an OOM (Out of heap space) error. If
> given "--executor-memory" values, it won't even start. Even (!) if the
> values given ARE THE SAME AS THE DEFAULT.
>
>
>
> Without --executor-memory:
>
> 14/10/16 17:14:58 INFO TaskSetManager: Serialized task 1.0:64 as 14710
> bytes in 1 ms
> 14/10/16 17:14:58 WARN TaskSetManager: Lost TID 26 (task 1.0:25)
> 14/10/16 17:14:58 WARN TaskSetManager: Loss was due to
> java.lang.OutOfMemoryError
> java.lang.OutOfMemoryError: Java heap space
> at
> parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:609)
> at
> parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
> ...
>
>
> USING --executor-memory (WITH ANY VALUE), even "1G" which is the default:
>
> Parsed arguments:
> master spark://<redacted>:7077
> deployMode null
> executorMemory 1G
> ...
>
> System properties:
> spark.executor.memory -> 1G
> spark.eventLog.enabled -> true
> ...
>
> 14/10/16 17:14:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 678
> tasks
> 14/10/16 17:14:38 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
>
>
> Spark 1.0.0. Is this a bug?
>
>
--
Cell : 425-233-8271