You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Darin McBeath <dd...@yahoo.com.INVALID> on 2015/02/26 00:02:08 UTC
job keeps failing with
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 1
I'm using Spark 1.2, stand-alone cluster on ec2 I have a cluster of 8 r3.8xlarge machines but limit the job to only 128 cores. I have also tried other things such as setting 4 workers per r3.8xlarge and 67gb each but this made no difference.
The job frequently fails at the end in this step (saveasHadoopFile). It will sometimes work.
finalNewBaselinePairRDD is hashPartitioned with 1024 partitions and a total size around 1TB. There are about 13.5M records in finalNewBaselinePairRDD. finalNewBaselinePairRDD is <String,String>
JavaPairRDD<Text, Text> finalBaselineRDDWritable = finalNewBaselinePairRDD.mapToPair(new ConvertToWritableTypes()).persist(StorageLevel.MEMORY_AND_DISK_SER());
// Save to hdfs (gzip)
finalBaselineRDDWritable.saveAsHadoopFile("hdfs:///sparksync/", Text.class, Text.class, SequenceFileOutputFormat.class,org.apache.hadoop.io.compress.GzipCodec.class);
If anyone has any tips for what I should look into it would be appreciated.
Thanks.
Darin.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: job keeps failing with org.apache.spark.shuffle.MetadataFetchFailedException:
Missing an output location for shuffle 1
Posted by Kelvin Chu <2d...@gmail.com>.
Hi Darin, you might increase spark.yarn.executor.memoryOverhead to see if
it fixes the problem. Please take a look of this report:
https://issues.apache.org/jira/browse/SPARK-4996
On Fri, Feb 27, 2015 at 12:38 AM, Arush Kharbanda <
arush@sigmoidanalytics.com> wrote:
> Can you share what error you are getting when the job fails.
>
> On Thu, Feb 26, 2015 at 4:32 AM, Darin McBeath <
> ddmcbeath@yahoo.com.invalid> wrote:
>
>> I'm using Spark 1.2, stand-alone cluster on ec2 I have a cluster of 8
>> r3.8xlarge machines but limit the job to only 128 cores. I have also tried
>> other things such as setting 4 workers per r3.8xlarge and 67gb each but
>> this made no difference.
>>
>> The job frequently fails at the end in this step (saveasHadoopFile). It
>> will sometimes work.
>>
>> finalNewBaselinePairRDD is hashPartitioned with 1024 partitions and a
>> total size around 1TB. There are about 13.5M records in
>> finalNewBaselinePairRDD. finalNewBaselinePairRDD is <String,String>
>>
>>
>> JavaPairRDD<Text, Text> finalBaselineRDDWritable =
>> finalNewBaselinePairRDD.mapToPair(new
>> ConvertToWritableTypes()).persist(StorageLevel.MEMORY_AND_DISK_SER());
>>
>> // Save to hdfs (gzip)
>> finalBaselineRDDWritable.saveAsHadoopFile("hdfs:///sparksync/",
>> Text.class, Text.class,
>> SequenceFileOutputFormat.class,org.apache.hadoop.io.compress.GzipCodec.class);
>>
>>
>> If anyone has any tips for what I should look into it would be
>> appreciated.
>>
>> Thanks.
>>
>> Darin.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
>
> [image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>
>
> *Arush Kharbanda* || Technical Teamlead
>
> arush@sigmoidanalytics.com || www.sigmoidanalytics.com
>
Re: job keeps failing with org.apache.spark.shuffle.MetadataFetchFailedException:
Missing an output location for shuffle 1
Posted by Arush Kharbanda <ar...@sigmoidanalytics.com>.
Can you share what error you are getting when the job fails.
On Thu, Feb 26, 2015 at 4:32 AM, Darin McBeath <dd...@yahoo.com.invalid>
wrote:
> I'm using Spark 1.2, stand-alone cluster on ec2 I have a cluster of 8
> r3.8xlarge machines but limit the job to only 128 cores. I have also tried
> other things such as setting 4 workers per r3.8xlarge and 67gb each but
> this made no difference.
>
> The job frequently fails at the end in this step (saveasHadoopFile). It
> will sometimes work.
>
> finalNewBaselinePairRDD is hashPartitioned with 1024 partitions and a
> total size around 1TB. There are about 13.5M records in
> finalNewBaselinePairRDD. finalNewBaselinePairRDD is <String,String>
>
>
> JavaPairRDD<Text, Text> finalBaselineRDDWritable =
> finalNewBaselinePairRDD.mapToPair(new
> ConvertToWritableTypes()).persist(StorageLevel.MEMORY_AND_DISK_SER());
>
> // Save to hdfs (gzip)
> finalBaselineRDDWritable.saveAsHadoopFile("hdfs:///sparksync/",
> Text.class, Text.class,
> SequenceFileOutputFormat.class,org.apache.hadoop.io.compress.GzipCodec.class);
>
>
> If anyone has any tips for what I should look into it would be appreciated.
>
> Thanks.
>
> Darin.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>
*Arush Kharbanda* || Technical Teamlead
arush@sigmoidanalytics.com || www.sigmoidanalytics.com