You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Bing Jiang <ji...@gmail.com> on 2014/12/01 06:45:37 UTC
Re: Hadoop Terasort Benchmark Failure - Need Inputs
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782
I have some workaround from that jira.
-Bing
2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <as...@in.ibm.com>:
> Hi ,
>
> I am facing issue when i run teragen / terasort benchmark . Can someone
> suggest if you have also faced the same issue
>
> *Command Used*
> yarn jar
> /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> terasort input output
>
> *Exception*
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> mapred.compress.map.output is deprecated. Instead, use
> mapreduce.map.output.compress
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> min.num.spills.for.combine is deprecated. Instead, use
> mapreduce.map.combine.minspills
> 14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.IllegalArgumentException: can't read paritions file
> at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
> Caused by: java.io.FileNotFoundException: File _partition.lst does not
> exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
> at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
> at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
> ... 6 more
> 14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
> is deprecated. Instead, use mapreduce.job.end-notification.url
> 14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0%
> 14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
> 14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
> 14/11/29 07:03:11 INFO terasort.TeraSort: done
>
> *Investigations done so far*
>
> - thoroughly validated mapred-site.xml and am completely in sync with
> below recommendations
>
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
>
> - teragen executes successfully but terasort fails with above exception
> - some sites suggest that i should use the property "" in
> mapred-site.xml but looks like this property is no more valid .
> - Granted full access to hdfs directory
> hadoop fs -chmod -R 775 /
>
>
> Thanks and Regards,
> Ashish Kumar
Re: Hadoop Terasort Benchmark Failure - Need Inputs
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I suppose you are suggesting something like below which i tried and did
not help .
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort -Dmapreduce.totalorderpartitioner.path =_sortPartitioning input
output
Thanks
Ashish
From: Bing Jiang <ji...@gmail.com>
To: user@hadoop.apache.org
Date: 12/01/2014 11:13 AM
Subject: Re: Hadoop Terasort Benchmark Failure - Need Inputs
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782
I have some workaround from that jira.
-Bing
2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <as...@in.ibm.com>:
Hi ,
I am facing issue when i run teragen / terasort benchmark . Can someone
suggest if you have also faced the same issue
Command Used
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort input output
Exception
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.output.value.class is deprecated. Instead, use
mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.compress.map.output is deprecated. Instead, use
mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation:
min.num.spills.for.combine is deprecated. Instead, use
mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done
Investigations done so far
thoroughly validated mapred-site.xml and am completely in sync with below
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
teragen executes successfully but terasort fails with above exception
some sites suggest that i should use the property "" in mapred-site.xml
but looks like this property is no more valid .
Granted full access to hdfs directory
hadoop fs -chmod -R 775 /
Thanks and Regards,
Ashish Kumar
Re: Hadoop Terasort Benchmark Failure - Need Inputs
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I suppose you are suggesting something like below which i tried and did
not help .
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort -Dmapreduce.totalorderpartitioner.path =_sortPartitioning input
output
Thanks
Ashish
From: Bing Jiang <ji...@gmail.com>
To: user@hadoop.apache.org
Date: 12/01/2014 11:13 AM
Subject: Re: Hadoop Terasort Benchmark Failure - Need Inputs
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782
I have some workaround from that jira.
-Bing
2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <as...@in.ibm.com>:
Hi ,
I am facing issue when i run teragen / terasort benchmark . Can someone
suggest if you have also faced the same issue
Command Used
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort input output
Exception
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.output.value.class is deprecated. Instead, use
mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.compress.map.output is deprecated. Instead, use
mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation:
min.num.spills.for.combine is deprecated. Instead, use
mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done
Investigations done so far
thoroughly validated mapred-site.xml and am completely in sync with below
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
teragen executes successfully but terasort fails with above exception
some sites suggest that i should use the property "" in mapred-site.xml
but looks like this property is no more valid .
Granted full access to hdfs directory
hadoop fs -chmod -R 775 /
Thanks and Regards,
Ashish Kumar
Re: Hadoop Terasort Benchmark Failure - Need Inputs
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I suppose you are suggesting something like below which i tried and did
not help .
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort -Dmapreduce.totalorderpartitioner.path =_sortPartitioning input
output
Thanks
Ashish
From: Bing Jiang <ji...@gmail.com>
To: user@hadoop.apache.org
Date: 12/01/2014 11:13 AM
Subject: Re: Hadoop Terasort Benchmark Failure - Need Inputs
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782
I have some workaround from that jira.
-Bing
2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <as...@in.ibm.com>:
Hi ,
I am facing issue when i run teragen / terasort benchmark . Can someone
suggest if you have also faced the same issue
Command Used
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort input output
Exception
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.output.value.class is deprecated. Instead, use
mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.compress.map.output is deprecated. Instead, use
mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation:
min.num.spills.for.combine is deprecated. Instead, use
mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done
Investigations done so far
thoroughly validated mapred-site.xml and am completely in sync with below
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
teragen executes successfully but terasort fails with above exception
some sites suggest that i should use the property "" in mapred-site.xml
but looks like this property is no more valid .
Granted full access to hdfs directory
hadoop fs -chmod -R 775 /
Thanks and Regards,
Ashish Kumar
Re: Hadoop Terasort Benchmark Failure - Need Inputs
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I suppose you are suggesting something like below which i tried and did
not help .
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort -Dmapreduce.totalorderpartitioner.path =_sortPartitioning input
output
Thanks
Ashish
From: Bing Jiang <ji...@gmail.com>
To: user@hadoop.apache.org
Date: 12/01/2014 11:13 AM
Subject: Re: Hadoop Terasort Benchmark Failure - Need Inputs
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782
I have some workaround from that jira.
-Bing
2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <as...@in.ibm.com>:
Hi ,
I am facing issue when i run teragen / terasort benchmark . Can someone
suggest if you have also faced the same issue
Command Used
yarn jar
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
terasort input output
Exception
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.output.value.class is deprecated. Instead, use
mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation:
mapred.compress.map.output is deprecated. Instead, use
mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation:
min.num.spills.for.combine is deprecated. Instead, use
mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done
Investigations done so far
thoroughly validated mapred-site.xml and am completely in sync with below
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
teragen executes successfully but terasort fails with above exception
some sites suggest that i should use the property "" in mapred-site.xml
but looks like this property is no more valid .
Granted full access to hdfs directory
hadoop fs -chmod -R 775 /
Thanks and Regards,
Ashish Kumar