You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by Tianying Chang <ty...@gmail.com> on 2014/05/08 02:46:29 UTC

using distcp v2 on mapreduce V1?

Hi,

I am trying to use the throttle feature of distcp V2. But we are not using
YARN, we are using cdh4.2, which is MapReduce V1.  I replaced the
hadoop-tool.jar to point to hadoop-distcp-2.0.3-alpha.jar. But it seems
still complain about NoSuchMethodError.

Anyone know if we can use distcp v2 without upgrading to YARN?

Thanks
Tian-Ying

Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(Lorg/apache/hadoop/mapreduce/Cluster;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/fs/Path;

at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:352)

at org.apache.hadoop.tools.DistCp.execute(DistCp.java:146)

at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.tools.DistCp.main(DistCp.java:374)

Re: using distcp v2 on mapreduce V1?

Posted by Ted Yu <yu...@gmail.com>.

Have you looked at MAPREDUCE-5081. Backport DistCpV2 ?

Cheers


On Wed, May 7, 2014 at 5:46 PM, Tianying Chang <ty...@gmail.com> wrote:

> Hi,
>
> I am trying to use the throttle feature of distcp V2. But we are not using
> YARN, we are using cdh4.2, which is MapReduce V1.  I replaced the
> hadoop-tool.jar to point to hadoop-distcp-2.0.3-alpha.jar. But it seems
> still complain about NoSuchMethodError.
>
> Anyone know if we can use distcp v2 without upgrading to YARN?
>
> Thanks
> Tian-Ying
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(Lorg/apache/hadoop/mapreduce/Cluster;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/fs/Path;
>
> at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:352)
>
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:146)
>
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:374)
>

Re: using distcp v2 on mapreduce V1?

Posted by Ted Yu <yu...@gmail.com>.

Have you looked at MAPREDUCE-5081. Backport DistCpV2 ?

Cheers


On Wed, May 7, 2014 at 5:46 PM, Tianying Chang <ty...@gmail.com> wrote:

> Hi,
>
> I am trying to use the throttle feature of distcp V2. But we are not using
> YARN, we are using cdh4.2, which is MapReduce V1.  I replaced the
> hadoop-tool.jar to point to hadoop-distcp-2.0.3-alpha.jar. But it seems
> still complain about NoSuchMethodError.
>
> Anyone know if we can use distcp v2 without upgrading to YARN?
>
> Thanks
> Tian-Ying
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(Lorg/apache/hadoop/mapreduce/Cluster;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/fs/Path;
>
> at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:352)
>
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:146)
>
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:374)
>

Re: using distcp v2 on mapreduce V1?

Posted by Ted Yu <yu...@gmail.com>.

Have you looked at MAPREDUCE-5081. Backport DistCpV2 ?

Cheers


On Wed, May 7, 2014 at 5:46 PM, Tianying Chang <ty...@gmail.com> wrote:

> Hi,
>
> I am trying to use the throttle feature of distcp V2. But we are not using
> YARN, we are using cdh4.2, which is MapReduce V1.  I replaced the
> hadoop-tool.jar to point to hadoop-distcp-2.0.3-alpha.jar. But it seems
> still complain about NoSuchMethodError.
>
> Anyone know if we can use distcp v2 without upgrading to YARN?
>
> Thanks
> Tian-Ying
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(Lorg/apache/hadoop/mapreduce/Cluster;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/fs/Path;
>
> at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:352)
>
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:146)
>
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:374)
>

Re: using distcp v2 on mapreduce V1?

Posted by Ted Yu <yu...@gmail.com>.

Have you looked at MAPREDUCE-5081. Backport DistCpV2 ?

Cheers


On Wed, May 7, 2014 at 5:46 PM, Tianying Chang <ty...@gmail.com> wrote:

> Hi,
>
> I am trying to use the throttle feature of distcp V2. But we are not using
> YARN, we are using cdh4.2, which is MapReduce V1.  I replaced the
> hadoop-tool.jar to point to hadoop-distcp-2.0.3-alpha.jar. But it seems
> still complain about NoSuchMethodError.
>
> Anyone know if we can use distcp v2 without upgrading to YARN?
>
> Thanks
> Tian-Ying
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(Lorg/apache/hadoop/mapreduce/Cluster;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/fs/Path;
>
> at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:352)
>
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:146)
>
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:374)
>