You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Alan Miller <Al...@synopsys.com> on 2012/07/04 17:49:46 UTC
MR job runs on CDH3 but not on CDH4
Hi,
I'm trying to move from CDH3U3 to CDH4.
My existing MR program works fine on CDH3U3 but I cant get it to run on CDH4.
Basically my Driver class
1. queries a PG DB and writes some HashMaps to files in the Distributed Cache,
2. then writes some Avro files (avro 1.7.0) to HDFS,
3. and then triggers a MRv1 job to process the Avro files.
The DC & Avro files get written so HDFS is working, but my job is not getting started.
I get an error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()
...
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
Seems the job doesn't even get accepted .
At MyDriver:397 (see below) I call job.submit, but that's it.
.....
12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012]
12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
at com.mycompany.MyDriver.run(MyDriver.java:397)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
at com.mycompany.MyDriver.main(MyDriver.java:1532)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Alan
Re: MR job runs on CDH3 but not on CDH4
Posted by Alan Miller <Al...@synopsys.com>.
Ok thanks, I'll post there.
I realized that the issue has to do with the extra jars that I added to the hadoop installation.
My job wasnt getting submitted becauae my tasktrackers don't seem to start if I have avro-1.7.0.jar & avro-tools-1.7.0.jar in my hadoop/lib directory.
But I need these jars in order to write the avro file.
Alan
Sent from my iPhone
On Jul 4, 2012, at 19:21, "Arun C Murthy" <ac...@hortonworks.com>> wrote:
It's hard for folks here to help you on CDH - please ask their own user lists.
Arun
On Jul 4, 2012, at 8:49 AM, Alan Miller wrote:
Hi,
I’m trying to move from CDH3U3 to CDH4.
My existing MR program works fine on CDH3U3 but I cant get it to run on CDH4.
Basically my Driver class
1. queries a PG DB and writes some HashMaps to files in the Distributed Cache,
2. then writes some Avro files (avro 1.7.0) to HDFS,
3. and then triggers a MRv1 job to process the Avro files.
The DC & Avro files get written so HDFS is working, but my job is not getting started.
I get an error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()
…
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
Seems the job doesn’t even get accepted .
At MyDriver:397 (see below) I call job.submit, but that’s it.
…..
12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012]
12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache.
12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
at com.mycompany.MyDriver.run(MyDriver.java:397)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
at com.mycompany.MyDriver.main(MyDriver.java:1532)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Alan
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: MR job runs on CDH3 but not on CDH4
Posted by Arun C Murthy <ac...@hortonworks.com>.
It's hard for folks here to help you on CDH - please ask their own user lists.
Arun
On Jul 4, 2012, at 8:49 AM, Alan Miller wrote:
> Hi,
>
> I’m trying to move from CDH3U3 to CDH4.
> My existing MR program works fine on CDH3U3 but I cant get it to run on CDH4.
>
> Basically my Driver class
> 1. queries a PG DB and writes some HashMaps to files in the Distributed Cache,
> 2. then writes some Avro files (avro 1.7.0) to HDFS,
> 3. and then triggers a MRv1 job to process the Avro files.
>
> The DC & Avro files get written so HDFS is working, but my job is not getting started.
> I get an error:
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy()
> …
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
>
> Seems the job doesn’t even get accepted .
> At MyDriver:397 (see below) I call job.submit, but that’s it.
>
> …..
> 12/07/04 08:16:54 INFO MyDriver: Processing data [my-data]
> 12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012]
> 12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s)
> 12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache.
> 12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list.
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol;
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
> at com.mycompany.MyDriver.run(MyDriver.java:397)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308)
> at com.mycompany.MyDriver.main(MyDriver.java:1532)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
> Alan
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/