You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sanchita Adhya <sa...@infocepts.com> on 2012/07/05 13:49:38 UTC

Error using MultipleInputs

Hi,

 

I am using cloudera's hadoop version - Hadoop 0.20.2-cdh3u3 and trying to
use the MultipleInputs incorporating separate mapper class in the following
manner-

 

public static void main(String[] args) throws Exception {

     JobConf conf = new JobConf(IntegrateExisting.class);

     conf.setJobName("IntegrateExisting");

 

     conf.setOutputKeyClass(Text.class);

     conf.setOutputValueClass(Text.class);

 

     Path existingKeysInputPath = new Path(args[0]);

     Path newKeysInputPath = new Path(args[1]);

    Path outputPath = new Path(args[2]);

 

     MultipleInputs.addInputPath(conf, existingKeysInputPath,
TextInputFormat.class, MapExisting.class);

     MultipleInputs.addInputPath(conf, newKeysInputPath,
TextInputFormat.class, MapNew.class);

    

     conf.setCombinerClass(ReduceAndFilterOut.class);

     conf.setReducerClass(ReduceAndFilterOut.class);

     

     conf.setInputFormat(TextInputFormat.class);

     conf.setOutputFormat(TextOutputFormat.class);

 

     FileOutputFormat.setOutputPath(conf, outputPath);

 

 

    //FileInputFormat.addInputPath(conf,existingKeysInputPath);

   //FileInputFormat.addInputPath(conf,newKeysInputPath);

 

     JobClient.runJob(conf);

   }

 

Without the commented lines in the above code, the MR job fails with the
following error-

 

12/07/05 16:59:25 ERROR security.UserGroupInformation:
PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException:
No input paths specified in job

Exception in thread "main" java.io.IOException: No input paths specified in
job

        at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:153
)

        at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:205)

        at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:971)

        at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:963)

        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)

        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)

        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1157)

        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)

        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)

        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)

        at org.myorg.IntegrateExisting.main(IntegrateExisting.java:122)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

 

Uncommenting the lines, leads to the following error in the mappers-

 

java.lang.ClassCastException: org.apache.hadoop.mapred.FileSplit cannot be
cast to org.apache.hadoop.mapred.lib.TaggedInputSplit

        at
org.apache.hadoop.mapred.lib.DelegatingMapper.map(DelegatingMapper.java:48)

        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)

        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1157)

        at org.apache.hadoop.mapred.Child.main(Child.java:264)

 

I see the MAPREDUCE-1178 that discusses the second error is included in the
CDH3 version. Is there any code missing from the above piece?

 

Thanks for the help. 

 

Regards,

Sanchita

 


RE: Error using MultipleInputs

Posted by Sanchita Adhya <sa...@infocepts.com>.
Thank you Bejoy for your prompt response! I have gotten past the error!

-----Original Message-----
From: Bejoy Ks [mailto:bejoy.hadoop@gmail.com] 
Sent: Thursday, July 05, 2012 5:39 PM
To: common-user@hadoop.apache.org
Subject: Re: Error using MultipleInputs

Hi Sanchita

Try your code after commenting the following Line of code,

//conf.setInputFormat(TextInputFormat.class);

AFAIK This explicitly sets the input format as TextInputFormat instead of
MultipleInput and hence the compiler throws an error stating 'no input path
specified'.

Regards
Bejoy KS

On Thu, Jul 5, 2012 at 5:19 PM, Sanchita Adhya <sa...@infocepts.com> wrote:
> Hi,
>
>
>
> I am using cloudera's hadoop version - Hadoop 0.20.2-cdh3u3 and trying 
> to use the MultipleInputs incorporating separate mapper class in the 
> following
> manner-
>
>
>
> public static void main(String[] args) throws Exception {
>
>      JobConf conf = new JobConf(IntegrateExisting.class);
>
>      conf.setJobName("IntegrateExisting");
>
>
>
>      conf.setOutputKeyClass(Text.class);
>
>      conf.setOutputValueClass(Text.class);
>
>
>
>      Path existingKeysInputPath = new Path(args[0]);
>
>      Path newKeysInputPath = new Path(args[1]);
>
>     Path outputPath = new Path(args[2]);
>
>
>
>      MultipleInputs.addInputPath(conf, existingKeysInputPath, 
> TextInputFormat.class, MapExisting.class);
>
>      MultipleInputs.addInputPath(conf, newKeysInputPath, 
> TextInputFormat.class, MapNew.class);
>
>
>
>      conf.setCombinerClass(ReduceAndFilterOut.class);
>
>      conf.setReducerClass(ReduceAndFilterOut.class);
>
>
>
>      conf.setInputFormat(TextInputFormat.class);
>
>      conf.setOutputFormat(TextOutputFormat.class);
>
>
>
>      FileOutputFormat.setOutputPath(conf, outputPath);
>
>
>
>
>
>     //FileInputFormat.addInputPath(conf,existingKeysInputPath);
>
>    //FileInputFormat.addInputPath(conf,newKeysInputPath);
>
>
>
>      JobClient.runJob(conf);
>
>    }
>
>
>
> Without the commented lines in the above code, the MR job fails with 
> the following error-
>
>
>
> 12/07/05 16:59:25 ERROR security.UserGroupInformation:
> PriviledgedActionException as:root (auth:SIMPLE)
cause:java.io.IOException:
> No input paths specified in job
>
> Exception in thread "main" java.io.IOException: No input paths 
> specified in job
>
>         at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.ja
> va:153
> )
>
>         at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.jav
> a:205)
>
>         at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:971)
>
>         at
> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:963)
>
>         at 
> org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>
>         at 
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
>
>         at 
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> ion.ja
> va:1157)
>
>         at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:83
> 3)
>
>         at 
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
>
>         at 
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
>
>         at 
> org.myorg.IntegrateExisting.main(IntegrateExisting.java:122)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39
> )
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl
> .java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
>
> Uncommenting the lines, leads to the following error in the mappers-
>
>
>
> java.lang.ClassCastException: org.apache.hadoop.mapred.FileSplit 
> cannot be cast to org.apache.hadoop.mapred.lib.TaggedInputSplit
>
>         at
> org.apache.hadoop.mapred.lib.DelegatingMapper.map(DelegatingMapper.jav
> a:48)
>
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>
>         at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
> ion.ja
> va:1157)
>
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>
>
>
> I see the MAPREDUCE-1178 that discusses the second error is included 
> in the
> CDH3 version. Is there any code missing from the above piece?
>
>
>
> Thanks for the help.
>
>
>
> Regards,
>
> Sanchita
>
>
>


Re: Error using MultipleInputs

Posted by Bejoy Ks <be...@gmail.com>.
Hi Sanchita

Try your code after commenting the following Line of code,

//conf.setInputFormat(TextInputFormat.class);

AFAIK This explicitly sets the input format as TextInputFormat instead
of MultipleInput and hence the compiler throws an error stating 'no
input path specified'.

Regards
Bejoy KS

On Thu, Jul 5, 2012 at 5:19 PM, Sanchita Adhya <sa...@infocepts.com> wrote:
> Hi,
>
>
>
> I am using cloudera's hadoop version - Hadoop 0.20.2-cdh3u3 and trying to
> use the MultipleInputs incorporating separate mapper class in the following
> manner-
>
>
>
> public static void main(String[] args) throws Exception {
>
>      JobConf conf = new JobConf(IntegrateExisting.class);
>
>      conf.setJobName("IntegrateExisting");
>
>
>
>      conf.setOutputKeyClass(Text.class);
>
>      conf.setOutputValueClass(Text.class);
>
>
>
>      Path existingKeysInputPath = new Path(args[0]);
>
>      Path newKeysInputPath = new Path(args[1]);
>
>     Path outputPath = new Path(args[2]);
>
>
>
>      MultipleInputs.addInputPath(conf, existingKeysInputPath,
> TextInputFormat.class, MapExisting.class);
>
>      MultipleInputs.addInputPath(conf, newKeysInputPath,
> TextInputFormat.class, MapNew.class);
>
>
>
>      conf.setCombinerClass(ReduceAndFilterOut.class);
>
>      conf.setReducerClass(ReduceAndFilterOut.class);
>
>
>
>      conf.setInputFormat(TextInputFormat.class);
>
>      conf.setOutputFormat(TextOutputFormat.class);
>
>
>
>      FileOutputFormat.setOutputPath(conf, outputPath);
>
>
>
>
>
>     //FileInputFormat.addInputPath(conf,existingKeysInputPath);
>
>    //FileInputFormat.addInputPath(conf,newKeysInputPath);
>
>
>
>      JobClient.runJob(conf);
>
>    }
>
>
>
> Without the commented lines in the above code, the MR job fails with the
> following error-
>
>
>
> 12/07/05 16:59:25 ERROR security.UserGroupInformation:
> PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException:
> No input paths specified in job
>
> Exception in thread "main" java.io.IOException: No input paths specified in
> job
>
>         at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:153
> )
>
>         at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:205)
>
>         at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:971)
>
>         at
> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:963)
>
>         at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
>
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1157)
>
>         at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
>
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
>
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1242)
>
>         at org.myorg.IntegrateExisting.main(IntegrateExisting.java:122)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
>
> Uncommenting the lines, leads to the following error in the mappers-
>
>
>
> java.lang.ClassCastException: org.apache.hadoop.mapred.FileSplit cannot be
> cast to org.apache.hadoop.mapred.lib.TaggedInputSplit
>
>         at
> org.apache.hadoop.mapred.lib.DelegatingMapper.map(DelegatingMapper.java:48)
>
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> va:1157)
>
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>
>
>
> I see the MAPREDUCE-1178 that discusses the second error is included in the
> CDH3 version. Is there any code missing from the above piece?
>
>
>
> Thanks for the help.
>
>
>
> Regards,
>
> Sanchita
>
>
>