Re: joins in map reduce

[Jason Venner <ja...@attributor.com>]

I have just started to try using the Join operators.

The join I am trying is this;
join is 
outer(tbl(org.apache.hadoop.mapred.SequenceFileInputFormat,"Input1"),tbl(org.apache.hadoop.mapred.SequenceFileInputFormat,"IndexedTry1"))

but I get an error
08/06/30 08:55:13 INFO mapred.FileInputFormat: Total input paths to 
process : 10
Exception in thread "main" java.io.IOException: No input paths specified 
in input
    at 
org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputFormat.java:115)
    at org.apache.hadoop.mapred.join.Parser$WNode.getSplits(Parser.java:304)
    at org.apache.hadoop.mapred.join.Parser$CNode.getSplits(Parser.java:375)
    at 
org.apache.hadoop.mapred.join.CompositeInputFormat.getSplits(CompositeInputFormat.java:131)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:544)

I am clearly missing something basic...

        conf.setInputFormat(CompositeInputFormat.class);
        conf.setOutputPath( outputDirectory );
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(Text.class);
        conf.setOutputFormat(MapFileOutputFormat.class);
        conf.setMapperClass( LeftHandJoinMapper.class );
        conf.setReducerClass( IdentityReducer.class );
        conf.setNumReduceTasks(0);

        System.err.println( "join is " + 
CompositeInputFormat.compose("outer", SequenceFileInputFormat.class, 
allTables ) );
        conf.set("mapred.join.expr", 
CompositeInputFormat.compose("outer", SequenceFileInputFormat.class, 
allTables ));
       
        JobClient client = new JobClient();
       
        client.setConf( conf );

        RunningJob job = JobClient.runJob( conf );



Shirley Cohen wrote:
> Hi,
>
> How does one do a join operation in map reduce? Is there more than one 
> way to do a join? Which way works better and why?
>
> Thanks,
>
> Shirley
-- 
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if 
interested