You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Paolo Castagna <ca...@googlemail.com> on 2011/06/28 18:42:31 UTC

Sequence of MapReduce jobs... is it possible to upload the jar only once?

Hi,
I have a sequence of three MapReduce jobs which I run this way:

  FirstDriver first = new FirstDriver(configuration);
  first.run(new String[] { args[0], args[1] + "_1" });
  SecondDriver second = new SecondDriver(configuration);
  second.run(new String[] { args[1] + "_1", args[1] + "_2" });
  ThirdDriver third = new ThirdDriver(configuration);
  third.run(new String[] { args[1] + "_2", args[1] + "_3" });

Each {First|Second|Third}Driver extends Configured and implements Tool.

My jar has quite a few dependencies, is there a way to upload to HDFS
the jar only once?

Thank you,
Paolo