You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Eric Yang <ey...@yahoo-inc.com> on 2010/01/01 00:21:33 UTC
Add user jars to mapreduce
Hi,
I have a mapreduce program embeded in a java application, and I am trying to
load additional jar files as add-on for the execution of my map reduce job.
My program works like this:
JobConf conf = new JobConf(new Configuration(), Demux.class);
conf.setBoolean("mapred.used.genericoptionsparser",true);
args[x]="-libjars"
args[x+1]="/path/to/addon.jar";
int res = ToolRunner.run(conf, new Demux(), args);
When the mapreduce job is running, the task failed because it is unable to
find the additional classes inside addon.jar. If I extract
/mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
like only the main jar file was in the job.jar. Does the addon.jar needs to
be preloaded into all task tracker node? Could the addon.jar get uploaded
via the client? If yes, how to do this properly?
I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
advance.
Regards,
Eric
Re: Add user jars to mapreduce
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
On 1/1/10 5:58 AM, "Eric Yang" <ey...@yahoo-inc.com> wrote:
Any idea? What are the difference between addArchiveToClassPath and
addFiletoClassPath?
If you add an archive using addArchiveToClassPath, it will be added to class path and will be unarchived.
If you add an archive using addFileToClassPath, it will be added to class path and wont be unarchived.
Regards,
Eric
On 12/31/09 3:25 PM, "Philip Zeyliger" <ph...@cloudera.com> wrote:
> You should use the DistributedCache.
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Distributed
> Cache
>
>
>
> On Thu, Dec 31, 2009 at 3:21 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> Hi,
>>
>> I have a mapreduce program embeded in a java application, and I am trying to
>> load additional jar files as add-on for the execution of my map reduce job.
>> My program works like this:
>>
>> JobConf conf = new JobConf(new Configuration(), Demux.class);
>> conf.setBoolean("mapred.used.genericoptionsparser",true);
>>
>> args[x]="-libjars"
>> args[x+1]="/path/to/addon.jar";
>>
>> int res = ToolRunner.run(conf, new Demux(), args);
>>
>> When the mapreduce job is running, the task failed because it is unable to
>> find the additional classes inside addon.jar. If I extract
>> /mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
>> like only the main jar file was in the job.jar. Does the addon.jar needs to
>> be preloaded into all task tracker node? Could the addon.jar get uploaded
>> via the client? If yes, how to do this properly?
>>
>> I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
>> advance.
>>
>> Regards,
>> Eric
>>
>
>
Re: Add user jars to mapreduce
Posted by Eric Yang <ey...@yahoo-inc.com>.
Hi Philip,
Thanks for the pointer, and I implemented
DistributedCache.addFileToClassPath like this for testing:
String parserPath = conf.get("chukwa.data.dir")+File.separator+"demux";
try {
FileSystem fs = FileSystem.get(new Configuration());
FileStatus[] fstatus = fs.listStatus(new Path(parserPath));
if(fstatus!=null) {
for(FileStatus parser : fstatus) {
DistributedCache.addFileToClassPath(parser.getPath(), conf);
}
}
} catch (IOException e) {
log.error(ExceptionUtil.getStackTrace(e));
}
In job.xml file, it shows:
mapred.job.classpath.files hdfs://abc.example.com/chukwa/demux/parsers.jar
mapred.cache.files hdfs://abc.example.com/chukwa/demux/parsers.jar
But the task still failed with:
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFacto
ry
The MapProcessorFactory is in parsers.jar file, and file permission are the
same for the running user and parsers.jar file.
Any idea? What are the difference between addArchiveToClassPath and
addFiletoClassPath?
Regards,
Eric
On 12/31/09 3:25 PM, "Philip Zeyliger" <ph...@cloudera.com> wrote:
> You should use the DistributedCache.
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Distributed
> Cache
>
>
>
> On Thu, Dec 31, 2009 at 3:21 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> Hi,
>>
>> I have a mapreduce program embeded in a java application, and I am trying to
>> load additional jar files as add-on for the execution of my map reduce job.
>> My program works like this:
>>
>> JobConf conf = new JobConf(new Configuration(), Demux.class);
>> conf.setBoolean("mapred.used.genericoptionsparser",true);
>>
>> args[x]="-libjars"
>> args[x+1]="/path/to/addon.jar";
>>
>> int res = ToolRunner.run(conf, new Demux(), args);
>>
>> When the mapreduce job is running, the task failed because it is unable to
>> find the additional classes inside addon.jar. If I extract
>> /mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
>> like only the main jar file was in the job.jar. Does the addon.jar needs to
>> be preloaded into all task tracker node? Could the addon.jar get uploaded
>> via the client? If yes, how to do this properly?
>>
>> I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
>> advance.
>>
>> Regards,
>> Eric
>>
>
>
Re: Add user jars to mapreduce
Posted by Philip Zeyliger <ph...@cloudera.com>.
You should use the DistributedCache.
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#DistributedCache
On Thu, Dec 31, 2009 at 3:21 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Hi,
>
> I have a mapreduce program embeded in a java application, and I am trying
> to
> load additional jar files as add-on for the execution of my map reduce job.
> My program works like this:
>
> JobConf conf = new JobConf(new Configuration(), Demux.class);
> conf.setBoolean("mapred.used.genericoptionsparser",true);
>
> args[x]="-libjars"
> args[x+1]="/path/to/addon.jar";
>
> int res = ToolRunner.run(conf, new Demux(), args);
>
> When the mapreduce job is running, the task failed because it is unable to
> find the additional classes inside addon.jar. If I extract
> /mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
> like only the main jar file was in the job.jar. Does the addon.jar needs
> to
> be preloaded into all task tracker node? Could the addon.jar get uploaded
> via the client? If yes, how to do this properly?
>
> I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
> advance.
>
> Regards,
> Eric
>
>
Re: Add user jars to mapreduce
Posted by Philip Zeyliger <ph...@cloudera.com>.
Visit the machine that's running the job. Every job has a working
directory, where the cache files should have been "localized". See if the
file is there.
You might also look at the logs of the TaskTracker, to see if it's throwing
any errors during the localization process.
-- Philip
On Mon, Jan 4, 2010 at 10:11 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>
>
> >> Your above code should work properly. Added jars will not be part of
> job.jar.
> >> The addon.jar will be uploaded by the client, the added libjars will be
> added
> >> to classpath using DistributedCache.
> >> So, above code should work unless your path to jar or the jar itself was
> >> wrong.
>
> When the path is wrong or jar is corrupted, there are exceptions thrown. I
> am not getting neither exception messages. Therefore, I assume the jar
> file
> is loaded as expected.
>
> I also tried another method by using DistributedCache syntax to add jar
> from
> HDFS to jobconf, and run the job. It didsn't work neither.
>
> Is there any procedure that I could follow to debug this further?
>
> Regards,
> Eric
>
>
Re: Add user jars to mapreduce
Posted by Eric Yang <ey...@yahoo-inc.com>.
>> Your above code should work properly. Added jars will not be part of job.jar.
>> The addon.jar will be uploaded by the client, the added libjars will be added
>> to classpath using DistributedCache.
>> So, above code should work unless your path to jar or the jar itself was
>> wrong.
When the path is wrong or jar is corrupted, there are exceptions thrown. I
am not getting neither exception messages. Therefore, I assume the jar file
is loaded as expected.
I also tried another method by using DistributedCache syntax to add jar from
HDFS to jobconf, and run the job. It didsn't work neither.
Is there any procedure that I could follow to debug this further?
Regards,
Eric
Re: Add user jars to mapreduce
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
On 1/1/10 4:51 AM, "Eric Yang" <ey...@yahoo-inc.com> wrote:
Hi,
I have a mapreduce program embeded in a java application, and I am trying to
load additional jar files as add-on for the execution of my map reduce job.
My program works like this:
JobConf conf = new JobConf(new Configuration(), Demux.class);
conf.setBoolean("mapred.used.genericoptionsparser",true);
args[x]="-libjars"
args[x+1]="/path/to/addon.jar";
int res = ToolRunner.run(conf, new Demux(), args);
When the mapreduce job is running, the task failed because it is unable to
find the additional classes inside addon.jar. If I extract
/mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
like only the main jar file was in the job.jar. Does the addon.jar needs to
be preloaded into all task tracker node? Could the addon.jar get uploaded
via the client? If yes, how to do this properly?
Your above code should work properly. Added jars will not be part of job.jar.
The addon.jar will be uploaded by the client, the added libjars will be added to classpath using DistributedCache.
So, above code should work unless your path to jar or the jar itself was wrong.
I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
advance.
Regards,
Eric