You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Thomas FRIOL (JIRA)" <ji...@apache.org> on 2006/07/17 14:14:14 UTC
[jira] Created: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Should be able to specify more than one jar into a JobConf file
---------------------------------------------------------------
Key: HADOOP-366
URL: http://issues.apache.org/jira/browse/HADOOP-366
Project: Hadoop
Issue Type: Improvement
Components: mapred
Affects Versions: 0.4.0
Reporter: Thomas FRIOL
A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12421655 ]
Owen O'Malley commented on HADOOP-366:
--------------------------------------
There is functionality in the system that is close to this functionality. Specifically, you can wrap multiple jars into a single jar in the "lib" directory and they will be expanded. So if you have 3 jars map.jar, reduce.jar and format.jar, you could put them together into:
% mkdir lib
% mv map.jar reduce.jar format.jar lib
% jar uvf app.jar lib
That will make all 3 jars (and the rest of your jar) available to the framework. Does that meet your need?
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Jean-Baptiste Quenot (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12430995 ]
Jean-Baptiste Quenot commented on HADOOP-366:
---------------------------------------------
In fact Owen's point is to say that you can put jars in the jar.
Is there any reason for this "hack" to unpack mapred.jar in TaskRunner and look for a "lib" directory containing jars? Isn't that too much of magic?
Would it be a viable option to allow MapReduce tasks to take a list of jars instead? Or is there a compelling reason to only give a single Jar?
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "James P. White (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431629 ]
James P. White commented on HADOOP-366:
---------------------------------------
I don't think WAR layout really applies here, and I do think packaging jobs as a JAR make sense. WAR is for an application, not a library/fragment. The application packaging I'ld like to see for Hadoop is JNLP (Java Web Start).
For the situation where common jars are used by multiple job instances, the Extension mechanism supports job jars which are simply a configuration file and dependency(s) on application jars:
http://java.sun.com/j2se/1.4.2/docs/guide/extensions/spec.html#deployment
And as for job jars being executable or not, they certainly are executables from Hadoop's perspective. And while Hadoop packaging needs its own issue, I'll point out that putting the dependencies in the job jar to launch Hadoop for the job if it is executed might be a reasonable choice.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431614 ]
Doug Cutting commented on HADOOP-366:
-------------------------------------
> a Manifest in the JAR is th standard way of doing this
I'd guess there are far more war files with jars in lib directories in the world than there are executable jar files with external jars named in the manifest. The latter is much more fragile.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Thomas FRIOL (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12430959 ]
Thomas FRIOL commented on HADOOP-366:
-------------------------------------
Owen,
More than one month without news about this bug. What do you think about it ?
Thanks for your help.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Jean-Baptiste Quenot (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431619 ]
Jean-Baptiste Quenot commented on HADOOP-366:
---------------------------------------------
I hear you. But "lib" at the root is not standard WAR, is it?
The structure is:
{code}
WEB-INF/
|
\__ classes
|
\__ lib
| |
| \__ lib1.jar
| |
| \__ lib2.jar
|
\__ web.xml
{code}
And in web.xml we could define a webapp listener that would be invoked upon webapp startup.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431065 ]
Doug Cutting commented on HADOOP-366:
-------------------------------------
The rationale for keeping the config file separate from the jar was that folks might use a single big jar and then submit a sequence of jobs with different parameters, but all of whose code is in the same jar. The system could be smart about caching the jar on nodes. Rebuilding a jar for each job submitted seemed heavyweight to me. In fact, that's what this bug was originally complaining about!
The lib directory is like that in war files. So it too has a precedent in Java's standards!
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "James P. White (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431003 ]
James P. White commented on HADOOP-366:
---------------------------------------
A more comprehensive approach to the issue Thomas raises is to use a format that addresses all aspects of Java program execution dependency. The Java standard has two formats that address this in somewhat different ways:
JNLP (Java Network Launch Protocol)
http://java.sun.com/j2se/1.4.2/docs/guide/jws/index.html
Used by Java Web Start. Addresses all of the Java platform execution issues, including using native code.
But this is probably not the right level for packaing MapReduce modules, but rather should be used to package Hadoop itself.
Extensions
http://java.sun.com/j2se/1.4.2/docs/guide/extensions/index.html
This is exactly what Thomas is asking for, and provides the format for included jars and the setting of the classpath.
http://java.sun.com/j2se/1.4.2/docs/guide/extensions/spec.html
Bundled Optional Packages
The manifest for an application or optional package can specify one or more relative URLs referring to the JAR files and directories for the optional packages (and other libraries) that it needs. These relative URLs will be treated relative to the code base that the containing application or optional package JAR file was loaded from.
An application (or, more generally, JAR file) specifies the relative URLs of the optional packages (and libraries) that it needs via the manifest attribute Class-Path. This attribute lists the URLs to search for implementations of optional packages (or other libraries) if they cannot be found as optional packages installed on the host Java virtual machine*. These relative URLs may include JAR files and directories for any libraries or resources needed by the application or optional package. Relative URLs not ending with '/' are assumed to refer to JAR files. For example,
Class-Path: servlet.jar infobus.jar acme/beans.jar images/
Multiple Class-Path headers may be specified, and are combined sequentially.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Jean-Baptiste Quenot (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431526 ]
Jean-Baptiste Quenot commented on HADOOP-366:
---------------------------------------------
James you're right, using a Manifest in the JAR is th standard way of doing this, thanks for reminding. And about parameters, we could just pick the right config file in the JAR by passing a JVM argument:
java -Dconfig=mybigcluster -jar mapred.jar
Or even using command-line arguments:
java -jar mapred.jar mybigcluster
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Friol resolved HADOOP-366.
---------------------------------
Resolution: Fixed
Duplicate
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: https://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas Friol
> Assignee: Owen O'Malley
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431620 ]
Doug Cutting commented on HADOOP-366:
-------------------------------------
Sorry, I wasn't arguing that job jar's conform to the war file spec, rather that, if we're looking for sucessful methods of including libraries in jar files, the war file provides a good analogy, one that is more widely used than executable jars.
A job jar is not an executable jar. It is not standalone: it always gets the hadoop classes externaly, from the environment it is loaded into. In this way it is like a war file, which always gets servlet runtime from the servlet runner.
A job jar is not a war file: it doesn't specify servlets, include jsp pages, etc. The reason for the WEB-INF directory in a war file is to hide these files, so that, by default, files in the war are web pages, image files, etc. to be served, unless they're in WEB-INF.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "James P. White (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12431013 ]
James P. White commented on HADOOP-366:
---------------------------------------
I didn't finish that thought...
I also don't think jobs should have this "external-config-file-plus-jar-file" structure. The job should just be a jar file and the configuration contained within it.
There are several ways to do that. The one that is the least change to the current implementation is to simply put the JobConf file (sans the mapred.jar property) in the root of the jar file.
Putting a suitable property in the manifest to tag the jar as a Hadoop job file is a good idea too, although it then opens the question of what other properties might belong there as well.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-366) Should be able to specify more than
one jar into a JobConf file
Posted by "Thomas FRIOL (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-366?page=comments#action_12421664 ]
Thomas FRIOL commented on HADOOP-366:
-------------------------------------
Owen,
I think you did not understand correctly my explanations. I will try to be more precise.
In fact, when I am writting a new job using a jobConf.xml file, I am able to define a custom jar using the property 'mapred.jar'. This jar is the uploaded to the DFS and then appended to the Hadoop classpath of each task tracker.
The exact purpose of my problem is that today, we can specify only one jar and I think it could be great to be able to specify a comma separated list of jars so that we do not have to build a single jar with differents existing jars to give it to each Hadoop task tracker.
Am I more precise now ?
Thanks for your help.
> Should be able to specify more than one jar into a JobConf file
> ---------------------------------------------------------------
>
> Key: HADOOP-366
> URL: http://issues.apache.org/jira/browse/HADOOP-366
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.4.0
> Reporter: Thomas FRIOL
>
> A job should be able to specify more than one jar file into its JobConf file because sometimes custom Map and Reduce classes or just InputFormat classes uses objects coming from other jar files. For now, we have to build a unique jar to make Hadoop mapreduce operations works.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira