You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Benjamin Reed (JIRA)" <ji...@apache.org> on 2006/08/09 16:43:13 UTC
[jira] Created: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Encapsulating startup scripts and jars in a single Jar file.
------------------------------------------------------------
Key: HADOOP-435
URL: http://issues.apache.org/jira/browse/HADOOP-435
Project: Hadoop
Issue Type: New Feature
Affects Versions: 0.5.0
Reporter: Benjamin Reed
Attachments: hadoopit.patch
Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
Here are the available commands from hadoop.jar:
USAGE: hadoop [-l logdir] command
User commands:
dfs run a DFS admin client
jar run a JAR file
job manipulate MapReduce jobs
fsck run a DFS filesystem check utility
Runtime startup commands:
datanode run a DFS datanode
jobtracker run the MapReduce job Tracker node
namenode run the DFS namenode (namenode -format formats the FS)
tasktracker run a MapReduce task Tracker node
HadoopLoader commands:
buildJar builds the HadoopLoader jar file
conf dump hadoop configuration
Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491476 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
Yes, bin/hadoop should invoke HadoopExe. The big if/else in bin/hadoop is already done in HadoopExe as you point out. My point was just that we should do it as a separate issue so that we can get this patch applied.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: hadoop-exe.patch
Removed erroneous update of version
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-435:
--------------------------------
Fix Version/s: (was: 0.12.0)
Status: Open (was: Patch Available)
I think it could be useful to have an executable-jar-based version of Hadoop, and Java-based top-level command dispatch. However I have a few concerns with this patch as it stands.
1. I don't like the name 'HadoopIt'. I'd prefer something descriptive, like "hadoop-exe".
2. It would be best if this used ToolBase and CLI, as recommended for all Hadoop command-line programs, for command parsing, error reporting, etc.
3. There are some unexplained changes in StatusHttpServer.
4. The formatting is non-standard for Hadoop, using tabs instead of 2 spaces for indentation.
5. This doesn't completely replace the shell scripts. For example, when starting daemons, it doesn't check whether such a daemon is already running. It doesn't look for appropriate native libraries. Etc. Perhaps some of the script logic can be moved into Java, but this may be difficult or impossible in some cases. This suggests that instead we might consider moving as much as possible from scripts into Java (e.g., command dispatch) but not attempt to move everything. Other successful Java systems (e.g., Tomcat & Ant) use scripts to bootstrap, so I think that's an acceptable solution.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491421 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
My point is that we should really move forward on this issue. Once we have everything in a single jar we can deal with incorporating in the scripts as a separate issue. As it stands the patch gives you one single jar file to use as a library to build against and as a client to Hadoop. It also makes it simple to write your own startup scripts.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-435:
--------------------------------
Fix Version/s: (was: 0.12.1)
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-435:
-----------------------------
Attachment: hadoop-exe.patch
I have created a patch which addresses these points. I've also made lots of style changes to conform to the style guide better (ant checkstyle). I appreciate that maintaining a patch can be inconvenient, but it is important to get consensus before committing new features. Also, since this patch only adds a new class it is fairly resistant to invalidation by changes in trunk.
I'm afraid I've got some more questions/observations!
1. The code seems to set properties file "properties" twice.
2. How is streaming intended to be invoked since it isn't in the jar?
3. Shouldn't the jar command use org.apache.hadoop.util.RunJar? (Also, the logic in the main method of command doesn't need to check if jar is non-null.)
4. I think it would be better to have a "dump resource" command rather than the special case for an argument that begins "/".
5. Some commands are missing from the list: version, secondarynamenode.
6. Rather than having a "cmds" file, why not just allow users to specify other commands by fully-qualified name (like the hadoop script)? (Possible future enhancement.)
7. More generally, it would be good to have a strategy for how to move the scripts to use HadoopExe. (This would be another Jira issue.) Is this your aim?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489551 ]
Hadoop QA commented on HADOOP-435:
----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12355712/hadoop-exe.patch applied and successfully tested against trunk revision r529763.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/57/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/57/console
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12428000 ]
Yoram Arnon commented on HADOOP-435:
------------------------------------
any start/stop mechanism must allow separate control of DFS and map-reduce, and separate nodes for namenode, jobtracker, datanodes and tasktrackers.
jvm options, the pid matter etc. are also required.
A trivial method for starting up a toy hadoop cluster would be very welcome, lowering the bar for new users, but let's be careful before removing any functionality, or changing the behavior of the hadoop script.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: hadoop-exe.patch
I've incorporated Doug's comments. The patch to StatusHttpServer has been moved to HADOOP-1137, so that patch needs to be applied for this patch to work. I looked at using Toolbase, but this really is just a general startup class for other classes who parse their own arguments using Toolbase.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Benjamin Reed
> Attachments: hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12427636 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
I'm not suggesting that we use my scripts to replace the hadoop scripts, I'm just showing how things can be simplified in some cases. One of the daunting things to me (yes I'm simple minded) was getting hadoop setup and running on my little cluster. With the encapsulated jar file you can actually just copy the file into a shared directory and start everything up to at least try it. (It's also very nice to quickly test things.)
If you use the encapsulated jar file, the only thing you need to rsync is a single jar: hadoop.jar, so you simplify things.
The point of the scripts is to show that in some cases you may not need the web of scripts to manage hadoop. Especially for those just playing with hadoop or using some other process management system such as Condor. The encapsulated jar file allows for a convenient way to deploy a cluster: you plunk down the jar file and run it.
In the current distribution the only thing that would change would be the hadoop script, which would basically become a single line.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-435:
--------------------------------
Status: Open (was: Patch Available)
This patch is not currently ready to be committed, as it duplicates logic.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490966 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
1) Yes, that is silly. I was actually thinking of changing this so that we only set system properties that are not already set. That way we don't override things that are set with a -D one the command line.
2) Streaming is part of contrib. If you add the streaming classes to the jar file, streaming will work. That can be done after the build has run.
3) No, RunJar creates temp files. The nice thing about Hadoop-exe is that no temp files are created. (I agree you don't need to check for non-null.)
4) That's fine
5) This patch has been available since Aug of last year. I'm sure some more commands will appear before this ever gets committed :)
6) The cmds file allows you to have short names and descriptions. If you know the fully-qualified name, it's not clear to me why you would rather do "java -jar hadoop.jar org.apache.hadoop.ClassToRun" than "java -cp hadoop.jar org.apache.hadoop.ClassToRun". (We already support the latter :)
7) That is another issue and it is not my aim. As I mentioned in the description I use this patch along with extremely simple startup and shutdown scripts to manage our cluster, but I'm not proposing my scripts to be the standard ones.
BTW, where is the style guide?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491561 ]
Tom White commented on HADOOP-435:
----------------------------------
I like what this patch does, but I think the reluctance to commit it might stem from the fact that it is introducing duplication, coupled with the fact that there doesn't seem to be anyone who will remove this duplication in a future patch. I might be wrong on this, but this seems to be the perception at least.
As long as we have two independent ways of launching hadoop, the likelihood of divergence is high. For example, if this patch had been committed last August, it's unlikely that it would have been patched to include the version or secondarynamenode commands that were subsequently introduced.
So to move forward I would suggest we make bin/hadoop use HadoopExe and see it can be made to work. The difference between the implementations of the jar command (point 3 above) needs to be resolved first, I think.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Fix Version/s: 0.12.1
0.13.0
Affects Version/s: (was: 0.12.0)
0.12.1
Status: Patch Available (was: Open)
Patch from HADOOP-1137 is needed for this patch to work.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0, 0.12.1
>
> Attachments: hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Status: Patch Available (was: Open)
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Fix Version/s: 0.12.0
Affects Version/s: (was: 0.5.0)
0.12.0
Status: Patch Available (was: Open)
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Benjamin Reed
> Fix For: 0.12.0
>
> Attachments: hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-435:
-----------------------------
Attachment: hadoop-exe.patch
Here's another patch that addresses points 1, 3, 4 and 5.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490166 ]
Tom White commented on HADOOP-435:
----------------------------------
Looks like the wiki formatting caught me out. Point 2 should be
2. What is the use case for the {{-l}} command? It seems to sets a system property strangely: on line 222 cmd is {{-l}} so the log file begins {{hadoop--l}}. Is this intended?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490193 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
1) There are two special commands conf and setup. IMHO I think making them into Commands just adds bloat, but feel free to do it.
2) Yes that was dumb. It should be args[firstArgs++]
3) Agreed. I'll add.
4) I would hope that eventually conf.entries() will be fixed...
Your comments are great, but I wish you wouldn't make it a commit stopper. Maintaining a patch is a big pain!
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12427608 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
I think moving much of the logic from bin/hadoop into Java is a good idea. We still need scripts to manage daemons, but client operation would be simplified.
I don't see your start and stop scripts here.
Specifying a conf directory could be done by manipulating the class loader or classpath, rather than adding a lot of special case code.
Shouldn't we standardize on commons-cli for command-line processing? Shouldn't this somehow integrate with the Tool and ToolBase stuff?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-435:
-----------------------------
Comment: was deleted
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: hadoopit.patch
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=all ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: start.sh
This script starts up a datanode and tasktracker. If the config indicates that the node is the jobtracker, it also starts the jobtracker and namenode.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12428141 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
This does separate control of all the daemons just like the hadoop script. It encapsulates all the jar files, the configuration files (optionally), and most of the hadoop script into one self contained jar file. You still must start up each daemon separately. The command line arguments are the same as those for the hadoop script. No functionality is removed, I just added a bit more, like support for hadoopStreaming and the ability to dump the properties retrieved from the merged config files to standard out.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491659 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
Unjaring is gratuitous and annoying. There are environments, such as ours, where it is not practical for the classes to be extracted from the jar and put in the filesystem. This is especially the case when you are submitting jobs from your workstation. Java is built to be run from Jar files. (It's kind of what they are for...) Before HADOOP-1137, there was a single instance where StatusHttpServer assumed resources were in the file system and returned a file URL, which was silly since you can get a URL from the classloader to a resource. With HADOOP-1137 you can run all of hadoop out of a Jar file. Thus, using RunJar is not something we want to do.
Once this gets committed I would be happy to open the issue to patch bin/hadoop and I'll post the patch.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491361 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
bq. Tom: it would be good to have a strategy for how to move the scripts to use HadoopExe
bq. Ben: That is another issue and it is not my aim.
But we should, if possible, avoid replicating logic. Why can't this replace the command dispatch in bin/hadoop? It can't yet replace all of bin/hadoop, but those parts that it replicates it should be able to replace, no? Even if this is not Ben's aim, it is a goal of the project to not replicate logic.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12427621 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
> To start the cluster I simply do "for i in `cat machines'; do ssh $i:hadoop/start.sh [ ...]
Why is this better than the bin/slaves.sh script?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475522 ]
Hadoop QA commented on HADOOP-435:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12351921/hadoopit.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/511100. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Benjamin Reed
> Fix For: 0.12.0
>
> Attachments: hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491747 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
bq. Once this gets committed I would be happy to open the issue to patch bin/hadoop and I'll post the patch.
Again, why can't we patch bin/hadoop as a part of this patch? That would be preferable, as it would avoid duplication of logic.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-435:
-------------------------------
Status: Open (was: Patch Available)
Ben, this patch no longer applies to trunk. Looks like a small change is needed.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490168 ]
Tom White commented on HADOOP-435:
----------------------------------
Trying again! Point 2 should be (ignore the spaces afer dashes)
2. What is the use case for the {{- l}} command? It seems to sets a system property strangely: on line 222 cmd is {{- l}} so the log file begins {{hadoop- - l}}. Is this intended?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Resolved: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by Doug Cutting <cu...@apache.org>.
Nigel Daley wrote:
> Just expressing my disappointment that this patch wasn't applied as is
> and then another Jira opened to rework the scripts. Should have spoken
> up earlier.
Re-open it if you like. I don't see why we can't re-work the script as
a part of this issue. Otherwise we end up with duplicated logic with no
guarantee that it will ever be removed. The script change should be easy.
Doug
Re: [jira] Resolved: (HADOOP-435) Encapsulating startup scripts and jars in a single Jar file.
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Just expressing my disappointment that this patch wasn't applied as
is and then another Jira opened to rework the scripts. Should have
spoken up earlier.
On May 2, 2007, at 2:26 PM, Benjamin Reed (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HADOOP-435?
> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Benjamin Reed resolved HADOOP-435.
> ----------------------------------
>
> Resolution: Won't Fix
>
> The encapsulating Jar aspect doesn't seem to be an issue to most
> people. This should probably be reopened as an issue to redo the
> hadoop script since that seems to be the direction the issue is
> heading.
>
>> Encapsulating startup scripts and jars in a single Jar file.
>> ------------------------------------------------------------
>>
>> Key: HADOOP-435
>> URL: https://issues.apache.org/jira/browse/HADOOP-435
>> Project: Hadoop
>> Issue Type: New Feature
>> Affects Versions: 0.12.1
>> Reporter: Benjamin Reed
>> Fix For: 0.13.0
>>
>> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-
>> exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch,
>> hadoopit.patch, start.sh, stop.sh
>>
>>
>> Currently, hadoop is a set of scripts, configurations, and jar
>> files. It makes it a pain to install on compute and datanodes. It
>> also makes it a pain to setup clients so that they can use hadoop.
>> Everytime things are updated the pain begins again.
>> I suggest that we should be able to build a single Jar file that
>> has a Main-Class defined with the configuration built in so that
>> we can distribute that one file to nodes and clients on updates.
>> One nice thing that I haven't done would be to make the jarfile
>> downloadable from the JobTracker webpage so that clients can
>> easily submit the jobs.
>> I currently use such a setup on my small cluster. To start the job
>> tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to
>> submit a job I use "java -jar hadoop.jar jar wordcount.jar". I
>> used the client on my linux and Mac OSX machines and I'll I need
>> installed in java and the hadoop.jar file.
>> hadoop.jar helps with logfiles and configurations. The default of
>> pulling the config files from the jar file can be overridden by
>> specifying a config directory so that you can easily have machine
>> specific configs and still have the same hadoop.jar on all machines.
>> Here are the available commands from hadoop.jar:
>> USAGE: hadoop [-l logdir] command
>> User commands:
>> dfs run a DFS admin client
>> jar run a JAR file
>> job manipulate MapReduce jobs
>> fsck run a DFS filesystem check utility
>> Runtime startup commands:
>> datanode run a DFS datanode
>> jobtracker run the MapReduce job Tracker node
>> namenode run the DFS namenode (namenode -format formats
>> the FS)
>> tasktracker run a MapReduce task Tracker node
>> HadoopLoader commands:
>> buildJar builds the HadoopLoader jar file
>> conf dump hadoop configuration
>> Note, I don't have the classes for hadoop streaming built into
>> this Jar file, but if I had that would also be an option (it
>> checks for needed classes before displaying an option). It makes
>> it very easy for users that just write scripts to use hadoop
>> straight from their machines.
>> I'm also attaching the start.sh and stop.sh scripts that I use.
>> These are the only scripts I use to startup the daemons. They are
>> very simple and the start.sh script uses the config file to figure
>> out whether or not to start the jobtracker and the nameserver.
>> The attached patch adds the HadoopIt patch, modifies the
>> Configuration class to find the config files correctly, and
>> modifies the build to make a fully contained hadoop.jar. To update
>> the configuration in a hadoop.jar you simply use "zip hadoop.jar
>> hadoop-site.xml".
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Resolved: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed resolved HADOOP-435.
----------------------------------
Resolution: Won't Fix
The encapsulating Jar aspect doesn't seem to be an issue to most people. This should probably be reopened as an issue to redo the hadoop script since that seems to be the direction the issue is heading.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=all ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: stop.sh
Stops everything running against a jar file.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-435) Encapsulating startup scripts and jars
in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated HADOOP-435:
---------------------------------
Attachment: hadoopit.patch
Patch against version 10.1. Added target hadoopit that builds a self contained jar file with defined main class.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12427618 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
I've attached the scripts. They are quite convenient to use. To start the cluster I simply do "for i in `cat machines'; do ssh $i:hadoop/start.sh hadoop/hadoop.jar; done". I stop everything using "for i in `cat machines'; do ssh $i:hadoop/stop.sh hadoop/hadoop.jar; done"
The start script currently assumes that the jobtracker and namenode run on the same machine. The namenode should only be started if the machine is the namenode as indicated by the configuration just like jobtracker. There should also be a flag to indicate whether a datanode and tasktracker should be started if the host is a jobtracker or namenode. Despite the limitation, hopefully you get the point :)
The nice thing about using a conf dir is that you don't have to worry about complicated classpaths. I don't usually use it since I generally just use the config in the jar file, but there are cases where it is convenient. Fortunately, the code is structured such that the special case code is in a single method.
I'm not familiar enough with Tool and ToolBase to know how well it fits. From the brief look I took it seemed to address a more specific kind of application. (I wouldn't feel bad if someone made it use Tool and ToolBase :)
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491448 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
bq. Once we have everything in a single jar we can deal with incorporating in the scripts as a separate issue.
I'll repeat my question, stated more precisely: is there a reason why we can't or shouldn't switch bin/hadoop to invoke Java on the HadoopExe class, eliminating the big if/else there that's duplicated by the logic in HadoopExe.java. There's a strong reason to do this: replicated logic makes code hard to maintain. Is there a strong reason not to?
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476384 ]
Benjamin Reed commented on HADOOP-435:
--------------------------------------
1. I'm not stuck on the name. Anything is good.
2. Agreed.
3. Since everything runs out of the Jar file, we can't assume that resources are in files. I just removed that assumption from StatusHttpServer. All we need is a URL to resources in StatusHttpServer and getResource() gives us that.
4. Is there an eclipse code format template somewhere for Hadoop? I'm sure there are other formatting rules I'm missing.
5. I'm not looking to replace shell scripts, although it does effectively replace the hadoop script. It simply eases the task of managing the code. On our 100 node cluster we use this one jar file and two 10 line shell scripts to start and stop the cluster. It's also nice that we just have one jar file that we can code against, use to start up the cluster, and use to submit jobs. It's one file so it is easy to pass around, and we can be sure that everyone is using the same code.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-435?page=comments#action_12427620 ]
Doug Cutting commented on HADOOP-435:
-------------------------------------
Are you proposing that these should replace hadoop-daemon.sh? Your start/stop scripts lack many features of hadoop-daemon.sh, like using a pid file to keep from starting multiple daemons of the same type on a host, optionally rsyncing code from a master, specifying jvm options, etc.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: http://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.5.0
> Reporter: Benjamin Reed
> Attachments: hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-435) Encapsulating startup scripts and
jars in a single Jar file.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490165 ]
Tom White commented on HADOOP-435:
----------------------------------
I've looked at this patch in more detail and would suggest the following improvements before committing:
1. It would be much clearer if all the commands were implemented as Command objects. There are a number of special cases (e.g. conf and setup) that clutter the code.
2. What is the use case for the -l command? It seems to sets a system property strangely: on line 222 cmd is "-l" so the log file begins "hadoop--l". Is this intended?
3. Behaviour for commands that begin with a '/' seems to be undocumented in the help message.
4. Line 253:
Iterator<Map.Entry<String,String>> props = conf.entries();
produces an unchecked warning. I think we need a SuppressWarnings annotation in this case.
> Encapsulating startup scripts and jars in a single Jar file.
> ------------------------------------------------------------
>
> Key: HADOOP-435
> URL: https://issues.apache.org/jira/browse/HADOOP-435
> Project: Hadoop
> Issue Type: New Feature
> Affects Versions: 0.12.1
> Reporter: Benjamin Reed
> Fix For: 0.13.0
>
> Attachments: hadoop-exe.patch, hadoop-exe.patch, hadoopit.patch, hadoopit.patch, hadoopit.patch, start.sh, stop.sh
>
>
> Currently, hadoop is a set of scripts, configurations, and jar files. It makes it a pain to install on compute and datanodes. It also makes it a pain to setup clients so that they can use hadoop. Everytime things are updated the pain begins again.
> I suggest that we should be able to build a single Jar file that has a Main-Class defined with the configuration built in so that we can distribute that one file to nodes and clients on updates. One nice thing that I haven't done would be to make the jarfile downloadable from the JobTracker webpage so that clients can easily submit the jobs.
> I currently use such a setup on my small cluster. To start the job tracker I used "java -jar hadoop.jar -l /tmp/log jobtracker" to submit a job I use "java -jar hadoop.jar jar wordcount.jar". I used the client on my linux and Mac OSX machines and I'll I need installed in java and the hadoop.jar file.
> hadoop.jar helps with logfiles and configurations. The default of pulling the config files from the jar file can be overridden by specifying a config directory so that you can easily have machine specific configs and still have the same hadoop.jar on all machines.
> Here are the available commands from hadoop.jar:
> USAGE: hadoop [-l logdir] command
> User commands:
> dfs run a DFS admin client
> jar run a JAR file
> job manipulate MapReduce jobs
> fsck run a DFS filesystem check utility
> Runtime startup commands:
> datanode run a DFS datanode
> jobtracker run the MapReduce job Tracker node
> namenode run the DFS namenode (namenode -format formats the FS)
> tasktracker run a MapReduce task Tracker node
> HadoopLoader commands:
> buildJar builds the HadoopLoader jar file
> conf dump hadoop configuration
> Note, I don't have the classes for hadoop streaming built into this Jar file, but if I had that would also be an option (it checks for needed classes before displaying an option). It makes it very easy for users that just write scripts to use hadoop straight from their machines.
> I'm also attaching the start.sh and stop.sh scripts that I use. These are the only scripts I use to startup the daemons. They are very simple and the start.sh script uses the config file to figure out whether or not to start the jobtracker and the nameserver.
> The attached patch adds the HadoopIt patch, modifies the Configuration class to find the config files correctly, and modifies the build to make a fully contained hadoop.jar. To update the configuration in a hadoop.jar you simply use "zip hadoop.jar hadoop-site.xml".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.