You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Pete Wyckoff (JIRA)" <ji...@apache.org> on 2007/04/27 22:42:15 UTC
[jira] Created: (HADOOP-1301) resource management proviosioning for
Hadoop
resource management proviosioning for Hadoop
--------------------------------------------
Key: HADOOP-1301
URL: https://issues.apache.org/jira/browse/HADOOP-1301
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Reporter: Pete Wyckoff
Priority: Minor
The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Attachment: hod-hadoop.v2.patch
Fixed the ant build file adding a dependency on the package target to the compile target.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553608 ]
Hadoop QA commented on HADOOP-1301:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372008/hod-hadoop.v3.patch
against trunk revision r605811.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1403/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1403/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1403/console
This message is automatically generated.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1301:
----------------------------------
Component/s: (was: contrib/hbase)
mapred
I don't think that Hadoop On Demand (HOD) is related to HBase more than tangentially. (see the HBase page on the Hadoop Wiki (http://wiki.apache.org/lucene-hadoop/Hbase), so I am moving it to the map-reduce component which is where it appears to be more relevant.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555845#action_12555845 ]
Hadoop QA commented on HADOOP-1301:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372452/hod-hadoop.v4.patch
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1462/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1462/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1462/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1462/console
This message is automatically generated.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552201 ]
Hadoop QA commented on HADOOP-1301:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371682/hod-hadoop.patch
against trunk revision r604451.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to cause Findbugs to fail.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1360/testReport/
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1360/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1360/console
This message is automatically generated.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Attachment: hod-hadoop.v4.patch
This patch addresses the points Karam has raised as part of his review.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12492726 ]
Michael Bieniosek commented on HADOOP-1301:
-------------------------------------------
It would be nice to have some documentation about how to install this.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1301:
----------------------------------
Fix Version/s: 0.16.0
Priority: Major (was: Minor)
{blockquote}
The following files serve as the documentation for this patch:
README: Gives an overview of HOD
getting_started.txt: Gives instructions on how to try out HOD
config.txt: A more detailed description of how to configure HOD.
{blockquote}
Hemanth, could you please make them Apache Forrest based documentation? We could then put these up on the website etc.
Thanks!
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552216 ]
Hemanth Yamijala commented on HADOOP-1301:
------------------------------------------
Found the problem why the core tests and findbugs are failing. The hod build file needs to be modified. Working on a new patch.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-1301) resource management
proviosioning for Hadoop
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552406 ]
acmurthy edited comment on HADOOP-1301 at 12/17/07 5:30 AM:
-----------------------------------------------------------------
{quote}
The following files serve as the documentation for this patch:
README: Gives an overview of HOD
getting_started.txt: Gives instructions on how to try out HOD
config.txt: A more detailed description of how to configure HOD.
{quote}
Hemanth, could you please make them Apache Forrest based documentation? We could then put these up on the website etc.
Thanks!
was (Author: acmurthy):
{blockquote}
The following files serve as the documentation for this patch:
README: Gives an overview of HOD
getting_started.txt: Gives instructions on how to try out HOD
config.txt: A more detailed description of how to configure HOD.
{blockquote}
Hemanth, could you please make them Apache Forrest based documentation? We could then put these up on the website etc.
Thanks!
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555541#action_12555541 ]
Karam Singh commented on HADOOP-1301:
-------------------------------------
Ran hod as per the instructions provided in getting_started.txt
1.
Under section "3. Setting up HOD"
It is writeten that -:
* HOD is available under the 'contrib' section of Hadoop under the root
directory 'hod'.
* Distribute the files under this directory to all the nodes in the
cluster.
It would be better if it also mentions that "hod should be installed on same locations on all nodes"
2.
For multi-valued comma separated parameter e.g "--ringmaster.work-dirs" it should be mentioned that there should be no space between commas
3. Config.txt should also mentions the options mentioned in getting_started.txt
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Affects Version/s: 0.16.0
Status: Patch Available (was: Open)
Ran tests and findbugs - just to make sure - they don't crib about HOD. Seem to be ignoring the new directory for the most part.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Attachment: hod-open-4.tar.gz
This is the latest work-in-progress version of HOD. This version provides out-of-the-box integration with the Torque resource manager. In the tarball, you can find the following documentation to help install, configure and run HOD:
- README: Brief description of HOD.
- getting_started.txt: Brief instructions to quickly get you started on using HOD. It covers installation, basic configuration and commands.
- config.txt: More details on various important configuration options.
Appreciate any comments if you can try this out.
Plese note though, that some significant changes (particularly to the user interface) are in the works and might obsolete this version.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555857#action_12555857 ]
Hemanth Yamijala commented on HADOOP-1301:
------------------------------------------
JUnit test cases in HBase seem to have failed. As HOD is independent from that, it is unlikely to have caused the problem.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Attachment: hod-hadoop.patch
The attached patch is the latest version of HOD, which we would like to submit under hadoop contrib. This version of HOD works with the Hadoop 0.16 trunk.
The following files serve as the documentation for this patch:
README: Gives an overview of HOD
getting_started.txt: Gives instructions on how to try out HOD
config.txt: A more detailed description of how to configure HOD.
We request interested users to give it a try and provide us feedback.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (HADOOP-1301) resource management proviosioning for Hadoop
Posted by Nigel Daley <nd...@yahoo-inc.com>.
With this new contrib component, I have added contrib/hod as a Jira
component.
On Jan 4, 2008, at 10:23 AM, Nigel Daley (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HADOOP-1301?
> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Nigel Daley updated HADOOP-1301:
> --------------------------------
>
> Resolution: Fixed
> Status: Resolved (was: Patch Available)
>
>> resource management proviosioning for Hadoop
>> --------------------------------------------
>>
>> Key: HADOOP-1301
>> URL: https://issues.apache.org/jira/browse/
>> HADOOP-1301
>> Project: Hadoop
>> Issue Type: New Feature
>> Components: mapred
>> Affects Versions: 0.16.0
>> Reporter: Pete Wyckoff
>> Assignee: Hemanth Yamijala
>> Fix For: 0.16.0
>>
>> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-
>> hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.
>> 0.2.2.tar.gz
>>
>>
>> The Hadoop On Demand (HOD) project addresses the provisioning and
>> managing of MapReduce instances on cluster resources. With HOD,
>> the MapReduce user interacts with the cluster solely through a
>> self-service interface and the JT, TT info ports. The user never
>> needs to log into the cluster or even have an account on the
>> cluster for that matter. HOD allocates nodes, provisions MapReduce
>> (and optionally HDFS) on the cluster and when the user is done
>> with MapReduce jobs, cleanly shuts down MapReduce and de-allocates
>> the nodes (i.e., re-introducing them to the pool of available
>> resources in the cluster).
>> Using HOD, a cluster can be shared among different users in a fair
>> and efficient manner. HOD is not a replacement or re-
>> implementation of a traditional resource manager. HOD is
>> implemented using the resource manager paradigm and at present is
>> envisioned supporting Torque and Condor out of the box. It also
>> supports "static" resources, i.e., a dedicated set of resources
>> not using a resource manager.
>> HOD is also self provisioning and, thus, can be used on systems
>> such as EC2 or a campus cluster not already running MapReduce
>> software or a resouce manager. Figure 1 depicts a cluster using
>> HOD. As the figure shows, the user never logs into the cluster
>> itself. The user's jobs run as the 'hod' user (a configurable unix
>> id).
>> The user interacts with MapReduce and the cluster using the hod
>> shell, hodsh. Once in the hodsh, the user can allocate/de-allocate
>> nodes and automatically run JT, TTs, NN, DNs on those nodes
>> without knowing the specifics of which nodes are running which or
>> logging into any of those boxes. HOD transparently masks failures
>> by allocating nodes to replace failed nodes. Once the user has
>> allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/
>> MapReduce my2.jar ... from within the hod shell which
>> automatically generates the configuration file for the MapReduce
>> script. When done, the user will exit the shell.
>> The hod shell has an automatic timeout so that users cannot hog
>> resources they aren't using. The timeout applies only when there
>> is no MapReduce job running. In addition, hod also has the option
>> of tracking and enforcing user/group resource limits.
>> Optionally, HOD can run dedicated log and directory services in
>> the cluster. The log services are a central repository for
>> collecting and retrieving Hadoop logs for any given job. The
>> directory service provides an easy way to inspect what's running
>> in the cluster or for the end user and html interfacing for
>> getting to their JT and TT info ports.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-1301:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555587#action_12555587 ]
Karam Singh commented on HADOOP-1301:
-------------------------------------
+1 overall
+1 Document getting_started.txt : it now mentions that installation path for hod should be same on all nodes
+1 Document config.txt : Mentions that special character such space, comma are not handled by hod as known issue
+1 Document config.txt : Options mentioned in getting_started.txt are also present in config.txt
+1 Default conf/hodrc : Space between comma separated values for --ringmaster.work-dirs has been removed
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Status: Patch Available (was: In Progress)
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1301) resource management proviosioning
for Hadoop
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556009#action_12556009 ]
Nigel Daley commented on HADOOP-1301:
-------------------------------------
I just committed this. Thanks Hemanth and Pete!
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-hadoop.v4.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Status: Patch Available (was: In Progress)
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Assignee: Hemanth Yamijala
Status: In Progress (was: Patch Available)
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Priority: Minor
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Attachment: hod-hadoop.v3.patch
Patch addressing Arun's comment. This one has documentation in Forrest format. But the original text files with the documentation are also there, so one can start looking at them without having to build the Forrest based documentation (as there is nothing existing right now).
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pete Wyckoff updated HADOOP-1301:
---------------------------------
Attachment: hod.0.2.2.tar.gz
The python code that implements HOD.
Note there is a .hodrc file in the conf directory that should be copied to your home directory.
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: Pete Wyckoff
> Priority: Minor
> Attachments: hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1301) resource management proviosioning for
Hadoop
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-1301:
-------------------------------------
Status: In Progress (was: Patch Available)
> resource management proviosioning for Hadoop
> --------------------------------------------
>
> Key: HADOOP-1301
> URL: https://issues.apache.org/jira/browse/HADOOP-1301
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Pete Wyckoff
> Assignee: Hemanth Yamijala
> Fix For: 0.16.0
>
> Attachments: hod-hadoop.patch, hod-hadoop.v2.patch, hod-hadoop.v3.patch, hod-open-4.tar.gz, hod.0.2.2.tar.gz
>
>
> The Hadoop On Demand (HOD) project addresses the provisioning and managing of MapReduce instances on cluster resources. With HOD, the MapReduce user interacts with the cluster solely through a self-service interface and the JT, TT info ports. The user never needs to log into the cluster or even have an account on the cluster for that matter. HOD allocates nodes, provisions MapReduce (and optionally HDFS) on the cluster and when the user is done with MapReduce jobs, cleanly shuts down MapReduce and de-allocates the nodes (i.e., re-introducing them to the pool of available resources in the cluster).
> Using HOD, a cluster can be shared among different users in a fair and efficient manner. HOD is not a replacement or re-implementation of a traditional resource manager. HOD is implemented using the resource manager paradigm and at present is envisioned supporting Torque and Condor out of the box. It also supports "static" resources, i.e., a dedicated set of resources not using a resource manager.
> HOD is also self provisioning and, thus, can be used on systems such as EC2 or a campus cluster not already running MapReduce software or a resouce manager. Figure 1 depicts a cluster using HOD. As the figure shows, the user never logs into the cluster itself. The user's jobs run as the 'hod' user (a configurable unix id).
> The user interacts with MapReduce and the cluster using the hod shell, hodsh. Once in the hodsh, the user can allocate/de-allocate nodes and automatically run JT, TTs, NN, DNs on those nodes without knowing the specifics of which nodes are running which or logging into any of those boxes. HOD transparently masks failures by allocating nodes to replace failed nodes. Once the user has allocated nodes, she can run /bin/MapReduce my1.jar and then /bin/MapReduce my2.jar ... from within the hod shell which automatically generates the configuration file for the MapReduce script. When done, the user will exit the shell.
> The hod shell has an automatic timeout so that users cannot hog resources they aren't using. The timeout applies only when there is no MapReduce job running. In addition, hod also has the option of tracking and enforcing user/group resource limits.
> Optionally, HOD can run dedicated log and directory services in the cluster. The log services are a central repository for collecting and retrieving Hadoop logs for any given job. The directory service provides an easy way to inspect what's running in the cluster or for the end user and html interfacing for getting to their JT and TT info ports.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.