You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Riddle <dr...@gmail.com> on 2008/04/30 00:02:36 UTC
HOD and Condor
Hi,
I am trying to setup Hadoop on Demand on a cluster running Condor. I
downloaded hod-open-4.tar.gz from here:
http://issues.apache.org/jira/browse/HADOOP-1301
First of all I am not sure if this is the most recent version to use
or if this the appropriate place to retrieve it. The website for
Hadoop does not seem to have a download link for hod even though it
contains documentation for hod.
The version I download seems to have some support for condor but it is
not documented. I tryed to setup the hodrc as best I could but with
not much sucesses. I have included a copy of my config file below if
someone could help me to get hod working I would be grateful. When I
run the following command:
./hod -c ../conf/hodrc -m 3 -a "run dfs -ls /"
I recieve the following errors:
Failed to start hod shell: <type 'exceptions.AttributeError'>
NodePoolDesc instance has no attribute 'getPkgDirs'
Traceback (most recent call last):
File "hod", line 463, in <module>
exitStatus = main(hodConfig)
File "/home/ugrad/ddriddle/src/hod/hodlib/Hod/hod.py", line 120, in main
if np: np.finalize()
File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
73, in finalize
self.freeNodeSet(ns)
File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
64, in freeNodeSet
self._delNodeSet(nodeSet)
File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
219, in _delNodeSet
pkgdirs = npd.getPkgDirs()
AttributeError: NodePoolDesc instance has no attribute 'getPkgDirs'
Thanks,
David Riddle
# Template hodrc file
#
# -- hod shell specific configuration
[hod]
temp-dir = /tmp/hod-ddriddle
#java-home = ${JAVA_HOME}
java-home = /home/ugrad/ddriddle/lib/java
debug = 3
stream = True
non-interactive = false
allocate-timeout = 3600
#cluster = ${CLUSTER_NAME}
cluster = hal9000
#hadoop-install = ${HADOOP_HOME}
hadoop-install = /home/ugrad/ddriddle/src/hadoop-0.15.3
#log-dir = /var/hod/logs-${RM_USER}
#log-collection-dir = /var/hod/logs-${RM_USER}
#syslog-address = localhost:8814
#cluster-factor = 1.8
# -- resource manager configuration
[resource_manager]
#id = ${RM_ID}
id = condor
#pbs-user = ${RM_USER}
pbs-user = ddriddle
pbs-server = hal9000-server.cs.uiuc.edu
queue = parallel
batch-home = /home/hal/condor/condor
env-vars = HOD_PYTHON_HOME=/home/ugrad/ddriddle/bin
# -- serive_registry configuration
[service_registry]
debug = 2
http-address = :8002
xrs-address = :8879
#stream = False
#log-dir = /var/hod/logs-${RM_USER}
#xrs-port-range = 32768-65536
# -- ringmaster configuration
[ringmaster]
temp-dir = /tmp/hod-ddriddle
work-dirs =
/tmp/hod-ddriddle/1,/tmp/hod-ddriddle/2,/tmp/hod-ddriddle/3,/tmp/hod-ddriddle/4
register = true
xrs-port-range = 32768-65536
http-port-range = 8000-9000
debug = 2
#stream = False
#syslog-address = localhost:8814
#log-dir = /var/hod/logs-${RM_USER}
#svcrgy-addr = :8879
# -- hod ring configuration
[hodring]
temp-dir = /tmp/hod-ddriddle
register = true
#java-home = ${JAVA_HOME}
java-home = /home/ugrad/ddriddle/lib/java
debug = 2
xrs-port-range = 32768-65536
http-port-range = 8000-9000
#log-dir = /var/hod/logs-${RM_USER}
#log-destination-uri = local://var/log-${RM_USER}
#stream = false
#syslog-address = localhost:8814
#svcrgy-addr = :8879
[gridservice-mapred]
external = false
tracker_port = 8030
info_port = 50080
#pkgs = ${HADOOP_HOME}
pkgs = /home/ugrad/ddriddle/src/hadoop-0.15.3/
#host = localhost
[gridservice-hdfs]
external = false
fs_port = 8020
info_port = 50070
pkgs = /home/ugrad/ddriddle/src/hadoop-0.15.3/
#pkgs = ${HADOOP_HOME}
#host = localhost
--
Always code as if the (person) who ends up maintaining your code will
be a violent psychopath who knows where you live.
- M. Golding
Re: HOD and Condor
Posted by Rajiv Chittajallu <ra...@yahoo-inc.com>.
Hi David,
HOD is released as a contrrib in hadoop 0.16. The latest release is here
http://www.apache.org/dist/hadoop/core/hadoop-0.16.3/hadoop-0.16.3.tar.gz
Currently HOD has supports only torque as the resource manager.
-Rajiv
On 04/30/08 03:32, David Riddle wrote:
> Hi,
>
> I am trying to setup Hadoop on Demand on a cluster running Condor. I
> downloaded hod-open-4.tar.gz from here:
>
> http://issues.apache.org/jira/browse/HADOOP-1301
>
> First of all I am not sure if this is the most recent version to use
> or if this the appropriate place to retrieve it. The website for
> Hadoop does not seem to have a download link for hod even though it
> contains documentation for hod.
>
> The version I download seems to have some support for condor but it is
> not documented. I tryed to setup the hodrc as best I could but with
> not much sucesses. I have included a copy of my config file below if
> someone could help me to get hod working I would be grateful. When I
> run the following command:
>
> ./hod -c ../conf/hodrc -m 3 -a "run dfs -ls /"
>
> I recieve the following errors:
>
> Failed to start hod shell: <type 'exceptions.AttributeError'>
> NodePoolDesc instance has no attribute 'getPkgDirs'
>
>
> Traceback (most recent call last):
> File "hod", line 463, in <module>
> exitStatus = main(hodConfig)
> File "/home/ugrad/ddriddle/src/hod/hodlib/Hod/hod.py", line 120, in main
> if np: np.finalize()
> File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 73, in finalize
> self.freeNodeSet(ns)
> File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 64, in freeNodeSet
> self._delNodeSet(nodeSet)
> File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 219, in _delNodeSet
> pkgdirs = npd.getPkgDirs()
> AttributeError: NodePoolDesc instance has no attribute 'getPkgDirs'
>
>
> Thanks,
>
> David Riddle
>
> # Template hodrc file
> #
>
> # -- hod shell specific configuration
> [hod]
> temp-dir = /tmp/hod-ddriddle
> #java-home = ${JAVA_HOME}
> java-home = /home/ugrad/ddriddle/lib/java
> debug = 3
> stream = True
> non-interactive = false
> allocate-timeout = 3600
> #cluster = ${CLUSTER_NAME}
> cluster = hal9000
> #hadoop-install = ${HADOOP_HOME}
> hadoop-install = /home/ugrad/ddriddle/src/hadoop-0.15.3
> #log-dir = /var/hod/logs-${RM_USER}
> #log-collection-dir = /var/hod/logs-${RM_USER}
> #syslog-address = localhost:8814
> #cluster-factor = 1.8
>
> # -- resource manager configuration
> [resource_manager]
> #id = ${RM_ID}
> id = condor
> #pbs-user = ${RM_USER}
> pbs-user = ddriddle
> pbs-server = hal9000-server.cs.uiuc.edu
> queue = parallel
> batch-home = /home/hal/condor/condor
> env-vars = HOD_PYTHON_HOME=/home/ugrad/ddriddle/bin
>
> # -- serive_registry configuration
> [service_registry]
> debug = 2
> http-address = :8002
> xrs-address = :8879
> #stream = False
> #log-dir = /var/hod/logs-${RM_USER}
> #xrs-port-range = 32768-65536
>
> # -- ringmaster configuration
> [ringmaster]
> temp-dir = /tmp/hod-ddriddle
> work-dirs =
> /tmp/hod-ddriddle/1,/tmp/hod-ddriddle/2,/tmp/hod-ddriddle/3,/tmp/hod-ddriddle/4
> register = true
> xrs-port-range = 32768-65536
> http-port-range = 8000-9000
> debug = 2
> #stream = False
> #syslog-address = localhost:8814
> #log-dir = /var/hod/logs-${RM_USER}
> #svcrgy-addr = :8879
>
> # -- hod ring configuration
> [hodring]
> temp-dir = /tmp/hod-ddriddle
> register = true
> #java-home = ${JAVA_HOME}
> java-home = /home/ugrad/ddriddle/lib/java
> debug = 2
> xrs-port-range = 32768-65536
> http-port-range = 8000-9000
> #log-dir = /var/hod/logs-${RM_USER}
> #log-destination-uri = local://var/log-${RM_USER}
> #stream = false
> #syslog-address = localhost:8814
> #svcrgy-addr = :8879
>
> [gridservice-mapred]
> external = false
> tracker_port = 8030
> info_port = 50080
> #pkgs = ${HADOOP_HOME}
> pkgs = /home/ugrad/ddriddle/src/hadoop-0.15.3/
> #host = localhost
>
> [gridservice-hdfs]
> external = false
> fs_port = 8020
> info_port = 50070
> pkgs = /home/ugrad/ddriddle/src/hadoop-0.15.3/
> #pkgs = ${HADOOP_HOME}
> #host = localhost
>