You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jim Twensky <ji...@gmail.com> on 2009/12/28 18:14:04 UTC

Running Hadoop on demand

Hi,

I'd like to get Hadoop running on a large University cluster which is
used by many people to run different types of applications. We are
currently using Torque to assign nodes and manage the queue. What I
want to do is to enable people to request "n" processors, and
automatically start Hadoop on those "n" nodes where one of the nodes
becomes the name node and the rest become the workers. Once the job
completes, the name node should shut down the cluster and terminate
all instances of Hadoop processes.

I came across HOD (Hadoop On Demand), which seems to fit into my
needs. However, the most up to date documentation I could find was for
Hadoop 0.17.1 at:

http://hadoop.apache.org/common/docs/r0.17.1/hod.html

I'm wondering:

1- If I'm on the right track, ie. Can I use Hadoop for my needs or is
there any better alternative? We can't setup a dedicated Hadoop
cluster as there are other applications that need to be running on the
same cluster nodes.

2- If there is a more recent version of HOD that is able to work with
Hadoop 0.20.1 .

Thanks in advance,
Jim

Re: Running Hadoop on demand

Posted by Jim Twensky <ji...@gmail.com>.
Thanks for pointing that out Edward, I'll take a look at the most
recent documentation on HOD.

On Mon, Dec 28, 2009 at 7:39 PM, Edward Capriolo <ed...@gmail.com> wrote:
> Jim,
>
> Most components of hadoop now have a document branch per release, that
> documentation is inline and is generated by forest. Anything that you
> find not on the wiki, with a version number, should have a more up to
> date page.
>
> For example:
> http://hadoop.apache.org/common/
>
> Drop down 'Documentation'
> Release 0.18.3
> Release 0.19.2
> Release 0.20.1
>
> Usually you will find a fresher documentation page:
> http://hadoop.apache.org/common/docs/r0.20.1/hod_user_guide.html
>
>
> On Mon, Dec 28, 2009 at 12:14 PM, Jim Twensky <ji...@gmail.com> wrote:
>> Hi,
>>
>> I'd like to get Hadoop running on a large University cluster which is
>> used by many people to run different types of applications. We are
>> currently using Torque to assign nodes and manage the queue. What I
>> want to do is to enable people to request "n" processors, and
>> automatically start Hadoop on those "n" nodes where one of the nodes
>> becomes the name node and the rest become the workers. Once the job
>> completes, the name node should shut down the cluster and terminate
>> all instances of Hadoop processes.
>>
>> I came across HOD (Hadoop On Demand), which seems to fit into my
>> needs. However, the most up to date documentation I could find was for
>> Hadoop 0.17.1 at:
>>
>> http://hadoop.apache.org/common/docs/r0.17.1/hod.html
>>
>> I'm wondering:
>>
>> 1- If I'm on the right track, ie. Can I use Hadoop for my needs or is
>> there any better alternative? We can't setup a dedicated Hadoop
>> cluster as there are other applications that need to be running on the
>> same cluster nodes.
>>
>> 2- If there is a more recent version of HOD that is able to work with
>> Hadoop 0.20.1 .
>>
>> Thanks in advance,
>> Jim
>>
>

Re: Running Hadoop on demand

Posted by Edward Capriolo <ed...@gmail.com>.
Jim,

Most components of hadoop now have a document branch per release, that
documentation is inline and is generated by forest. Anything that you
find not on the wiki, with a version number, should have a more up to
date page.

For example:
http://hadoop.apache.org/common/

Drop down 'Documentation'
Release 0.18.3
Release 0.19.2
Release 0.20.1

Usually you will find a fresher documentation page:
http://hadoop.apache.org/common/docs/r0.20.1/hod_user_guide.html


On Mon, Dec 28, 2009 at 12:14 PM, Jim Twensky <ji...@gmail.com> wrote:
> Hi,
>
> I'd like to get Hadoop running on a large University cluster which is
> used by many people to run different types of applications. We are
> currently using Torque to assign nodes and manage the queue. What I
> want to do is to enable people to request "n" processors, and
> automatically start Hadoop on those "n" nodes where one of the nodes
> becomes the name node and the rest become the workers. Once the job
> completes, the name node should shut down the cluster and terminate
> all instances of Hadoop processes.
>
> I came across HOD (Hadoop On Demand), which seems to fit into my
> needs. However, the most up to date documentation I could find was for
> Hadoop 0.17.1 at:
>
> http://hadoop.apache.org/common/docs/r0.17.1/hod.html
>
> I'm wondering:
>
> 1- If I'm on the right track, ie. Can I use Hadoop for my needs or is
> there any better alternative? We can't setup a dedicated Hadoop
> cluster as there are other applications that need to be running on the
> same cluster nodes.
>
> 2- If there is a more recent version of HOD that is able to work with
> Hadoop 0.20.1 .
>
> Thanks in advance,
> Jim
>