You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Klaas Bosteels <kl...@ugent.be> on 2008/10/02 12:21:59 UTC
Re: Is there a way to pause a running hadoop job?
On Thu, Oct 2, 2008 at 2:44 AM, Steve Gao <st...@yahoo.com> wrote:
> I have 5 running jobs, each has 2 reducers. Because I set max number of reducers as 10 so any incoming job will be hold until some of the 5 jobs finish and release reducer quota.
>
> Now the problem is that an incoming job has a higher priority that I want to pause some of the 5 jobs, let the new job finish, and resume the old one.
>
> Is this doable in Hadoop? Thanks!
You could use the patch attached to this JIRA to do this:
https://issues.apache.org/jira/browse/HADOOP-3687
Since paused tasks are kept in memory there is a limit on how much you
can pause with this patch, but nevertheless is can be very useful in
practice.
-Klaas
Re: How to concatenate hadoop files to a single hadoop file
Posted by Francesco Salbaroli <fr...@ie.ibm.com>.
Maybe you can write a simple map/reduce task to do it.
Best Regards,
Francesco
-------------------------------------------------------------------------
Francesco Salbaroli
Stud./Intern @ Innovation Centre
IBM Technology Campus,
Damastown Industrial Estate, Mulhuddart, Dublin 15
E-mail: francesco.salbaroli(at)ie.ibm.com
Tel.: +353 01 815 5625
Graduate Student - Università di Bologna
E-mail: francesco.salbaroli(at)studio.unibo.it
-------------------------------------------------------------------------
Re: How to concatenate hadoop files to a single hadoop file
Posted by Michael Andrews <ma...@liveops.com>.
You might be able to use hars:
http://hadoop.apache.org/core/docs/current/hadoop_archives.html
On 10/2/08 2:51 PM, "Steve Gao" <st...@yahoo.com> wrote:
Anybody knows? Thanks a lot.
--- On Thu, 10/2/08, Steve Gao <st...@yahoo.com> wrote:
From: Steve Gao <st...@yahoo.com>
Subject: How to concatenate hadoop files to a single hadoop file
To: core-user@hadoop.apache.org
Cc: core-dev@hadoop.apache.org
Date: Thursday, October 2, 2008, 3:17 PM
Suppose I have 3 files in Hadoop that I want to "cat" them to a single
file. I know it can be done by "hadoop dfs -cat" to a local file and
updating it to Hadoop. But it's very expensive for large files. Is there an
internal way to do this in Hadoop itself? Thanks
Re: How to concatenate hadoop files to a single hadoop file
Posted by Steve Gao <st...@yahoo.com>.
Anybody knows? Thanks a lot.
--- On Thu, 10/2/08, Steve Gao <st...@yahoo.com> wrote:
From: Steve Gao <st...@yahoo.com>
Subject: How to concatenate hadoop files to a single hadoop file
To: core-user@hadoop.apache.org
Cc: core-dev@hadoop.apache.org
Date: Thursday, October 2, 2008, 3:17 PM
Suppose I have 3 files in Hadoop that I want to "cat" them to a single
file. I know it can be done by "hadoop dfs -cat" to a local file and
updating it to Hadoop. But it's very expensive for large files. Is there an
internal way to do this in Hadoop itself? Thanks
Re: How to concatenate hadoop files to a single hadoop file
Posted by Steve Gao <st...@yahoo.com>.
Anybody knows? Thanks a lot.
--- On Thu, 10/2/08, Steve Gao <st...@yahoo.com> wrote:
From: Steve Gao <st...@yahoo.com>
Subject: How to concatenate hadoop files to a single hadoop file
To: core-user@hadoop.apache.org
Cc: core-dev@hadoop.apache.org
Date: Thursday, October 2, 2008, 3:17 PM
Suppose I have 3 files in Hadoop that I want to "cat" them to a single
file. I know it can be done by "hadoop dfs -cat" to a local file and
updating it to Hadoop. But it's very expensive for large files. Is there an
internal way to do this in Hadoop itself? Thanks
How to concatenate hadoop files to a single hadoop file
Posted by Steve Gao <st...@yahoo.com>.
Suppose I have 3 files in Hadoop that I want to "cat" them to a single file. I know it can be done by "hadoop dfs -cat" to a local file and updating it to Hadoop. But it's very expensive for large files. Is there an internal way to do this in Hadoop itself? Thanks
How to concatenate hadoop files to a single hadoop file
Posted by Steve Gao <st...@yahoo.com>.
Suppose I have 3 files in Hadoop that I want to "cat" them to a single file. I know it can be done by "hadoop dfs -cat" to a local file and updating it to Hadoop. But it's very expensive for large files. Is there an internal way to do this in Hadoop itself? Thanks