You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Michael Parker <mi...@gmail.com> on 2012/08/19 18:46:15 UTC
Two-stage MR not terminating gracefully with JobControl?
Hi all,
I have a two-stage MR, written using entirely the new API, that I'm
running using Hadoop 1.0.3. In the end it generates a file named
part-r-0000 with the correct output, but the main method doesn't
terminate, despite me using the allFinished method of JobControl as I
thought I was supposed to. Any help would be greatly appreciated. A
snippet from my code follows.
>>>
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
...
public static void main(String[] args) throws IOException {
...
Job job1 = new Job(new Configuration(conf), "job1");
...
ControlledJob controlledJob1 = new ControlledJob(job1,
Collections.<ControlledJob>emptyList());
Job job2 = new Job(new Configuration(conf), "job2");
...
ControlledJob controlledJob2 = new ControlledJob(job2,
Arrays.asList(controlledJob1));
JobControl jobControl = new JobControl("main");
jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
Thread t = new Thread(jobControl);
t.start();
while (!jobControl.allFinished()) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// Ignore.
}
}
}
<<<
Thanks!
- Mike
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Oh it's _THAT_ thread. Wow, I'm embarrassed. Thanks for the help!
- Mike
On Mon, Aug 20, 2012 at 9:55 AM, Harsh J <ha...@cloudera.com> wrote:
> Mike,
>
> Perhaps t.setDaemon(true);?
>
> On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hello,
>>
>> So after adding a few print statements, I've found that the "while
>> (!jobControl.allFinished())" loop actually terminates, and so there
>> must be a non-daemon thread somewhere that is preventing the program
>> from terminating. Am I missing a configuration option to specify this?
>> Any ideas where it might be?
>>
>> Thanks!
>>
>> - Mike
>>
>>
>> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
>> <mi...@gmail.com> wrote:
>>> Hi all,
>>>
>>> I have a two-stage MR, written using entirely the new API, that I'm
>>> running using Hadoop 1.0.3. In the end it generates a file named
>>> part-r-0000 with the correct output, but the main method doesn't
>>> terminate, despite me using the allFinished method of JobControl as I
>>> thought I was supposed to. Any help would be greatly appreciated. A
>>> snippet from my code follows.
>>>
>>>>>>
>>>
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>>
>>> ...
>>>
>>> public static void main(String[] args) throws IOException {
>>> ...
>>>
>>> Job job1 = new Job(new Configuration(conf), "job1");
>>> ...
>>> ControlledJob controlledJob1 = new ControlledJob(job1,
>>> Collections.<ControlledJob>emptyList());
>>>
>>> Job job2 = new Job(new Configuration(conf), "job2");
>>> ...
>>> ControlledJob controlledJob2 = new ControlledJob(job2,
>>> Arrays.asList(controlledJob1));
>>>
>>> JobControl jobControl = new JobControl("main");
>>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>>> Thread t = new Thread(jobControl);
>>> t.start();
>>>
>>> while (!jobControl.allFinished()) {
>>> try {
>>> Thread.sleep(1000);
>>> } catch (InterruptedException e) {
>>> // Ignore.
>>> }
>>> }
>>> }
>>>
>>> <<<
>>>
>>> Thanks!
>>>
>>> - Mike
>
>
>
> --
> Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Oh it's _THAT_ thread. Wow, I'm embarrassed. Thanks for the help!
- Mike
On Mon, Aug 20, 2012 at 9:55 AM, Harsh J <ha...@cloudera.com> wrote:
> Mike,
>
> Perhaps t.setDaemon(true);?
>
> On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hello,
>>
>> So after adding a few print statements, I've found that the "while
>> (!jobControl.allFinished())" loop actually terminates, and so there
>> must be a non-daemon thread somewhere that is preventing the program
>> from terminating. Am I missing a configuration option to specify this?
>> Any ideas where it might be?
>>
>> Thanks!
>>
>> - Mike
>>
>>
>> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
>> <mi...@gmail.com> wrote:
>>> Hi all,
>>>
>>> I have a two-stage MR, written using entirely the new API, that I'm
>>> running using Hadoop 1.0.3. In the end it generates a file named
>>> part-r-0000 with the correct output, but the main method doesn't
>>> terminate, despite me using the allFinished method of JobControl as I
>>> thought I was supposed to. Any help would be greatly appreciated. A
>>> snippet from my code follows.
>>>
>>>>>>
>>>
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>>
>>> ...
>>>
>>> public static void main(String[] args) throws IOException {
>>> ...
>>>
>>> Job job1 = new Job(new Configuration(conf), "job1");
>>> ...
>>> ControlledJob controlledJob1 = new ControlledJob(job1,
>>> Collections.<ControlledJob>emptyList());
>>>
>>> Job job2 = new Job(new Configuration(conf), "job2");
>>> ...
>>> ControlledJob controlledJob2 = new ControlledJob(job2,
>>> Arrays.asList(controlledJob1));
>>>
>>> JobControl jobControl = new JobControl("main");
>>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>>> Thread t = new Thread(jobControl);
>>> t.start();
>>>
>>> while (!jobControl.allFinished()) {
>>> try {
>>> Thread.sleep(1000);
>>> } catch (InterruptedException e) {
>>> // Ignore.
>>> }
>>> }
>>> }
>>>
>>> <<<
>>>
>>> Thanks!
>>>
>>> - Mike
>
>
>
> --
> Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Oh it's _THAT_ thread. Wow, I'm embarrassed. Thanks for the help!
- Mike
On Mon, Aug 20, 2012 at 9:55 AM, Harsh J <ha...@cloudera.com> wrote:
> Mike,
>
> Perhaps t.setDaemon(true);?
>
> On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hello,
>>
>> So after adding a few print statements, I've found that the "while
>> (!jobControl.allFinished())" loop actually terminates, and so there
>> must be a non-daemon thread somewhere that is preventing the program
>> from terminating. Am I missing a configuration option to specify this?
>> Any ideas where it might be?
>>
>> Thanks!
>>
>> - Mike
>>
>>
>> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
>> <mi...@gmail.com> wrote:
>>> Hi all,
>>>
>>> I have a two-stage MR, written using entirely the new API, that I'm
>>> running using Hadoop 1.0.3. In the end it generates a file named
>>> part-r-0000 with the correct output, but the main method doesn't
>>> terminate, despite me using the allFinished method of JobControl as I
>>> thought I was supposed to. Any help would be greatly appreciated. A
>>> snippet from my code follows.
>>>
>>>>>>
>>>
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>>
>>> ...
>>>
>>> public static void main(String[] args) throws IOException {
>>> ...
>>>
>>> Job job1 = new Job(new Configuration(conf), "job1");
>>> ...
>>> ControlledJob controlledJob1 = new ControlledJob(job1,
>>> Collections.<ControlledJob>emptyList());
>>>
>>> Job job2 = new Job(new Configuration(conf), "job2");
>>> ...
>>> ControlledJob controlledJob2 = new ControlledJob(job2,
>>> Arrays.asList(controlledJob1));
>>>
>>> JobControl jobControl = new JobControl("main");
>>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>>> Thread t = new Thread(jobControl);
>>> t.start();
>>>
>>> while (!jobControl.allFinished()) {
>>> try {
>>> Thread.sleep(1000);
>>> } catch (InterruptedException e) {
>>> // Ignore.
>>> }
>>> }
>>> }
>>>
>>> <<<
>>>
>>> Thanks!
>>>
>>> - Mike
>
>
>
> --
> Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Oh it's _THAT_ thread. Wow, I'm embarrassed. Thanks for the help!
- Mike
On Mon, Aug 20, 2012 at 9:55 AM, Harsh J <ha...@cloudera.com> wrote:
> Mike,
>
> Perhaps t.setDaemon(true);?
>
> On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hello,
>>
>> So after adding a few print statements, I've found that the "while
>> (!jobControl.allFinished())" loop actually terminates, and so there
>> must be a non-daemon thread somewhere that is preventing the program
>> from terminating. Am I missing a configuration option to specify this?
>> Any ideas where it might be?
>>
>> Thanks!
>>
>> - Mike
>>
>>
>> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
>> <mi...@gmail.com> wrote:
>>> Hi all,
>>>
>>> I have a two-stage MR, written using entirely the new API, that I'm
>>> running using Hadoop 1.0.3. In the end it generates a file named
>>> part-r-0000 with the correct output, but the main method doesn't
>>> terminate, despite me using the allFinished method of JobControl as I
>>> thought I was supposed to. Any help would be greatly appreciated. A
>>> snippet from my code follows.
>>>
>>>>>>
>>>
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>>
>>> ...
>>>
>>> public static void main(String[] args) throws IOException {
>>> ...
>>>
>>> Job job1 = new Job(new Configuration(conf), "job1");
>>> ...
>>> ControlledJob controlledJob1 = new ControlledJob(job1,
>>> Collections.<ControlledJob>emptyList());
>>>
>>> Job job2 = new Job(new Configuration(conf), "job2");
>>> ...
>>> ControlledJob controlledJob2 = new ControlledJob(job2,
>>> Arrays.asList(controlledJob1));
>>>
>>> JobControl jobControl = new JobControl("main");
>>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>>> Thread t = new Thread(jobControl);
>>> t.start();
>>>
>>> while (!jobControl.allFinished()) {
>>> try {
>>> Thread.sleep(1000);
>>> } catch (InterruptedException e) {
>>> // Ignore.
>>> }
>>> }
>>> }
>>>
>>> <<<
>>>
>>> Thanks!
>>>
>>> - Mike
>
>
>
> --
> Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Harsh J <ha...@cloudera.com>.
Mike,
Perhaps t.setDaemon(true);?
On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
<mi...@gmail.com> wrote:
> Hello,
>
> So after adding a few print statements, I've found that the "while
> (!jobControl.allFinished())" loop actually terminates, and so there
> must be a non-daemon thread somewhere that is preventing the program
> from terminating. Am I missing a configuration option to specify this?
> Any ideas where it might be?
>
> Thanks!
>
> - Mike
>
>
> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hi all,
>>
>> I have a two-stage MR, written using entirely the new API, that I'm
>> running using Hadoop 1.0.3. In the end it generates a file named
>> part-r-0000 with the correct output, but the main method doesn't
>> terminate, despite me using the allFinished method of JobControl as I
>> thought I was supposed to. Any help would be greatly appreciated. A
>> snippet from my code follows.
>>
>>>>>
>>
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>
>> ...
>>
>> public static void main(String[] args) throws IOException {
>> ...
>>
>> Job job1 = new Job(new Configuration(conf), "job1");
>> ...
>> ControlledJob controlledJob1 = new ControlledJob(job1,
>> Collections.<ControlledJob>emptyList());
>>
>> Job job2 = new Job(new Configuration(conf), "job2");
>> ...
>> ControlledJob controlledJob2 = new ControlledJob(job2,
>> Arrays.asList(controlledJob1));
>>
>> JobControl jobControl = new JobControl("main");
>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>> Thread t = new Thread(jobControl);
>> t.start();
>>
>> while (!jobControl.allFinished()) {
>> try {
>> Thread.sleep(1000);
>> } catch (InterruptedException e) {
>> // Ignore.
>> }
>> }
>> }
>>
>> <<<
>>
>> Thanks!
>>
>> - Mike
--
Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Harsh J <ha...@cloudera.com>.
Mike,
Perhaps t.setDaemon(true);?
On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
<mi...@gmail.com> wrote:
> Hello,
>
> So after adding a few print statements, I've found that the "while
> (!jobControl.allFinished())" loop actually terminates, and so there
> must be a non-daemon thread somewhere that is preventing the program
> from terminating. Am I missing a configuration option to specify this?
> Any ideas where it might be?
>
> Thanks!
>
> - Mike
>
>
> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hi all,
>>
>> I have a two-stage MR, written using entirely the new API, that I'm
>> running using Hadoop 1.0.3. In the end it generates a file named
>> part-r-0000 with the correct output, but the main method doesn't
>> terminate, despite me using the allFinished method of JobControl as I
>> thought I was supposed to. Any help would be greatly appreciated. A
>> snippet from my code follows.
>>
>>>>>
>>
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>
>> ...
>>
>> public static void main(String[] args) throws IOException {
>> ...
>>
>> Job job1 = new Job(new Configuration(conf), "job1");
>> ...
>> ControlledJob controlledJob1 = new ControlledJob(job1,
>> Collections.<ControlledJob>emptyList());
>>
>> Job job2 = new Job(new Configuration(conf), "job2");
>> ...
>> ControlledJob controlledJob2 = new ControlledJob(job2,
>> Arrays.asList(controlledJob1));
>>
>> JobControl jobControl = new JobControl("main");
>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>> Thread t = new Thread(jobControl);
>> t.start();
>>
>> while (!jobControl.allFinished()) {
>> try {
>> Thread.sleep(1000);
>> } catch (InterruptedException e) {
>> // Ignore.
>> }
>> }
>> }
>>
>> <<<
>>
>> Thanks!
>>
>> - Mike
--
Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Harsh J <ha...@cloudera.com>.
Mike,
Perhaps t.setDaemon(true);?
On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
<mi...@gmail.com> wrote:
> Hello,
>
> So after adding a few print statements, I've found that the "while
> (!jobControl.allFinished())" loop actually terminates, and so there
> must be a non-daemon thread somewhere that is preventing the program
> from terminating. Am I missing a configuration option to specify this?
> Any ideas where it might be?
>
> Thanks!
>
> - Mike
>
>
> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hi all,
>>
>> I have a two-stage MR, written using entirely the new API, that I'm
>> running using Hadoop 1.0.3. In the end it generates a file named
>> part-r-0000 with the correct output, but the main method doesn't
>> terminate, despite me using the allFinished method of JobControl as I
>> thought I was supposed to. Any help would be greatly appreciated. A
>> snippet from my code follows.
>>
>>>>>
>>
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>
>> ...
>>
>> public static void main(String[] args) throws IOException {
>> ...
>>
>> Job job1 = new Job(new Configuration(conf), "job1");
>> ...
>> ControlledJob controlledJob1 = new ControlledJob(job1,
>> Collections.<ControlledJob>emptyList());
>>
>> Job job2 = new Job(new Configuration(conf), "job2");
>> ...
>> ControlledJob controlledJob2 = new ControlledJob(job2,
>> Arrays.asList(controlledJob1));
>>
>> JobControl jobControl = new JobControl("main");
>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>> Thread t = new Thread(jobControl);
>> t.start();
>>
>> while (!jobControl.allFinished()) {
>> try {
>> Thread.sleep(1000);
>> } catch (InterruptedException e) {
>> // Ignore.
>> }
>> }
>> }
>>
>> <<<
>>
>> Thanks!
>>
>> - Mike
--
Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Harsh J <ha...@cloudera.com>.
Mike,
Perhaps t.setDaemon(true);?
On Mon, Aug 20, 2012 at 9:54 PM, Michael Parker
<mi...@gmail.com> wrote:
> Hello,
>
> So after adding a few print statements, I've found that the "while
> (!jobControl.allFinished())" loop actually terminates, and so there
> must be a non-daemon thread somewhere that is preventing the program
> from terminating. Am I missing a configuration option to specify this?
> Any ideas where it might be?
>
> Thanks!
>
> - Mike
>
>
> On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
> <mi...@gmail.com> wrote:
>> Hi all,
>>
>> I have a two-stage MR, written using entirely the new API, that I'm
>> running using Hadoop 1.0.3. In the end it generates a file named
>> part-r-0000 with the correct output, but the main method doesn't
>> terminate, despite me using the allFinished method of JobControl as I
>> thought I was supposed to. Any help would be greatly appreciated. A
>> snippet from my code follows.
>>
>>>>>
>>
>> import org.apache.hadoop.mapreduce.Job;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
>> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>>
>> ...
>>
>> public static void main(String[] args) throws IOException {
>> ...
>>
>> Job job1 = new Job(new Configuration(conf), "job1");
>> ...
>> ControlledJob controlledJob1 = new ControlledJob(job1,
>> Collections.<ControlledJob>emptyList());
>>
>> Job job2 = new Job(new Configuration(conf), "job2");
>> ...
>> ControlledJob controlledJob2 = new ControlledJob(job2,
>> Arrays.asList(controlledJob1));
>>
>> JobControl jobControl = new JobControl("main");
>> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
>> Thread t = new Thread(jobControl);
>> t.start();
>>
>> while (!jobControl.allFinished()) {
>> try {
>> Thread.sleep(1000);
>> } catch (InterruptedException e) {
>> // Ignore.
>> }
>> }
>> }
>>
>> <<<
>>
>> Thanks!
>>
>> - Mike
--
Harsh J
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Hello,
So after adding a few print statements, I've found that the "while
(!jobControl.allFinished())" loop actually terminates, and so there
must be a non-daemon thread somewhere that is preventing the program
from terminating. Am I missing a configuration option to specify this?
Any ideas where it might be?
Thanks!
- Mike
On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
<mi...@gmail.com> wrote:
> Hi all,
>
> I have a two-stage MR, written using entirely the new API, that I'm
> running using Hadoop 1.0.3. In the end it generates a file named
> part-r-0000 with the correct output, but the main method doesn't
> terminate, despite me using the allFinished method of JobControl as I
> thought I was supposed to. Any help would be greatly appreciated. A
> snippet from my code follows.
>
>>>>
>
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>
> ...
>
> public static void main(String[] args) throws IOException {
> ...
>
> Job job1 = new Job(new Configuration(conf), "job1");
> ...
> ControlledJob controlledJob1 = new ControlledJob(job1,
> Collections.<ControlledJob>emptyList());
>
> Job job2 = new Job(new Configuration(conf), "job2");
> ...
> ControlledJob controlledJob2 = new ControlledJob(job2,
> Arrays.asList(controlledJob1));
>
> JobControl jobControl = new JobControl("main");
> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
> Thread t = new Thread(jobControl);
> t.start();
>
> while (!jobControl.allFinished()) {
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> // Ignore.
> }
> }
> }
>
> <<<
>
> Thanks!
>
> - Mike
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Hello,
So after adding a few print statements, I've found that the "while
(!jobControl.allFinished())" loop actually terminates, and so there
must be a non-daemon thread somewhere that is preventing the program
from terminating. Am I missing a configuration option to specify this?
Any ideas where it might be?
Thanks!
- Mike
On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
<mi...@gmail.com> wrote:
> Hi all,
>
> I have a two-stage MR, written using entirely the new API, that I'm
> running using Hadoop 1.0.3. In the end it generates a file named
> part-r-0000 with the correct output, but the main method doesn't
> terminate, despite me using the allFinished method of JobControl as I
> thought I was supposed to. Any help would be greatly appreciated. A
> snippet from my code follows.
>
>>>>
>
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>
> ...
>
> public static void main(String[] args) throws IOException {
> ...
>
> Job job1 = new Job(new Configuration(conf), "job1");
> ...
> ControlledJob controlledJob1 = new ControlledJob(job1,
> Collections.<ControlledJob>emptyList());
>
> Job job2 = new Job(new Configuration(conf), "job2");
> ...
> ControlledJob controlledJob2 = new ControlledJob(job2,
> Arrays.asList(controlledJob1));
>
> JobControl jobControl = new JobControl("main");
> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
> Thread t = new Thread(jobControl);
> t.start();
>
> while (!jobControl.allFinished()) {
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> // Ignore.
> }
> }
> }
>
> <<<
>
> Thanks!
>
> - Mike
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Hello,
So after adding a few print statements, I've found that the "while
(!jobControl.allFinished())" loop actually terminates, and so there
must be a non-daemon thread somewhere that is preventing the program
from terminating. Am I missing a configuration option to specify this?
Any ideas where it might be?
Thanks!
- Mike
On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
<mi...@gmail.com> wrote:
> Hi all,
>
> I have a two-stage MR, written using entirely the new API, that I'm
> running using Hadoop 1.0.3. In the end it generates a file named
> part-r-0000 with the correct output, but the main method doesn't
> terminate, despite me using the allFinished method of JobControl as I
> thought I was supposed to. Any help would be greatly appreciated. A
> snippet from my code follows.
>
>>>>
>
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>
> ...
>
> public static void main(String[] args) throws IOException {
> ...
>
> Job job1 = new Job(new Configuration(conf), "job1");
> ...
> ControlledJob controlledJob1 = new ControlledJob(job1,
> Collections.<ControlledJob>emptyList());
>
> Job job2 = new Job(new Configuration(conf), "job2");
> ...
> ControlledJob controlledJob2 = new ControlledJob(job2,
> Arrays.asList(controlledJob1));
>
> JobControl jobControl = new JobControl("main");
> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
> Thread t = new Thread(jobControl);
> t.start();
>
> while (!jobControl.allFinished()) {
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> // Ignore.
> }
> }
> }
>
> <<<
>
> Thanks!
>
> - Mike
Re: Two-stage MR not terminating gracefully with JobControl?
Posted by Michael Parker <mi...@gmail.com>.
Hello,
So after adding a few print statements, I've found that the "while
(!jobControl.allFinished())" loop actually terminates, and so there
must be a non-daemon thread somewhere that is preventing the program
from terminating. Am I missing a configuration option to specify this?
Any ideas where it might be?
Thanks!
- Mike
On Sun, Aug 19, 2012 at 9:46 AM, Michael Parker
<mi...@gmail.com> wrote:
> Hi all,
>
> I have a two-stage MR, written using entirely the new API, that I'm
> running using Hadoop 1.0.3. In the end it generates a file named
> part-r-0000 with the correct output, but the main method doesn't
> terminate, despite me using the allFinished method of JobControl as I
> thought I was supposed to. Any help would be greatly appreciated. A
> snippet from my code follows.
>
>>>>
>
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
> import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
>
> ...
>
> public static void main(String[] args) throws IOException {
> ...
>
> Job job1 = new Job(new Configuration(conf), "job1");
> ...
> ControlledJob controlledJob1 = new ControlledJob(job1,
> Collections.<ControlledJob>emptyList());
>
> Job job2 = new Job(new Configuration(conf), "job2");
> ...
> ControlledJob controlledJob2 = new ControlledJob(job2,
> Arrays.asList(controlledJob1));
>
> JobControl jobControl = new JobControl("main");
> jobControl.addJobCollection(Arrays.asList(controlledJob1, controlledJob2));
> Thread t = new Thread(jobControl);
> t.start();
>
> while (!jobControl.allFinished()) {
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> // Ignore.
> }
> }
> }
>
> <<<
>
> Thanks!
>
> - Mike