You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by madhu phatak <ph...@gmail.com> on 2012/06/25 10:37:01 UTC
Cyclic dependency in JobControl job DAG
Hi,
In current implementation of JobControl, whenever there is a cyclic
dependency between the jobs it throws a Stack overflow exception .
For example,
ControlledJob job1 = new ControlledJob(new Configuration());
job1.setJobName("job1");
ControlledJob job2 = new ControlledJob(new Configuration());
job2.setJobName("job2");
job1.addDependingJob(job2);
job2.addDependingJob(job1);
JobControl jobControl = new JobControl("jobcontrol");
jobControl.addJob(job1);
jobControl.addJob(job2);
jobControl.run();
throws
java.lang.StackOverflowError
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)
Whenever we write complex application, there is always possibility of
cyclic dependencies.I have written a method which checks for the cyclic
dependency upfront and informs it to the user. I want to know from you
guys, do you think is it a useful feature? If yes I can contribute it as a
patch.
Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar
Re: Cyclic dependency in JobControl job DAG
Posted by madhu phatak <ph...@gmail.com>.
Hi,
Opened a jira ticket ( https://issues.apache.org/jira/browse/MAPREDUCE-4371 )
and attached initial patch. Can anyone please review the patch and give
feedback?
On Mon, Jun 25, 2012 at 11:43 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
>
> +1, can you please open a JIRA ticket (at
> https://issues.apache.org/jira/browse/MAPREDUCE) and attach a patch?
>
> Thanks,
> +Vinod
>
> On Jun 25, 2012, at 6:53 AM, Robert Evans wrote:
>
> > I personally think it is useful. I would say contribute it.
> >
> > (Moved common-dev to bcc, we try not to cross post on these lists)
> >
> > --Bobby Evans
> >
> > On 6/25/12 3:37 AM, "madhu phatak" <ph...@gmail.com> wrote:
> >
> > Hi,
> > In current implementation of JobControl, whenever there is a cyclic
> > dependency between the jobs it throws a Stack overflow exception .
> > For example,
> > ControlledJob job1 = new ControlledJob(new Configuration());
> > job1.setJobName("job1");
> > ControlledJob job2 = new ControlledJob(new Configuration());
> > job2.setJobName("job2");
> > job1.addDependingJob(job2);
> > job2.addDependingJob(job1);
> > JobControl jobControl = new JobControl("jobcontrol");
> > jobControl.addJob(job1);
> > jobControl.addJob(job2);
> > jobControl.run();
> >
> > throws
> > java.lang.StackOverflowError
> > at java.util.ArrayList.get(ArrayList.java:322)
> > at
> >
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)
> >
> > Whenever we write complex application, there is always possibility of
> > cyclic dependencies.I have written a method which checks for the cyclic
> > dependency upfront and informs it to the user. I want to know from you
> > guys, do you think is it a useful feature? If yes I can contribute it as
> a
> > patch.
> >
> > Regards,
> > Madhukara Phatak
> > --
> > https://github.com/zinnia-phatak-dev/Nectar
> >
>
>
--
https://github.com/zinnia-phatak-dev/Nectar
Re: Cyclic dependency in JobControl job DAG
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
+1, can you please open a JIRA ticket (at https://issues.apache.org/jira/browse/MAPREDUCE) and attach a patch?
Thanks,
+Vinod
On Jun 25, 2012, at 6:53 AM, Robert Evans wrote:
> I personally think it is useful. I would say contribute it.
>
> (Moved common-dev to bcc, we try not to cross post on these lists)
>
> --Bobby Evans
>
> On 6/25/12 3:37 AM, "madhu phatak" <ph...@gmail.com> wrote:
>
> Hi,
> In current implementation of JobControl, whenever there is a cyclic
> dependency between the jobs it throws a Stack overflow exception .
> For example,
> ControlledJob job1 = new ControlledJob(new Configuration());
> job1.setJobName("job1");
> ControlledJob job2 = new ControlledJob(new Configuration());
> job2.setJobName("job2");
> job1.addDependingJob(job2);
> job2.addDependingJob(job1);
> JobControl jobControl = new JobControl("jobcontrol");
> jobControl.addJob(job1);
> jobControl.addJob(job2);
> jobControl.run();
>
> throws
> java.lang.StackOverflowError
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)
>
> Whenever we write complex application, there is always possibility of
> cyclic dependencies.I have written a method which checks for the cyclic
> dependency upfront and informs it to the user. I want to know from you
> guys, do you think is it a useful feature? If yes I can contribute it as a
> patch.
>
> Regards,
> Madhukara Phatak
> --
> https://github.com/zinnia-phatak-dev/Nectar
>
Re: Cyclic dependency in JobControl job DAG
Posted by Robert Evans <ev...@yahoo-inc.com>.
I personally think it is useful. I would say contribute it.
(Moved common-dev to bcc, we try not to cross post on these lists)
--Bobby Evans
On 6/25/12 3:37 AM, "madhu phatak" <ph...@gmail.com> wrote:
Hi,
In current implementation of JobControl, whenever there is a cyclic
dependency between the jobs it throws a Stack overflow exception .
For example,
ControlledJob job1 = new ControlledJob(new Configuration());
job1.setJobName("job1");
ControlledJob job2 = new ControlledJob(new Configuration());
job2.setJobName("job2");
job1.addDependingJob(job2);
job2.addDependingJob(job1);
JobControl jobControl = new JobControl("jobcontrol");
jobControl.addJob(job1);
jobControl.addJob(job2);
jobControl.run();
throws
java.lang.StackOverflowError
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)
Whenever we write complex application, there is always possibility of
cyclic dependencies.I have written a method which checks for the cyclic
dependency upfront and informs it to the user. I want to know from you
guys, do you think is it a useful feature? If yes I can contribute it as a
patch.
Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar
Re: Cyclic dependency in JobControl job DAG
Posted by Robert Evans <ev...@yahoo-inc.com>.
I personally think it is useful. I would say contribute it.
(Moved common-dev to bcc, we try not to cross post on these lists)
--Bobby Evans
On 6/25/12 3:37 AM, "madhu phatak" <ph...@gmail.com> wrote:
Hi,
In current implementation of JobControl, whenever there is a cyclic
dependency between the jobs it throws a Stack overflow exception .
For example,
ControlledJob job1 = new ControlledJob(new Configuration());
job1.setJobName("job1");
ControlledJob job2 = new ControlledJob(new Configuration());
job2.setJobName("job2");
job1.addDependingJob(job2);
job2.addDependingJob(job1);
JobControl jobControl = new JobControl("jobcontrol");
jobControl.addJob(job1);
jobControl.addJob(job2);
jobControl.run();
throws
java.lang.StackOverflowError
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.checkState(ControlledJob.java:295)
Whenever we write complex application, there is always possibility of
cyclic dependencies.I have written a method which checks for the cyclic
dependency upfront and informs it to the user. I want to know from you
guys, do you think is it a useful feature? If yes I can contribute it as a
patch.
Regards,
Madhukara Phatak
--
https://github.com/zinnia-phatak-dev/Nectar