You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Mihai Varga <mi...@gmail.com> on 2016/04/07 13:27:49 UTC
How to access job/application parameters in the scheduler
Hello,
I hope this is the right place to ask this. I'm trying to write a new
scheduler that takes into consideration a deadline for a job and I'm not
sure how to pass the job specific deadline to the scheduler (say
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java).
I know that additional parameters can be passed through
Configuration conf = new Configuration();
conf.set("test", "123");
or with the -D command when submitting a job, but I'm not sure how to
retrieve those parameters in the scheduler where I only have an
ApplicationID.
I'm using Hadoop 2.7.2 and this is a school project and does not need to go
into production, so any hack would do. Any idea on how to do this?
Thanks in advance!
Best regards,
Mihai
Re: How to access job/application parameters in the scheduler
Posted by Mihai Varga <mi...@gmail.com>.
Hi Wangda,
Thank you for the idea, it worked!
On 8 April 2016 at 03:19, Wangda Tan <wh...@gmail.com> wrote:
> Hi Mihal,
>
> Configuration specified by job will not pass to RM.
>
> The only way you can do is to add a new field to
> ApplicationSubmissionContext so scheduler can see it.
>
> HTH,
> - Wangda
>
>
>
> On Thu, Apr 7, 2016 at 4:27 AM, Mihai Varga <mi...@gmail.com> wrote:
>
> > Hello,
> >
> > I hope this is the right place to ask this. I'm trying to write a new
> > scheduler that takes into consideration a deadline for a job and I'm not
> > sure how to pass the job specific deadline to the scheduler (say
> >
> >
> hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java).
> >
> > I know that additional parameters can be passed through
> >
> > Configuration conf = new Configuration();
> > conf.set("test", "123");
> >
> > or with the -D command when submitting a job, but I'm not sure how to
> > retrieve those parameters in the scheduler where I only have an
> > ApplicationID.
> >
> > I'm using Hadoop 2.7.2 and this is a school project and does not need to
> go
> > into production, so any hack would do. Any idea on how to do this?
> >
> > Thanks in advance!
> >
> > Best regards,
> > Mihai
> >
>
Re: How to access job/application parameters in the scheduler
Posted by Wangda Tan <wh...@gmail.com>.
Hi Mihal,
Configuration specified by job will not pass to RM.
The only way you can do is to add a new field to
ApplicationSubmissionContext so scheduler can see it.
HTH,
- Wangda
On Thu, Apr 7, 2016 at 4:27 AM, Mihai Varga <mi...@gmail.com> wrote:
> Hello,
>
> I hope this is the right place to ask this. I'm trying to write a new
> scheduler that takes into consideration a deadline for a job and I'm not
> sure how to pass the job specific deadline to the scheduler (say
>
> hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java).
>
> I know that additional parameters can be passed through
>
> Configuration conf = new Configuration();
> conf.set("test", "123");
>
> or with the -D command when submitting a job, but I'm not sure how to
> retrieve those parameters in the scheduler where I only have an
> ApplicationID.
>
> I'm using Hadoop 2.7.2 and this is a school project and does not need to go
> into production, so any hack would do. Any idea on how to do this?
>
> Thanks in advance!
>
> Best regards,
> Mihai
>