You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Tim Israel <ti...@timisrael.com> on 2015/01/31 02:22:39 UTC
Slider 0.50.2 requests container from decommissioned node.
Hi all,
I realize I'm using an older version of slider, but I think this was the
last release to be compiled against Hadoop 2.4 with Kerberos working. We
aren't positioned to upgrade to 2.6 yet. That being said, Slider has
worked pretty flawlessly for us so far.
I checked JIRA, and didn't find a bug report matching this scenario.
I recently decommissioned several of our NodeManagers. The NodeManagers
are in decommissioned status in the RM UI. However, Slider continues to
request those nodes (based on historical preference I assume).
15/01/30 17:01:26 INFO state.RoleHistory: There're 1 nodes to consider for
STORM_UI_SERVER
15/01/30 17:01:26 INFO state.OutstandingRequest: Submitting request for
container on [NODE A - FQDN censored]
NODE A in this case has been decommissioned and shows up in the RM UI as
decommissioned as well as in the yarn.exclude list.
If this is fixed in a later version, then I'm okay with that. I figured it
was worth mentioning though.
More to the point, is there a way to clear role history (or even modify it
by hand) and force requests to go somewhere else?
Thanks,
Tim
Re: Slider 0.50.2 requests container from decommissioned node.
Posted by Ted Yu <yu...@gmail.com>.
The two JIRAs you mentioned were in 0.60 release (HDP 2.2)
Please consider upgrading.
Cheers
On Sat, Jan 31, 2015 at 4:05 PM, Tim Israel <ti...@timisrael.com> wrote:
> Thanks Ted, SLIDER-647 and SLIDER-439 were exactly what I was looking for.
>
> Tim
>
> On Sat, Jan 31, 2015 at 6:34 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Tim:
> > We have been improving role history in develop branch.
> > Take a look at the recent JIRAs:
> >
> > SLIDER-743. Include node failure history when choosing placement hints
> > SLIDER-647 allocation requests not being satisfied when a cluster goes to
> > labels
> > SLIDER-81. Support placement of containers on labeled YARN nodes
> >
> > All the above would be integrated in the upcoming 0.70 release.
> >
> > Cheers
> >
> > On Fri, Jan 30, 2015 at 5:50 PM, Tim Israel <ti...@timisrael.com> wrote:
> >
> > > As soon as I sent this, I decided to just clear out the following
> folder
> > in
> > > hdfs
> > >
> > > /user/<APP USER>/.slider/cluster/<CLUSTER NAME>/history/*
> > >
> > > That resolved the issue.
> > >
> > > Thanks,
> > >
> > > Tim
> > >
> > > Tim
> > >
> > > On Fri, Jan 30, 2015 at 8:22 PM, Tim Israel <ti...@timisrael.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I realize I'm using an older version of slider, but I think this was
> > the
> > > > last release to be compiled against Hadoop 2.4 with Kerberos working.
> > We
> > > > aren't positioned to upgrade to 2.6 yet. That being said, Slider has
> > > > worked pretty flawlessly for us so far.
> > > >
> > > > I checked JIRA, and didn't find a bug report matching this scenario.
> > > >
> > > > I recently decommissioned several of our NodeManagers. The
> > NodeManagers
> > > > are in decommissioned status in the RM UI. However, Slider continues
> > to
> > > > request those nodes (based on historical preference I assume).
> > > >
> > > > 15/01/30 17:01:26 INFO state.RoleHistory: There're 1 nodes to
> consider
> > > for
> > > > STORM_UI_SERVER
> > > > 15/01/30 17:01:26 INFO state.OutstandingRequest: Submitting request
> for
> > > > container on [NODE A - FQDN censored]
> > > >
> > > > NODE A in this case has been decommissioned and shows up in the RM UI
> > as
> > > > decommissioned as well as in the yarn.exclude list.
> > > >
> > > > If this is fixed in a later version, then I'm okay with that. I
> > figured
> > > > it was worth mentioning though.
> > > >
> > > > More to the point, is there a way to clear role history (or even
> modify
> > > it
> > > > by hand) and force requests to go somewhere else?
> > > >
> > > > Thanks,
> > > >
> > > > Tim
> > > >
> > >
> >
>
Re: Slider 0.50.2 requests container from decommissioned node.
Posted by Tim Israel <ti...@timisrael.com>.
Thanks Ted, SLIDER-647 and SLIDER-439 were exactly what I was looking for.
Tim
On Sat, Jan 31, 2015 at 6:34 PM, Ted Yu <yu...@gmail.com> wrote:
> Tim:
> We have been improving role history in develop branch.
> Take a look at the recent JIRAs:
>
> SLIDER-743. Include node failure history when choosing placement hints
> SLIDER-647 allocation requests not being satisfied when a cluster goes to
> labels
> SLIDER-81. Support placement of containers on labeled YARN nodes
>
> All the above would be integrated in the upcoming 0.70 release.
>
> Cheers
>
> On Fri, Jan 30, 2015 at 5:50 PM, Tim Israel <ti...@timisrael.com> wrote:
>
> > As soon as I sent this, I decided to just clear out the following folder
> in
> > hdfs
> >
> > /user/<APP USER>/.slider/cluster/<CLUSTER NAME>/history/*
> >
> > That resolved the issue.
> >
> > Thanks,
> >
> > Tim
> >
> > Tim
> >
> > On Fri, Jan 30, 2015 at 8:22 PM, Tim Israel <ti...@timisrael.com> wrote:
> >
> > > Hi all,
> > >
> > > I realize I'm using an older version of slider, but I think this was
> the
> > > last release to be compiled against Hadoop 2.4 with Kerberos working.
> We
> > > aren't positioned to upgrade to 2.6 yet. That being said, Slider has
> > > worked pretty flawlessly for us so far.
> > >
> > > I checked JIRA, and didn't find a bug report matching this scenario.
> > >
> > > I recently decommissioned several of our NodeManagers. The
> NodeManagers
> > > are in decommissioned status in the RM UI. However, Slider continues
> to
> > > request those nodes (based on historical preference I assume).
> > >
> > > 15/01/30 17:01:26 INFO state.RoleHistory: There're 1 nodes to consider
> > for
> > > STORM_UI_SERVER
> > > 15/01/30 17:01:26 INFO state.OutstandingRequest: Submitting request for
> > > container on [NODE A - FQDN censored]
> > >
> > > NODE A in this case has been decommissioned and shows up in the RM UI
> as
> > > decommissioned as well as in the yarn.exclude list.
> > >
> > > If this is fixed in a later version, then I'm okay with that. I
> figured
> > > it was worth mentioning though.
> > >
> > > More to the point, is there a way to clear role history (or even modify
> > it
> > > by hand) and force requests to go somewhere else?
> > >
> > > Thanks,
> > >
> > > Tim
> > >
> >
>
Re: Slider 0.50.2 requests container from decommissioned node.
Posted by Ted Yu <yu...@gmail.com>.
Tim:
We have been improving role history in develop branch.
Take a look at the recent JIRAs:
SLIDER-743. Include node failure history when choosing placement hints
SLIDER-647 allocation requests not being satisfied when a cluster goes to
labels
SLIDER-81. Support placement of containers on labeled YARN nodes
All the above would be integrated in the upcoming 0.70 release.
Cheers
On Fri, Jan 30, 2015 at 5:50 PM, Tim Israel <ti...@timisrael.com> wrote:
> As soon as I sent this, I decided to just clear out the following folder in
> hdfs
>
> /user/<APP USER>/.slider/cluster/<CLUSTER NAME>/history/*
>
> That resolved the issue.
>
> Thanks,
>
> Tim
>
> Tim
>
> On Fri, Jan 30, 2015 at 8:22 PM, Tim Israel <ti...@timisrael.com> wrote:
>
> > Hi all,
> >
> > I realize I'm using an older version of slider, but I think this was the
> > last release to be compiled against Hadoop 2.4 with Kerberos working. We
> > aren't positioned to upgrade to 2.6 yet. That being said, Slider has
> > worked pretty flawlessly for us so far.
> >
> > I checked JIRA, and didn't find a bug report matching this scenario.
> >
> > I recently decommissioned several of our NodeManagers. The NodeManagers
> > are in decommissioned status in the RM UI. However, Slider continues to
> > request those nodes (based on historical preference I assume).
> >
> > 15/01/30 17:01:26 INFO state.RoleHistory: There're 1 nodes to consider
> for
> > STORM_UI_SERVER
> > 15/01/30 17:01:26 INFO state.OutstandingRequest: Submitting request for
> > container on [NODE A - FQDN censored]
> >
> > NODE A in this case has been decommissioned and shows up in the RM UI as
> > decommissioned as well as in the yarn.exclude list.
> >
> > If this is fixed in a later version, then I'm okay with that. I figured
> > it was worth mentioning though.
> >
> > More to the point, is there a way to clear role history (or even modify
> it
> > by hand) and force requests to go somewhere else?
> >
> > Thanks,
> >
> > Tim
> >
>
Re: Slider 0.50.2 requests container from decommissioned node.
Posted by Tim Israel <ti...@timisrael.com>.
As soon as I sent this, I decided to just clear out the following folder in
hdfs
/user/<APP USER>/.slider/cluster/<CLUSTER NAME>/history/*
That resolved the issue.
Thanks,
Tim
Tim
On Fri, Jan 30, 2015 at 8:22 PM, Tim Israel <ti...@timisrael.com> wrote:
> Hi all,
>
> I realize I'm using an older version of slider, but I think this was the
> last release to be compiled against Hadoop 2.4 with Kerberos working. We
> aren't positioned to upgrade to 2.6 yet. That being said, Slider has
> worked pretty flawlessly for us so far.
>
> I checked JIRA, and didn't find a bug report matching this scenario.
>
> I recently decommissioned several of our NodeManagers. The NodeManagers
> are in decommissioned status in the RM UI. However, Slider continues to
> request those nodes (based on historical preference I assume).
>
> 15/01/30 17:01:26 INFO state.RoleHistory: There're 1 nodes to consider for
> STORM_UI_SERVER
> 15/01/30 17:01:26 INFO state.OutstandingRequest: Submitting request for
> container on [NODE A - FQDN censored]
>
> NODE A in this case has been decommissioned and shows up in the RM UI as
> decommissioned as well as in the yarn.exclude list.
>
> If this is fixed in a later version, then I'm okay with that. I figured
> it was worth mentioning though.
>
> More to the point, is there a way to clear role history (or even modify it
> by hand) and force requests to go somewhere else?
>
> Thanks,
>
> Tim
>