You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Stephen Gran <st...@piksel.com> on 2016/06/30 19:53:08 UTC
NPE in removing container
Hi,
Just playing with the 0.2.0 release (congratulations, by the way!)
I have seen this twice now, although it is by no means consistent - I
will have a dozen successful runs, and then one of these. This exits
the RM, which makes it rather noticable.
2016-06-30 19:47:43,952 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Removed node slave2:24679 cluster capacity: <memory:4096, vCore
s:4>
2016-06-30 19:47:43,953 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
handling event type NODE_RESOURCE_UPDATE to the scheduler
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
at
org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
at
org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
at java.lang.Thread.run(Thread.java:745)
2016-06-30 19:47:43,972 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
bbye..
--
Stephen Gran
Senior Technical Architect
picture the possibilities | piksel.com
This message is private and confidential. If you have received this message in error, please notify the sender or servicedesk@piksel.com and remove it from your system.
Piksel Inc is a company registered in the United States New York City, 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
Re: NPE in removing container
Posted by Darin Johnson <db...@gmail.com>.
Hey Stephen,
I was on vacation last week, I'm looking over the logs this week. I've got
a few ideas for a first but may take me a while as I get back into work.
Darin
On Fri, Jul 1, 2016 at 2:43 AM, Stephen Gran <st...@piksel.com>
wrote:
> Hi,
>
> It's not a problem at all. Anything I can do to help.
>
> I've attached the log file for the relevant time period. This is hadoop
> 2.7.2 - you have a good memory :)
>
> Cheers,
>
> On 30/06/16 22:56, Darin Johnson wrote:
> > Hey Steven,
> >
> > Looks like this might be slightly different than what I was originally
> > expecting. Sorry to keep asking for more info but it will help me
> recreate
> > the issue. Could you possibly get me more of the ResourceManager logs?
> In
> > particular, I'm trying to figure out where upgradeNodeCapacity is getting
> > called from and any transitions of slave2. Also, what version of hadoop
> > are you running, I think I recall it being 2.72 but should verify.
> >
> > Thanks for taken the time to work with me on this.
> >
> > Darin
> >
> > On Thu, Jun 30, 2016 at 5:10 PM, Stephen Gran <st...@piksel.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Yes - the imaginatively named slave2 was a zero-sized nm at that point -
> >> I am looking at how small a pool of reserved resource I can get away
> >> with, and use FGS for burst activity.
> >>
> >>
> >> Here are all the logs related to that host:port combination around that
> >> time:
> >>
> >> 2016-06-30 19:47:43,756 INFO
> >> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> >> Expired:slave2:24679 Timed out after 2 secs
> >> 2016-06-30 19:47:43,771 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> Deactivating Node slave2:24679 as it is now LOST
> >> 2016-06-30 19:47:43,771 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> slave2:24679 Node Transitioned from RUNNING to LOST
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> >> yarn_Container: [ContainerId: container_1467314892573_0009_01_000005,
> >> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> >> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> >> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> >> cpu and 1 mem.
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> >> Released container container_1467314892573_0009_01_000005 of capacity
> >> <memory:2048, vCores:1> on host slave2:24679, which currently has 1
> >> containers, <memory:2048, vCores:1> used and <memory:2048, vCores:1>
> >> available, release resources=true
> >> 2016-06-30 19:47:43,909 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Application attempt appattempt_1467314892573_0009_000001 released
> >> container container_1467314892573_0009_01_000005 on node: host:
> >> slave2:24679 #containers=1 available=<memory:2048, vCores:1>
> >> used=<memory:2048, vCores:1> with event: KILL
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> >> Node not found resyncing slave2:24679
> >> 2016-06-30 19:47:43,952 INFO
> >> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> >> yarn_Container: [ContainerId: container_1467314892573_0009_01_000006,
> >> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> >> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> >> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> >> cpu and 1 mem.
> >> 2016-06-30 19:47:43,952 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> >> Released container container_1467314892573_0009_01_000006 of capacity
> >> <memory:2048, vCores:1> on host slave2:24679, which currently has 0
> >> containers, <memory:0, vCores:0> used and <memory:4096, vCores:2>
> >> available, release resources=true
> >> 2016-06-30 19:47:43,952 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Application attempt appattempt_1467314892573_0009_000001 released
> >> container container_1467314892573_0009_01_000006 on node: host:
> >> slave2:24679 #containers=0 available=<memory:4096, vCores:2>
> >> used=<memory:0, vCores:0> with event: KILL
> >> 2016-06-30 19:47:43,952 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Removed node slave2:24679 cluster capacity: <memory:4096, vCores:4>
> >> 2016-06-30 19:47:47,573 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> slave2:24679 Node Transitioned from NEW to RUNNING
> >> 2016-06-30 19:47:47,936 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> >> NodeManager from node slave2(cmPort: 24679 httpPort: 23177) registered
> >> with capability: <memory:0, vCores:0>, assigned nodeId slave2:24679
> >>
> >>
> >> Looks like it did go into LOST for a bit.
> >>
> >> Cheers,
> >>
> >> On 30/06/16 21:36, Darin Johnson wrote:
> >>> Steven, thanks. I thought I had fixed that but perhaps a regression
> was
> >>> made in another merge. I'll look into it, can you answer a few
> >> questions?
> >>> Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
> >>> manager logs had it recently become unhealthy? I'm pretty concerned
> >> about
> >>> this and will try to get a patch soon.
> >>>
> >>> Thanks,
> >>>
> >>> Darin
> >>> On Jun 30, 2016 3:53 PM, "Stephen Gran" <st...@piksel.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Just playing with the 0.2.0 release (congratulations, by the way!)
> >>>>
> >>>> I have seen this twice now, although it is by no means consistent - I
> >>>> will have a dozen successful runs, and then one of these. This exits
> >>>> the RM, which makes it rather noticable.
> >>>>
> >>>> 2016-06-30 19:47:43,952 INFO
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >>>> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
> >>>> s:4>
> >>>> 2016-06-30 19:47:43,953 FATAL
> >>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> in
> >>>> handling event type NODE_RESOURCE_UPDATE to the scheduler
> >>>> java.lang.NullPointerException
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-06-30 19:47:43,972 INFO
> >>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
> Exiting,
> >>>> bbye..
> >>>>
> >>>> --
> >>>> Stephen Gran
> >>>> Senior Technical Architect
> >>>>
> >>>> picture the possibilities | piksel.com
> >>>> This message is private and confidential. If you have received this
> >>>> message in error, please notify the sender or servicedesk@piksel.com
> >> and
> >>>> remove it from your system.
> >>>>
> >>>> Piksel Inc is a company registered in the United States New York City,
> >>>> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
> >>>>
> >>>
> >>
> >> --
> >> Stephen Gran
> >> Senior Technical Architect
> >>
> >> picture the possibilities | piksel.com
> >>
> >
>
> --
> Stephen Gran
> Senior Technical Architect
>
> picture the possibilities | piksel.com
>
Re: NPE in removing container
Posted by Stephen Gran <st...@piksel.com>.
Hi,
It's not a problem at all. Anything I can do to help.
I've attached the log file for the relevant time period. This is hadoop
2.7.2 - you have a good memory :)
Cheers,
On 30/06/16 22:56, Darin Johnson wrote:
> Hey Steven,
>
> Looks like this might be slightly different than what I was originally
> expecting. Sorry to keep asking for more info but it will help me recreate
> the issue. Could you possibly get me more of the ResourceManager logs? In
> particular, I'm trying to figure out where upgradeNodeCapacity is getting
> called from and any transitions of slave2. Also, what version of hadoop
> are you running, I think I recall it being 2.72 but should verify.
>
> Thanks for taken the time to work with me on this.
>
> Darin
>
> On Thu, Jun 30, 2016 at 5:10 PM, Stephen Gran <st...@piksel.com>
> wrote:
>
>> Hi,
>>
>> Yes - the imaginatively named slave2 was a zero-sized nm at that point -
>> I am looking at how small a pool of reserved resource I can get away
>> with, and use FGS for burst activity.
>>
>>
>> Here are all the logs related to that host:port combination around that
>> time:
>>
>> 2016-06-30 19:47:43,756 INFO
>> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
>> Expired:slave2:24679 Timed out after 2 secs
>> 2016-06-30 19:47:43,771 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
>> Deactivating Node slave2:24679 as it is now LOST
>> 2016-06-30 19:47:43,771 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
>> slave2:24679 Node Transitioned from RUNNING to LOST
>> 2016-06-30 19:47:43,909 INFO
>> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
>> yarn_Container: [ContainerId: container_1467314892573_0009_01_000005,
>> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
>> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
>> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
>> cpu and 1 mem.
>> 2016-06-30 19:47:43,909 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
>> Released container container_1467314892573_0009_01_000005 of capacity
>> <memory:2048, vCores:1> on host slave2:24679, which currently has 1
>> containers, <memory:2048, vCores:1> used and <memory:2048, vCores:1>
>> available, release resources=true
>> 2016-06-30 19:47:43,909 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> Application attempt appattempt_1467314892573_0009_000001 released
>> container container_1467314892573_0009_01_000005 on node: host:
>> slave2:24679 #containers=1 available=<memory:2048, vCores:1>
>> used=<memory:2048, vCores:1> with event: KILL
>> 2016-06-30 19:47:43,909 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
>> Node not found resyncing slave2:24679
>> 2016-06-30 19:47:43,952 INFO
>> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
>> yarn_Container: [ContainerId: container_1467314892573_0009_01_000006,
>> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
>> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
>> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
>> cpu and 1 mem.
>> 2016-06-30 19:47:43,952 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
>> Released container container_1467314892573_0009_01_000006 of capacity
>> <memory:2048, vCores:1> on host slave2:24679, which currently has 0
>> containers, <memory:0, vCores:0> used and <memory:4096, vCores:2>
>> available, release resources=true
>> 2016-06-30 19:47:43,952 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> Application attempt appattempt_1467314892573_0009_000001 released
>> container container_1467314892573_0009_01_000006 on node: host:
>> slave2:24679 #containers=0 available=<memory:4096, vCores:2>
>> used=<memory:0, vCores:0> with event: KILL
>> 2016-06-30 19:47:43,952 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> Removed node slave2:24679 cluster capacity: <memory:4096, vCores:4>
>> 2016-06-30 19:47:47,573 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
>> slave2:24679 Node Transitioned from NEW to RUNNING
>> 2016-06-30 19:47:47,936 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
>> NodeManager from node slave2(cmPort: 24679 httpPort: 23177) registered
>> with capability: <memory:0, vCores:0>, assigned nodeId slave2:24679
>>
>>
>> Looks like it did go into LOST for a bit.
>>
>> Cheers,
>>
>> On 30/06/16 21:36, Darin Johnson wrote:
>>> Steven, thanks. I thought I had fixed that but perhaps a regression was
>>> made in another merge. I'll look into it, can you answer a few
>> questions?
>>> Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
>>> manager logs had it recently become unhealthy? I'm pretty concerned
>> about
>>> this and will try to get a patch soon.
>>>
>>> Thanks,
>>>
>>> Darin
>>> On Jun 30, 2016 3:53 PM, "Stephen Gran" <st...@piksel.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Just playing with the 0.2.0 release (congratulations, by the way!)
>>>>
>>>> I have seen this twice now, although it is by no means consistent - I
>>>> will have a dozen successful runs, and then one of these. This exits
>>>> the RM, which makes it rather noticable.
>>>>
>>>> 2016-06-30 19:47:43,952 INFO
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>>>> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
>>>> s:4>
>>>> 2016-06-30 19:47:43,953 FATAL
>>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
>>>> handling event type NODE_RESOURCE_UPDATE to the scheduler
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
>>>> at
>>>>
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
>>>> at
>>>>
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
>>>> at
>>>>
>>>>
>> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
>>>> at
>>>>
>>>>
>> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
>>>> at
>>>>
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-06-30 19:47:43,972 INFO
>>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
>>>> bbye..
>>>>
>>>> --
>>>> Stephen Gran
>>>> Senior Technical Architect
>>>>
>>>> picture the possibilities | piksel.com
>>>> This message is private and confidential. If you have received this
>>>> message in error, please notify the sender or servicedesk@piksel.com
>> and
>>>> remove it from your system.
>>>>
>>>> Piksel Inc is a company registered in the United States New York City,
>>>> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
>>>>
>>>
>>
>> --
>> Stephen Gran
>> Senior Technical Architect
>>
>> picture the possibilities | piksel.com
>>
>
--
Stephen Gran
Senior Technical Architect
picture the possibilities | piksel.com
Re: NPE in removing container
Posted by Darin Johnson <db...@gmail.com>.
Hey Steven,
Looks like this might be slightly different than what I was originally
expecting. Sorry to keep asking for more info but it will help me recreate
the issue. Could you possibly get me more of the ResourceManager logs? In
particular, I'm trying to figure out where upgradeNodeCapacity is getting
called from and any transitions of slave2. Also, what version of hadoop
are you running, I think I recall it being 2.72 but should verify.
Thanks for taken the time to work with me on this.
Darin
On Thu, Jun 30, 2016 at 5:10 PM, Stephen Gran <st...@piksel.com>
wrote:
> Hi,
>
> Yes - the imaginatively named slave2 was a zero-sized nm at that point -
> I am looking at how small a pool of reserved resource I can get away
> with, and use FGS for burst activity.
>
>
> Here are all the logs related to that host:port combination around that
> time:
>
> 2016-06-30 19:47:43,756 INFO
> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> Expired:slave2:24679 Timed out after 2 secs
> 2016-06-30 19:47:43,771 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> Deactivating Node slave2:24679 as it is now LOST
> 2016-06-30 19:47:43,771 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> slave2:24679 Node Transitioned from RUNNING to LOST
> 2016-06-30 19:47:43,909 INFO
> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> yarn_Container: [ContainerId: container_1467314892573_0009_01_000005,
> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> cpu and 1 mem.
> 2016-06-30 19:47:43,909 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> Released container container_1467314892573_0009_01_000005 of capacity
> <memory:2048, vCores:1> on host slave2:24679, which currently has 1
> containers, <memory:2048, vCores:1> used and <memory:2048, vCores:1>
> available, release resources=true
> 2016-06-30 19:47:43,909 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application attempt appattempt_1467314892573_0009_000001 released
> container container_1467314892573_0009_01_000005 on node: host:
> slave2:24679 #containers=1 available=<memory:2048, vCores:1>
> used=<memory:2048, vCores:1> with event: KILL
> 2016-06-30 19:47:43,909 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> Node not found resyncing slave2:24679
> 2016-06-30 19:47:43,952 INFO
> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> yarn_Container: [ContainerId: container_1467314892573_0009_01_000006,
> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> cpu and 1 mem.
> 2016-06-30 19:47:43,952 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> Released container container_1467314892573_0009_01_000006 of capacity
> <memory:2048, vCores:1> on host slave2:24679, which currently has 0
> containers, <memory:0, vCores:0> used and <memory:4096, vCores:2>
> available, release resources=true
> 2016-06-30 19:47:43,952 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application attempt appattempt_1467314892573_0009_000001 released
> container container_1467314892573_0009_01_000006 on node: host:
> slave2:24679 #containers=0 available=<memory:4096, vCores:2>
> used=<memory:0, vCores:0> with event: KILL
> 2016-06-30 19:47:43,952 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Removed node slave2:24679 cluster capacity: <memory:4096, vCores:4>
> 2016-06-30 19:47:47,573 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> slave2:24679 Node Transitioned from NEW to RUNNING
> 2016-06-30 19:47:47,936 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> NodeManager from node slave2(cmPort: 24679 httpPort: 23177) registered
> with capability: <memory:0, vCores:0>, assigned nodeId slave2:24679
>
>
> Looks like it did go into LOST for a bit.
>
> Cheers,
>
> On 30/06/16 21:36, Darin Johnson wrote:
> > Steven, thanks. I thought I had fixed that but perhaps a regression was
> > made in another merge. I'll look into it, can you answer a few
> questions?
> > Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
> > manager logs had it recently become unhealthy? I'm pretty concerned
> about
> > this and will try to get a patch soon.
> >
> > Thanks,
> >
> > Darin
> > On Jun 30, 2016 3:53 PM, "Stephen Gran" <st...@piksel.com> wrote:
> >
> >> Hi,
> >>
> >> Just playing with the 0.2.0 release (congratulations, by the way!)
> >>
> >> I have seen this twice now, although it is by no means consistent - I
> >> will have a dozen successful runs, and then one of these. This exits
> >> the RM, which makes it rather noticable.
> >>
> >> 2016-06-30 19:47:43,952 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
> >> s:4>
> >> 2016-06-30 19:47:43,953 FATAL
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> >> handling event type NODE_RESOURCE_UPDATE to the scheduler
> >> java.lang.NullPointerException
> >> at
> >>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
> >> at
> >>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
> >> at
> >>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
> >> at
> >>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
> >> at
> >>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
> >> at
> >>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2016-06-30 19:47:43,972 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
> >> bbye..
> >>
> >> --
> >> Stephen Gran
> >> Senior Technical Architect
> >>
> >> picture the possibilities | piksel.com
> >> This message is private and confidential. If you have received this
> >> message in error, please notify the sender or servicedesk@piksel.com
> and
> >> remove it from your system.
> >>
> >> Piksel Inc is a company registered in the United States New York City,
> >> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
> >>
> >
>
> --
> Stephen Gran
> Senior Technical Architect
>
> picture the possibilities | piksel.com
>
Re: NPE in removing container
Posted by Stephen Gran <st...@piksel.com>.
Hi,
Yes - the imaginatively named slave2 was a zero-sized nm at that point -
I am looking at how small a pool of reserved resource I can get away
with, and use FGS for burst activity.
Here are all the logs related to that host:port combination around that
time:
2016-06-30 19:47:43,756 INFO
org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
Expired:slave2:24679 Timed out after 2 secs
2016-06-30 19:47:43,771 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
Deactivating Node slave2:24679 as it is now LOST
2016-06-30 19:47:43,771 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
slave2:24679 Node Transitioned from RUNNING to LOST
2016-06-30 19:47:43,909 INFO
org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
yarn_Container: [ContainerId: container_1467314892573_0009_01_000005,
NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
<memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
cpu and 1 mem.
2016-06-30 19:47:43,909 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
Released container container_1467314892573_0009_01_000005 of capacity
<memory:2048, vCores:1> on host slave2:24679, which currently has 1
containers, <memory:2048, vCores:1> used and <memory:2048, vCores:1>
available, release resources=true
2016-06-30 19:47:43,909 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application attempt appattempt_1467314892573_0009_000001 released
container container_1467314892573_0009_01_000005 on node: host:
slave2:24679 #containers=1 available=<memory:2048, vCores:1>
used=<memory:2048, vCores:1> with event: KILL
2016-06-30 19:47:43,909 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
Node not found resyncing slave2:24679
2016-06-30 19:47:43,952 INFO
org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
yarn_Container: [ContainerId: container_1467314892573_0009_01_000006,
NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
<memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
cpu and 1 mem.
2016-06-30 19:47:43,952 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
Released container container_1467314892573_0009_01_000006 of capacity
<memory:2048, vCores:1> on host slave2:24679, which currently has 0
containers, <memory:0, vCores:0> used and <memory:4096, vCores:2>
available, release resources=true
2016-06-30 19:47:43,952 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application attempt appattempt_1467314892573_0009_000001 released
container container_1467314892573_0009_01_000006 on node: host:
slave2:24679 #containers=0 available=<memory:4096, vCores:2>
used=<memory:0, vCores:0> with event: KILL
2016-06-30 19:47:43,952 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Removed node slave2:24679 cluster capacity: <memory:4096, vCores:4>
2016-06-30 19:47:47,573 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
slave2:24679 Node Transitioned from NEW to RUNNING
2016-06-30 19:47:47,936 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
NodeManager from node slave2(cmPort: 24679 httpPort: 23177) registered
with capability: <memory:0, vCores:0>, assigned nodeId slave2:24679
Looks like it did go into LOST for a bit.
Cheers,
On 30/06/16 21:36, Darin Johnson wrote:
> Steven, thanks. I thought I had fixed that but perhaps a regression was
> made in another merge. I'll look into it, can you answer a few questions?
> Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
> manager logs had it recently become unhealthy? I'm pretty concerned about
> this and will try to get a patch soon.
>
> Thanks,
>
> Darin
> On Jun 30, 2016 3:53 PM, "Stephen Gran" <st...@piksel.com> wrote:
>
>> Hi,
>>
>> Just playing with the 0.2.0 release (congratulations, by the way!)
>>
>> I have seen this twice now, although it is by no means consistent - I
>> will have a dozen successful runs, and then one of these. This exits
>> the RM, which makes it rather noticable.
>>
>> 2016-06-30 19:47:43,952 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
>> s:4>
>> 2016-06-30 19:47:43,953 FATAL
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
>> handling event type NODE_RESOURCE_UPDATE to the scheduler
>> java.lang.NullPointerException
>> at
>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
>> at
>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
>> at
>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
>> at
>>
>> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
>> at
>>
>> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
>> at
>>
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-06-30 19:47:43,972 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
>> bbye..
>>
>> --
>> Stephen Gran
>> Senior Technical Architect
>>
>> picture the possibilities | piksel.com
>> This message is private and confidential. If you have received this
>> message in error, please notify the sender or servicedesk@piksel.com and
>> remove it from your system.
>>
>> Piksel Inc is a company registered in the United States New York City,
>> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
>>
>
--
Stephen Gran
Senior Technical Architect
picture the possibilities | piksel.com
Re: NPE in removing container
Posted by Darin Johnson <db...@gmail.com>.
Steven, thanks. I thought I had fixed that but perhaps a regression was
made in another merge. I'll look into it, can you answer a few questions?
Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
manager logs had it recently become unhealthy? I'm pretty concerned about
this and will try to get a patch soon.
Thanks,
Darin
On Jun 30, 2016 3:53 PM, "Stephen Gran" <st...@piksel.com> wrote:
> Hi,
>
> Just playing with the 0.2.0 release (congratulations, by the way!)
>
> I have seen this twice now, although it is by no means consistent - I
> will have a dozen successful runs, and then one of these. This exits
> the RM, which makes it rather noticable.
>
> 2016-06-30 19:47:43,952 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
> s:4>
> 2016-06-30 19:47:43,953 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type NODE_RESOURCE_UPDATE to the scheduler
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
> at
>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
> at
>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
> at
>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
> at java.lang.Thread.run(Thread.java:745)
> 2016-06-30 19:47:43,972 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting,
> bbye..
>
> --
> Stephen Gran
> Senior Technical Architect
>
> picture the possibilities | piksel.com
> This message is private and confidential. If you have received this
> message in error, please notify the sender or servicedesk@piksel.com and
> remove it from your system.
>
> Piksel Inc is a company registered in the United States New York City,
> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
>