Re: Completed Tasks in YARN will not release resources

[Henning Blohm <he...@zfabrik.de>]

Did you find YarnChild processes still running?

We had problems with Jobs that created non-daemon threads that would 
prevent the job VM from terminating orderly

(I wish the YARN-main of the job would make sure the VM terminates by 
calling system.exit (or at least make that configurable)).

Henning

On 04/30/2016 02:56 PM, Benjamin Kim wrote:
> Has anyone encountered this problem with YARN. It all started after an attempt to upgrade from CDH 5.4.8 to CDH 5.5.2.
>
> I ran jobs overnight, and they never completed. But, it did take down the YARN ResourceManager and multiple NodeManagers after 5 or 6 hours. There was one job that out of 450 mappers, only 64 completed, 386 pending, and 0 running. The pending mappers are in a Scheduled state.
>
> Each data node has 24 cores, 64 GB, 6 drives x 2 TB. NodeManager is allocated 45GB; Mappers are allocated 4GB (3.2GB Heap); Reducers are allocated 8GB (6.4GB Heap); AM is allocated 8GB (6.4 GB Heap).
>
> There was a note for each completed task attempt stating that it was in the finished state for too long. I believe that the task is not releasing the container when it\u2019s done or is not communicating it back.
>
> Does anyone have any ideas?
>
> Thanks,
> Ben
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org