You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Krishna Kishore Bonagiri <wr...@gmail.com> on 2012/10/18 12:30:23 UTC
Avoid Killing of YARN container on excess virtual memory usage
Hi,
Is there a way we can ask the YARN RM for not killing a container when it
uses excess virtual memory than the maximum it can use as per the
specification in the configuration file yarn-site.xml? We can't always
estimate the amount of virtual memory needed for our application running on
a container, but we don't want to get it killed in a case it exceeds the
maximum limit.
Please suggest as to how can we come across this issue.
Thanks,
Kishore
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.
Thanks,
Kishore
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > Hi,
> >
> > Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> > Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.
Thanks,
Kishore
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > Hi,
> >
> > Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> > Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
I filed https://issues.apache.org/jira/browse/YARN-168.
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
>> Hi,
>>
>> Is there a way we can ask the YARN RM for not killing a container when it
>> uses excess virtual memory than the maximum it can use as per the
>> specification in the configuration file yarn-site.xml? We can't always
>> estimate the amount of virtual memory needed for our application running on
>> a container, but we don't want to get it killed in a case it exceeds the
>> maximum limit.
>>
>> Please suggest as to how can we come across this issue.
>>
>> Thanks,
>> Kishore
>
>
>
> --
> Harsh J
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.
Thanks,
Kishore
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > Hi,
> >
> > Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> > Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Thanks Harsha, I was actually looking this part of the code in
ContainerMonitorImpl.java trying to find how can I make use of it, but as
you said it's a bug and doesn't seem to be really handling the case.
Thanks,
Kishore
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > Hi,
> >
> > Is there a way we can ask the YARN RM for not killing a container when
> it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running
> on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> > Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
>
>
>
> --
> Harsh J
>
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
I filed https://issues.apache.org/jira/browse/YARN-168.
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
>> Hi,
>>
>> Is there a way we can ask the YARN RM for not killing a container when it
>> uses excess virtual memory than the maximum it can use as per the
>> specification in the configuration file yarn-site.xml? We can't always
>> estimate the amount of virtual memory needed for our application running on
>> a container, but we don't want to get it killed in a case it exceeds the
>> maximum limit.
>>
>> Please suggest as to how can we come across this issue.
>>
>> Thanks,
>> Kishore
>
>
>
> --
> Harsh J
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
I filed https://issues.apache.org/jira/browse/YARN-168.
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
>> Hi,
>>
>> Is there a way we can ask the YARN RM for not killing a container when it
>> uses excess virtual memory than the maximum it can use as per the
>> specification in the configuration file yarn-site.xml? We can't always
>> estimate the amount of virtual memory needed for our application running on
>> a container, but we don't want to get it killed in a case it exceeds the
>> maximum limit.
>>
>> Please suggest as to how can we come across this issue.
>>
>> Thanks,
>> Kishore
>
>
>
> --
> Harsh J
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
I filed https://issues.apache.org/jira/browse/YARN-168.
On Thu, Oct 18, 2012 at 5:07 PM, Harsh J <ha...@cloudera.com> wrote:
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
>
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
>
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
>
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
>> Hi,
>>
>> Is there a way we can ask the YARN RM for not killing a container when it
>> uses excess virtual memory than the maximum it can use as per the
>> specification in the configuration file yarn-site.xml? We can't always
>> estimate the amount of virtual memory needed for our application running on
>> a container, but we don't want to get it killed in a case it exceeds the
>> maximum limit.
>>
>> Please suggest as to how can we come across this issue.
>>
>> Thanks,
>> Kishore
>
>
>
> --
> Harsh J
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
This is possible to do, but you've hit a bug with the current YARN
implementation. Ideally you should be able to configure the vmem-pmem
ratio (or an equivalent config) to be -1, to indicate disabling of
virtual memory checks completely (and there's indeed checks for this),
but it seems like we are enforcing the ratio to be at least 1.0 (and
hence negatives are disallowed).
You can't workaround by setting the NM's offered resource.mb to -1
either, as you'll lose out on controlling maximum allocations.
Please file a YARN bug on JIRA. The code at fault lies under
ContainersMonitorImpl#init(…).
On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> Hi,
>
> Is there a way we can ask the YARN RM for not killing a container when it
> uses excess virtual memory than the maximum it can use as per the
> specification in the configuration file yarn-site.xml? We can't always
> estimate the amount of virtual memory needed for our application running on
> a container, but we don't want to get it killed in a case it exceeds the
> maximum limit.
>
> Please suggest as to how can we come across this issue.
>
> Thanks,
> Kishore
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
This is possible to do, but you've hit a bug with the current YARN
implementation. Ideally you should be able to configure the vmem-pmem
ratio (or an equivalent config) to be -1, to indicate disabling of
virtual memory checks completely (and there's indeed checks for this),
but it seems like we are enforcing the ratio to be at least 1.0 (and
hence negatives are disallowed).
You can't workaround by setting the NM's offered resource.mb to -1
either, as you'll lose out on controlling maximum allocations.
Please file a YARN bug on JIRA. The code at fault lies under
ContainersMonitorImpl#init(…).
On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> Hi,
>
> Is there a way we can ask the YARN RM for not killing a container when it
> uses excess virtual memory than the maximum it can use as per the
> specification in the configuration file yarn-site.xml? We can't always
> estimate the amount of virtual memory needed for our application running on
> a container, but we don't want to get it killed in a case it exceeds the
> maximum limit.
>
> Please suggest as to how can we come across this issue.
>
> Thanks,
> Kishore
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
This is possible to do, but you've hit a bug with the current YARN
implementation. Ideally you should be able to configure the vmem-pmem
ratio (or an equivalent config) to be -1, to indicate disabling of
virtual memory checks completely (and there's indeed checks for this),
but it seems like we are enforcing the ratio to be at least 1.0 (and
hence negatives are disallowed).
You can't workaround by setting the NM's offered resource.mb to -1
either, as you'll lose out on controlling maximum allocations.
Please file a YARN bug on JIRA. The code at fault lies under
ContainersMonitorImpl#init(…).
On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> Hi,
>
> Is there a way we can ask the YARN RM for not killing a container when it
> uses excess virtual memory than the maximum it can use as per the
> specification in the configuration file yarn-site.xml? We can't always
> estimate the amount of virtual memory needed for our application running on
> a container, but we don't want to get it killed in a case it exceeds the
> maximum limit.
>
> Please suggest as to how can we come across this issue.
>
> Thanks,
> Kishore
--
Harsh J
Re: Avoid Killing of YARN container on excess virtual memory usage
Posted by Harsh J <ha...@cloudera.com>.
This is possible to do, but you've hit a bug with the current YARN
implementation. Ideally you should be able to configure the vmem-pmem
ratio (or an equivalent config) to be -1, to indicate disabling of
virtual memory checks completely (and there's indeed checks for this),
but it seems like we are enforcing the ratio to be at least 1.0 (and
hence negatives are disallowed).
You can't workaround by setting the NM's offered resource.mb to -1
either, as you'll lose out on controlling maximum allocations.
Please file a YARN bug on JIRA. The code at fault lies under
ContainersMonitorImpl#init(…).
On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
<wr...@gmail.com> wrote:
> Hi,
>
> Is there a way we can ask the YARN RM for not killing a container when it
> uses excess virtual memory than the maximum it can use as per the
> specification in the configuration file yarn-site.xml? We can't always
> estimate the amount of virtual memory needed for our application running on
> a container, but we don't want to get it killed in a case it exceeds the
> maximum limit.
>
> Please suggest as to how can we come across this issue.
>
> Thanks,
> Kishore
--
Harsh J