You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by LiFu He <he...@gmail.com> on 2020/05/15 23:04:49 UTC
Who took impalad's memory
Hi,
Recently I deployed an Impala(3.2) cluster on aarch64 platform with some
modifications, and then ran some big queries that are from TPC-DS. In the
meanwhile, I found the memory of Impalad was not released to OS even though
the queries were all finished. I looked through the impalad's metrics tab
and really can't figure out where these memory(50.33GB) gone. And according
to the decription of /memz, it seems the tcmalloc(2.5) had released its
memory to OS. Here are the /memz and /metrics info:
https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9
Any tips will be appreciated. Thanks in advance :)
--
该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
--
该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
Re: Who took impalad's memory
Posted by Tim Armstrong <ta...@cloudera.com>.
Great find! I think the linked patch that disables MADV_FREE would be good
to include in native-toolchain.
We have done the most intensive stress/perf testing for Impala on older
kernels, especially CentOS7/RHEL7, so I think always using MADV_DONTNEED is
safer than having different behaviour on different OSes. We could also run
into trouble with containers, if Impala is compiled in a container with a
newer userspace but running on an older host.
On Mon, May 18, 2020 at 6:05 AM LiFu He <he...@gmail.com> wrote:
> Thanks Tim ^_^
> I have found the answer:
> https://github.com/gperftools/gperftools/issues/780
> By the way, is it necessary to disable MADV_FREE while the linux
> kernel version is >= 4.5? If yes, I will add a patch to native-toolchain :)
>
>
> Tim Armstrong <ta...@cloudera.com> 于2020年5月16日周六 上午9:21写道：
>
> > That is strange, I would expect TCMalloc's unmapped bytes to be *not*
> > counted as part of the process RSS. We depend on TCMalloc's
> > tcmalloc.aggressive_memory_decommit setting to release all freed memory
> to
> > the OS. If the memory page size on your system is bigger than TCMalloc
> > expects, or something along those lines, it could cause those symptoms.
> >
> > On Fri, May 15, 2020 at 4:04 PM LiFu He <he...@gmail.com> wrote:
> >
> > > Hi,
> > > Recently I deployed an Impala(3.2) cluster on aarch64 platform with
> some
> > > modifications, and then ran some big queries that are from TPC-DS. In
> the
> > > meanwhile, I found the memory of Impalad was not released to OS even
> > though
> > > the queries were all finished. I looked through the impalad's metrics
> tab
> > > and really can't figure out where these memory(50.33GB) gone. And
> > according
> > > to the decription of /memz, it seems the tcmalloc(2.5) had released its
> > > memory to OS. Here are the /memz and /metrics info:
> > > https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9
> > > Any tips will be appreciated. Thanks in advance :)
> > >
> > > --
> >
> >
>
Re: Who took impalad's memory
Posted by LiFu He <he...@gmail.com>.
Thanks Tim ^_^
I have found the answer: https://github.com/gperftools/gperftools/issues/780
By the way, is it necessary to disable MADV_FREE while the linux
kernel version is >= 4.5? If yes, I will add a patch to native-toolchain :)
Tim Armstrong <ta...@cloudera.com> 于2020年5月16日周六 上午9:21写道：
> That is strange, I would expect TCMalloc's unmapped bytes to be *not*
> counted as part of the process RSS. We depend on TCMalloc's
> tcmalloc.aggressive_memory_decommit setting to release all freed memory to
> the OS. If the memory page size on your system is bigger than TCMalloc
> expects, or something along those lines, it could cause those symptoms.
>
> On Fri, May 15, 2020 at 4:04 PM LiFu He <he...@gmail.com> wrote:
>
> > Hi,
> > Recently I deployed an Impala(3.2) cluster on aarch64 platform with some
> > modifications, and then ran some big queries that are from TPC-DS. In the
> > meanwhile, I found the memory of Impalad was not released to OS even
> though
> > the queries were all finished. I looked through the impalad's metrics tab
> > and really can't figure out where these memory(50.33GB) gone. And
> according
> > to the decription of /memz, it seems the tcmalloc(2.5) had released its
> > memory to OS. Here are the /memz and /metrics info:
> > https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9
> > Any tips will be appreciated. Thanks in advance :)
> >
> > --
>
>
Re: Who took impalad's memory
Posted by Tim Armstrong <ta...@cloudera.com>.
That is strange, I would expect TCMalloc's unmapped bytes to be *not*
counted as part of the process RSS. We depend on TCMalloc's
tcmalloc.aggressive_memory_decommit setting to release all freed memory to
the OS. If the memory page size on your system is bigger than TCMalloc
expects, or something along those lines, it could cause those symptoms.
On Fri, May 15, 2020 at 4:04 PM LiFu He <he...@gmail.com> wrote:
> Hi,
> Recently I deployed an Impala(3.2) cluster on aarch64 platform with some
> modifications, and then ran some big queries that are from TPC-DS. In the
> meanwhile, I found the memory of Impalad was not released to OS even though
> the queries were all finished. I looked through the impalad's metrics tab
> and really can't figure out where these memory(50.33GB) gone. And according
> to the decription of /memz, it seems the tcmalloc(2.5) had released its
> memory to OS. Here are the /memz and /metrics info:
> https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9
> Any tips will be appreciated. Thanks in advance :)
>
> --
> 该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
> 重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
>
>
>
> --
> 该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
> 重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
>
Re: Who took impalad's memory
Posted by Tim Armstrong <ta...@cloudera.com>.
That is strange, I would expect TCMalloc's unmapped bytes to be *not*
counted as part of the process RSS. We depend on TCMalloc's
tcmalloc.aggressive_memory_decommit setting to release all freed memory to
the OS. If the memory page size on your system is bigger than TCMalloc
expects, or something along those lines, it could cause those symptoms.
On Fri, May 15, 2020 at 4:04 PM LiFu He <he...@gmail.com> wrote:
> Hi,
> Recently I deployed an Impala(3.2) cluster on aarch64 platform with some
> modifications, and then ran some big queries that are from TPC-DS. In the
> meanwhile, I found the memory of Impalad was not released to OS even though
> the queries were all finished. I looked through the impalad's metrics tab
> and really can't figure out where these memory(50.33GB) gone. And according
> to the decription of /memz, it seems the tcmalloc(2.5) had released its
> memory to OS. Here are the /memz and /metrics info:
> https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9
> Any tips will be appreciated. Thanks in advance :)
>
> --
> 该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
> 重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
>
>
>
> --
> 该不该搁下重重的壳 寻找到底哪里有蓝天 随着轻轻的风轻轻的飘 历经的伤都不感觉疼 我要一步一步往上爬 等待阳光静静看着它的脸 小小的天有大大的梦想
> 重重的壳裹着轻轻的仰望 我要一步一步往上爬 在最高点乘着叶片往前飞 任风吹干流过的泪和汗总有一天我属于我的天
>