You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Abdel Hakim Deneche <ad...@maprtech.com> on 2015/07/27 18:53:53 UTC
Suspicious direct memory consumption when running queries concurrently
When running a set of, mostly window function, queries concurrently on a
single drillbit with a 8GB max direct memory. We are seeing a continuous
increase of direct memory allocation.
We repeat the following steps multiple times:
- we launch in "iteration" of tests that will run all queries in a random
order, 10 queries at a time
- after the iteration finishes, we wait for a couple of minute to give
Drill time to release the memory being held by the finishing fragments
Using Drill's memory logger ("drill.allocator") we were able to get
snapshots of how memory was internally used by Netty, we only focused on
the number of allocated chunks, if we take this number and multiply it by
16MB (netty's chunk size) we get approximately the same value reported by
Drill's direct memory allocation.
Here is a graph that shows the evolution of the number of allocated chunks
on a 500 iterations run (I'm working on improving the plots) :
http://bit.ly/1JL6Kp3
In this specific case, after the first iteration Drill was allocating ~2GB
of direct memory, this number kept rising after each iteration to ~6GB. We
suspect this caused one of our previous runs to crash the JVM.
If we only focus on the log lines between iterations (when Drill's memory
usage is below 10MB) then all allocated chunks are at most 2% usage. At
some point we end up with 288 nearly empty chunks, yet the next iteration
will cause more chunks to be allocated!!!
is this expected ?
PS: I am running more tests and will update this thread with more
informations.
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Parth Chandra <pc...@maprtech.com>.
It is possible that the ownership of the memory chunks is being passed on
to different threads from the one's that are allocating and that is causing
the problem.
One of the issues in Netty was that if a memory allocation was passed from
one thread to another, the receiving thread would add the memory to its own
cache (even though the memory came from an arena associated with the first
thread) and the memory would not be released. This was addressed at some
point, but either that fix does not cover all cases, or we may have a
related issue that still needs to be addressed.
Do we have a trace of the arenas and their respective allocations between
two consecutive iterations?
Parth
On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> When running a set of, mostly window function, queries concurrently on a
> single drillbit with a 8GB max direct memory. We are seeing a continuous
> increase of direct memory allocation.
>
> We repeat the following steps multiple times:
> - we launch in "iteration" of tests that will run all queries in a random
> order, 10 queries at a time
> - after the iteration finishes, we wait for a couple of minute to give
> Drill time to release the memory being held by the finishing fragments
>
> Using Drill's memory logger ("drill.allocator") we were able to get
> snapshots of how memory was internally used by Netty, we only focused on
> the number of allocated chunks, if we take this number and multiply it by
> 16MB (netty's chunk size) we get approximately the same value reported by
> Drill's direct memory allocation.
> Here is a graph that shows the evolution of the number of allocated chunks
> on a 500 iterations run (I'm working on improving the plots) :
>
> http://bit.ly/1JL6Kp3
>
> In this specific case, after the first iteration Drill was allocating ~2GB
> of direct memory, this number kept rising after each iteration to ~6GB. We
> suspect this caused one of our previous runs to crash the JVM.
>
> If we only focus on the log lines between iterations (when Drill's memory
> usage is below 10MB) then all allocated chunks are at most 2% usage. At
> some point we end up with 288 nearly empty chunks, yet the next iteration
> will cause more chunks to be allocated!!!
>
> is this expected ?
>
> PS: I am running more tests and will update this thread with more
> informations.
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Yes, and also why do we keep creating new chunks ? how come the first
iteration was able to run with less than half the direct memory used by the
later iterations ? I thought maybe the chunks weren't distributed equitably
between Netty's memory arenas, so new iterations may actually hit other
direct arenas, but I'm not sure this is the case.
We are trying to figure out a small set of queries to help reproduce this
issue more easily, and will run a Jmap to get more informations.
On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> It sounds like your statement is that we're cacheing too many unused
> chunks. Hanifi and I previously discussed implementing a separate flushing
> mechanism to release unallocated chunks that are hanging around. The main
> question is, why are so many chunks hanging around and what threads are
> they associated with. A Jmap dump and analysis should allow you to do
> determine which thread owns the excess chunks. My guess would be the RPC
> pool since those are long lasting (as opposed to the WorkManager pool,
> which is contracting).
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > When running a set of, mostly window function, queries concurrently on a
> > single drillbit with a 8GB max direct memory. We are seeing a continuous
> > increase of direct memory allocation.
> >
> > We repeat the following steps multiple times:
> > - we launch in "iteration" of tests that will run all queries in a random
> > order, 10 queries at a time
> > - after the iteration finishes, we wait for a couple of minute to give
> > Drill time to release the memory being held by the finishing fragments
> >
> > Using Drill's memory logger ("drill.allocator") we were able to get
> > snapshots of how memory was internally used by Netty, we only focused on
> > the number of allocated chunks, if we take this number and multiply it by
> > 16MB (netty's chunk size) we get approximately the same value reported by
> > Drill's direct memory allocation.
> > Here is a graph that shows the evolution of the number of allocated
> chunks
> > on a 500 iterations run (I'm working on improving the plots) :
> >
> > http://bit.ly/1JL6Kp3
> >
> > In this specific case, after the first iteration Drill was allocating
> ~2GB
> > of direct memory, this number kept rising after each iteration to ~6GB.
> We
> > suspect this caused one of our previous runs to crash the JVM.
> >
> > If we only focus on the log lines between iterations (when Drill's memory
> > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > some point we end up with 288 nearly empty chunks, yet the next iteration
> > will cause more chunks to be allocated!!!
> >
> > is this expected ?
> >
> > PS: I am running more tests and will update this thread with more
> > informations.
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
I should be able to do that, Vicky was able to reproduce it with a single
query run concurrently. Let me try to find it
On Sun, Aug 2, 2015 at 9:11 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> If you give me 5 sample queries, a simple harness should be easy to
> create.
> On Aug 2, 2015 9:06 AM, "Abdel Hakim Deneche" <ad...@maprtech.com>
> wrote:
>
> > @yulia,
> > Just checked the available disk space and there is more than enough for
> the
> > dump :(
> >
> > @Jacques,
> > You'll need the test framework to be able to reproduce this. I'm already
> > using one single node and it's just a matter of running a bunch of window
> > function queries concurrently and repeat this a lot.
> >
> > Looking at the memory growth it seems to become stable after some time,
> > which may suggest it's not a memory leak, I still have 3 questions I will
> > try to find answers for:
> > - why Netty doesn't release memory chunks when no queries are running (up
> > to 5GB if you run enough iterations)
> > - are all those allocated chunks being used when you run one more
> > iteration, or does Netty only use some of them and leave the rest
> allocated
> > for no reason ? (I should be able to get this from the memory logs I
> > already have)
> > - is this an "expected" behavior of Netty's allocator and we should just
> > learn to live with it ?
> >
> >
> > On Fri, Jul 31, 2015 at 10:40 PM, yuliya Feldman <
> > yufeldman@yahoo.com.invalid> wrote:
> >
> > > How much memory your jvm is taking?
> > > Do you even have enough disk space to dump it.
> > > From: Abdel Hakim Deneche <ad...@maprtech.com>
> > > To: "dev@drill.apache.org" <de...@drill.apache.org>
> > > Sent: Friday, July 31, 2015 9:19 PM
> > > Subject: Re: Suspicious direct memory consumption when running queries
> > > concurrently
> > >
> > > I tried getting a jmap dump multiple times without success, each time
> it
> > > crashes the jvm with the following exception:
> > >
> > > Dumping heap to
> /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> > > > ...
> > > > Exception in thread "main" java.io.IOException: Premature EOF
> > > > at
> > > >
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> > > > at
> > > >
> > >
> >
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> > > > at
> > > >
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> > > > at
> > > >
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> > > > at sun.tools.jmap.JMap.dump(JMap.java:242)
> > > > at sun.tools.jmap.JMap.main(JMap.java:140)
> > >
> > >
> > > On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com>
> > > wrote:
> > >
> > > > A allocate -> release cycle all on the same thread goes into a per
> > thread
> > > > cache.
> > > >
> > > > A bunch of Netty arena settings are configurable. The big issue I
> > > believe
> > > > is that the limits are soft limits implemented by the allocation-time
> > > > release mechanism. As such, if you allocate a bunch of memory, then
> > > > release it all, that won't necessarily trigger any actual chunk
> > releases.
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> > > > adeneche@maprtech.com
> > > > > wrote:
> > > >
> > > > > @Jacques, my understanding is that chunks are not owned by
> specific a
> > > > > thread but they are part of a specific memory arena which is in
> turn
> > > only
> > > > > accessed by specific threads. Do you want me to find which threads
> > are
> > > > > associated with the same arena where we have hanging chunks ?
> > > > >
> > > > >
> > > > > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <
> jacques@dremio.com
> > >
> > > > > wrote:
> > > > >
> > > > > > It sounds like your statement is that we're cacheing too many
> > unused
> > > > > > chunks. Hanifi and I previously discussed implementing a
> separate
> > > > > flushing
> > > > > > mechanism to release unallocated chunks that are hanging around.
> > The
> > > > > main
> > > > > > question is, why are so many chunks hanging around and what
> threads
> > > are
> > > > > > they associated with. A Jmap dump and analysis should allow you
> to
> > > do
> > > > > > determine which thread owns the excess chunks. My guess would be
> > the
> > > > RPC
> > > > > > pool since those are long lasting (as opposed to the WorkManager
> > > pool,
> > > > > > which is contracting).
> > > > > >
> > > > > > --
> > > > > > Jacques Nadeau
> > > > > > CTO and Co-Founder, Dremio
> > > > > >
> > > > > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > > > > adeneche@maprtech.com>
> > > > > > wrote:
> > > > > >
> > > > > > > When running a set of, mostly window function, queries
> > concurrently
> > > > on
> > > > > a
> > > > > > > single drillbit with a 8GB max direct memory. We are seeing a
> > > > > continuous
> > > > > > > increase of direct memory allocation.
> > > > > > >
> > > > > > > We repeat the following steps multiple times:
> > > > > > > - we launch in "iteration" of tests that will run all queries
> in
> > a
> > > > > random
> > > > > > > order, 10 queries at a time
> > > > > > > - after the iteration finishes, we wait for a couple of minute
> to
> > > > give
> > > > > > > Drill time to release the memory being held by the finishing
> > > > fragments
> > > > > > >
> > > > > > > Using Drill's memory logger ("drill.allocator") we were able to
> > get
> > > > > > > snapshots of how memory was internally used by Netty, we only
> > > focused
> > > > > on
> > > > > > > the number of allocated chunks, if we take this number and
> > multiply
> > > > it
> > > > > by
> > > > > > > 16MB (netty's chunk size) we get approximately the same value
> > > > reported
> > > > > by
> > > > > > > Drill's direct memory allocation.
> > > > > > > Here is a graph that shows the evolution of the number of
> > allocated
> > > > > > chunks
> > > > > > > on a 500 iterations run (I'm working on improving the plots) :
> > > > > > >
> > > > > > > http://bit.ly/1JL6Kp3
> > > > > > >
> > > > > > > In this specific case, after the first iteration Drill was
> > > allocating
> > > > > > ~2GB
> > > > > > > of direct memory, this number kept rising after each iteration
> to
> > > > ~6GB.
> > > > > > We
> > > > > > > suspect this caused one of our previous runs to crash the JVM.
> > > > > > >
> > > > > > > If we only focus on the log lines between iterations (when
> > Drill's
> > > > > memory
> > > > > > > usage is below 10MB) then all allocated chunks are at most 2%
> > > usage.
> > > > At
> > > > > > > some point we end up with 288 nearly empty chunks, yet the next
> > > > > iteration
> > > > > > > will cause more chunks to be allocated!!!
> > > > > > >
> > > > > > > is this expected ?
> > > > > > >
> > > > > > > PS: I am running more tests and will update this thread with
> more
> > > > > > > informations.
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Abdelhakim Deneche
> > > > > > >
> > > > > > > Software Engineer
> > > > > > >
> > > > > > > <http://www.mapr.com/>
> > > > > > >
> > > > > > >
> > > > > > > Now Available - Free Hadoop On-Demand Training
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Abdelhakim Deneche
> > > > >
> > > > > Software Engineer
> > > > >
> > > > > <http://www.mapr.com/>
> > > > >
> > > > >
> > > > > Now Available - Free Hadoop On-Demand Training
> > > > > <
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Jacques Nadeau <ja...@dremio.com>.
If you give me 5 sample queries, a simple harness should be easy to
create.
On Aug 2, 2015 9:06 AM, "Abdel Hakim Deneche" <ad...@maprtech.com> wrote:
> @yulia,
> Just checked the available disk space and there is more than enough for the
> dump :(
>
> @Jacques,
> You'll need the test framework to be able to reproduce this. I'm already
> using one single node and it's just a matter of running a bunch of window
> function queries concurrently and repeat this a lot.
>
> Looking at the memory growth it seems to become stable after some time,
> which may suggest it's not a memory leak, I still have 3 questions I will
> try to find answers for:
> - why Netty doesn't release memory chunks when no queries are running (up
> to 5GB if you run enough iterations)
> - are all those allocated chunks being used when you run one more
> iteration, or does Netty only use some of them and leave the rest allocated
> for no reason ? (I should be able to get this from the memory logs I
> already have)
> - is this an "expected" behavior of Netty's allocator and we should just
> learn to live with it ?
>
>
> On Fri, Jul 31, 2015 at 10:40 PM, yuliya Feldman <
> yufeldman@yahoo.com.invalid> wrote:
>
> > How much memory your jvm is taking?
> > Do you even have enough disk space to dump it.
> > From: Abdel Hakim Deneche <ad...@maprtech.com>
> > To: "dev@drill.apache.org" <de...@drill.apache.org>
> > Sent: Friday, July 31, 2015 9:19 PM
> > Subject: Re: Suspicious direct memory consumption when running queries
> > concurrently
> >
> > I tried getting a jmap dump multiple times without success, each time it
> > crashes the jvm with the following exception:
> >
> > Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> > > ...
> > > Exception in thread "main" java.io.IOException: Premature EOF
> > > at
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> > > at
> > >
> >
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> > > at
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> > > at
> > >
> >
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> > > at sun.tools.jmap.JMap.dump(JMap.java:242)
> > > at sun.tools.jmap.JMap.main(JMap.java:140)
> >
> >
> > On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > A allocate -> release cycle all on the same thread goes into a per
> thread
> > > cache.
> > >
> > > A bunch of Netty arena settings are configurable. The big issue I
> > believe
> > > is that the limits are soft limits implemented by the allocation-time
> > > release mechanism. As such, if you allocate a bunch of memory, then
> > > release it all, that won't necessarily trigger any actual chunk
> releases.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> > > adeneche@maprtech.com
> > > > wrote:
> > >
> > > > @Jacques, my understanding is that chunks are not owned by specific a
> > > > thread but they are part of a specific memory arena which is in turn
> > only
> > > > accessed by specific threads. Do you want me to find which threads
> are
> > > > associated with the same arena where we have hanging chunks ?
> > > >
> > > >
> > > > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <jacques@dremio.com
> >
> > > > wrote:
> > > >
> > > > > It sounds like your statement is that we're cacheing too many
> unused
> > > > > chunks. Hanifi and I previously discussed implementing a separate
> > > > flushing
> > > > > mechanism to release unallocated chunks that are hanging around.
> The
> > > > main
> > > > > question is, why are so many chunks hanging around and what threads
> > are
> > > > > they associated with. A Jmap dump and analysis should allow you to
> > do
> > > > > determine which thread owns the excess chunks. My guess would be
> the
> > > RPC
> > > > > pool since those are long lasting (as opposed to the WorkManager
> > pool,
> > > > > which is contracting).
> > > > >
> > > > > --
> > > > > Jacques Nadeau
> > > > > CTO and Co-Founder, Dremio
> > > > >
> > > > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > > > adeneche@maprtech.com>
> > > > > wrote:
> > > > >
> > > > > > When running a set of, mostly window function, queries
> concurrently
> > > on
> > > > a
> > > > > > single drillbit with a 8GB max direct memory. We are seeing a
> > > > continuous
> > > > > > increase of direct memory allocation.
> > > > > >
> > > > > > We repeat the following steps multiple times:
> > > > > > - we launch in "iteration" of tests that will run all queries in
> a
> > > > random
> > > > > > order, 10 queries at a time
> > > > > > - after the iteration finishes, we wait for a couple of minute to
> > > give
> > > > > > Drill time to release the memory being held by the finishing
> > > fragments
> > > > > >
> > > > > > Using Drill's memory logger ("drill.allocator") we were able to
> get
> > > > > > snapshots of how memory was internally used by Netty, we only
> > focused
> > > > on
> > > > > > the number of allocated chunks, if we take this number and
> multiply
> > > it
> > > > by
> > > > > > 16MB (netty's chunk size) we get approximately the same value
> > > reported
> > > > by
> > > > > > Drill's direct memory allocation.
> > > > > > Here is a graph that shows the evolution of the number of
> allocated
> > > > > chunks
> > > > > > on a 500 iterations run (I'm working on improving the plots) :
> > > > > >
> > > > > > http://bit.ly/1JL6Kp3
> > > > > >
> > > > > > In this specific case, after the first iteration Drill was
> > allocating
> > > > > ~2GB
> > > > > > of direct memory, this number kept rising after each iteration to
> > > ~6GB.
> > > > > We
> > > > > > suspect this caused one of our previous runs to crash the JVM.
> > > > > >
> > > > > > If we only focus on the log lines between iterations (when
> Drill's
> > > > memory
> > > > > > usage is below 10MB) then all allocated chunks are at most 2%
> > usage.
> > > At
> > > > > > some point we end up with 288 nearly empty chunks, yet the next
> > > > iteration
> > > > > > will cause more chunks to be allocated!!!
> > > > > >
> > > > > > is this expected ?
> > > > > >
> > > > > > PS: I am running more tests and will update this thread with more
> > > > > > informations.
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Abdelhakim Deneche
> > > > > >
> > > > > > Software Engineer
> > > > > >
> > > > > > <http://www.mapr.com/>
> > > > > >
> > > > > >
> > > > > > Now Available - Free Hadoop On-Demand Training
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Abdelhakim Deneche
> > > >
> > > > Software Engineer
> > > >
> > > > <http://www.mapr.com/>
> > > >
> > > >
> > > > Now Available - Free Hadoop On-Demand Training
> > > > <
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
> >
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
@yulia,
Just checked the available disk space and there is more than enough for the
dump :(
@Jacques,
You'll need the test framework to be able to reproduce this. I'm already
using one single node and it's just a matter of running a bunch of window
function queries concurrently and repeat this a lot.
Looking at the memory growth it seems to become stable after some time,
which may suggest it's not a memory leak, I still have 3 questions I will
try to find answers for:
- why Netty doesn't release memory chunks when no queries are running (up
to 5GB if you run enough iterations)
- are all those allocated chunks being used when you run one more
iteration, or does Netty only use some of them and leave the rest allocated
for no reason ? (I should be able to get this from the memory logs I
already have)
- is this an "expected" behavior of Netty's allocator and we should just
learn to live with it ?
On Fri, Jul 31, 2015 at 10:40 PM, yuliya Feldman <
yufeldman@yahoo.com.invalid> wrote:
> How much memory your jvm is taking?
> Do you even have enough disk space to dump it.
> From: Abdel Hakim Deneche <ad...@maprtech.com>
> To: "dev@drill.apache.org" <de...@drill.apache.org>
> Sent: Friday, July 31, 2015 9:19 PM
> Subject: Re: Suspicious direct memory consumption when running queries
> concurrently
>
> I tried getting a jmap dump multiple times without success, each time it
> crashes the jvm with the following exception:
>
> Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> > ...
> > Exception in thread "main" java.io.IOException: Premature EOF
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> > at
> >
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> > at sun.tools.jmap.JMap.dump(JMap.java:242)
> > at sun.tools.jmap.JMap.main(JMap.java:140)
>
>
> On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > A allocate -> release cycle all on the same thread goes into a per thread
> > cache.
> >
> > A bunch of Netty arena settings are configurable. The big issue I
> believe
> > is that the limits are soft limits implemented by the allocation-time
> > release mechanism. As such, if you allocate a bunch of memory, then
> > release it all, that won't necessarily trigger any actual chunk releases.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> > adeneche@maprtech.com
> > > wrote:
> >
> > > @Jacques, my understanding is that chunks are not owned by specific a
> > > thread but they are part of a specific memory arena which is in turn
> only
> > > accessed by specific threads. Do you want me to find which threads are
> > > associated with the same arena where we have hanging chunks ?
> > >
> > >
> > > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> > > wrote:
> > >
> > > > It sounds like your statement is that we're cacheing too many unused
> > > > chunks. Hanifi and I previously discussed implementing a separate
> > > flushing
> > > > mechanism to release unallocated chunks that are hanging around. The
> > > main
> > > > question is, why are so many chunks hanging around and what threads
> are
> > > > they associated with. A Jmap dump and analysis should allow you to
> do
> > > > determine which thread owns the excess chunks. My guess would be the
> > RPC
> > > > pool since those are long lasting (as opposed to the WorkManager
> pool,
> > > > which is contracting).
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > > adeneche@maprtech.com>
> > > > wrote:
> > > >
> > > > > When running a set of, mostly window function, queries concurrently
> > on
> > > a
> > > > > single drillbit with a 8GB max direct memory. We are seeing a
> > > continuous
> > > > > increase of direct memory allocation.
> > > > >
> > > > > We repeat the following steps multiple times:
> > > > > - we launch in "iteration" of tests that will run all queries in a
> > > random
> > > > > order, 10 queries at a time
> > > > > - after the iteration finishes, we wait for a couple of minute to
> > give
> > > > > Drill time to release the memory being held by the finishing
> > fragments
> > > > >
> > > > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > > > snapshots of how memory was internally used by Netty, we only
> focused
> > > on
> > > > > the number of allocated chunks, if we take this number and multiply
> > it
> > > by
> > > > > 16MB (netty's chunk size) we get approximately the same value
> > reported
> > > by
> > > > > Drill's direct memory allocation.
> > > > > Here is a graph that shows the evolution of the number of allocated
> > > > chunks
> > > > > on a 500 iterations run (I'm working on improving the plots) :
> > > > >
> > > > > http://bit.ly/1JL6Kp3
> > > > >
> > > > > In this specific case, after the first iteration Drill was
> allocating
> > > > ~2GB
> > > > > of direct memory, this number kept rising after each iteration to
> > ~6GB.
> > > > We
> > > > > suspect this caused one of our previous runs to crash the JVM.
> > > > >
> > > > > If we only focus on the log lines between iterations (when Drill's
> > > memory
> > > > > usage is below 10MB) then all allocated chunks are at most 2%
> usage.
> > At
> > > > > some point we end up with 288 nearly empty chunks, yet the next
> > > iteration
> > > > > will cause more chunks to be allocated!!!
> > > > >
> > > > > is this expected ?
> > > > >
> > > > > PS: I am running more tests and will update this thread with more
> > > > > informations.
> > > > >
> > > > > --
> > > > >
> > > > > Abdelhakim Deneche
> > > > >
> > > > > Software Engineer
> > > > >
> > > > > <http://www.mapr.com/>
> > > > >
> > > > >
> > > > > Now Available - Free Hadoop On-Demand Training
> > > > > <
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>
>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
>
>
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries
concurrently
Posted by yuliya Feldman <yu...@yahoo.com.INVALID>.
How much memory your jvm is taking?
Do you even have enough disk space to dump it.
From: Abdel Hakim Deneche <ad...@maprtech.com>
To: "dev@drill.apache.org" <de...@drill.apache.org>
Sent: Friday, July 31, 2015 9:19 PM
Subject: Re: Suspicious direct memory consumption when running queries concurrently
I tried getting a jmap dump multiple times without success, each time it
crashes the jvm with the following exception:
Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> ...
> Exception in thread "main" java.io.IOException: Premature EOF
> at
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> at
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> at
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> at
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> at sun.tools.jmap.JMap.dump(JMap.java:242)
> at sun.tools.jmap.JMap.main(JMap.java:140)
On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> A allocate -> release cycle all on the same thread goes into a per thread
> cache.
>
> A bunch of Netty arena settings are configurable. The big issue I believe
> is that the limits are soft limits implemented by the allocation-time
> release mechanism. As such, if you allocate a bunch of memory, then
> release it all, that won't necessarily trigger any actual chunk releases.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> > wrote:
>
> > @Jacques, my understanding is that chunks are not owned by specific a
> > thread but they are part of a specific memory arena which is in turn only
> > accessed by specific threads. Do you want me to find which threads are
> > associated with the same arena where we have hanging chunks ?
> >
> >
> > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > It sounds like your statement is that we're cacheing too many unused
> > > chunks. Hanifi and I previously discussed implementing a separate
> > flushing
> > > mechanism to release unallocated chunks that are hanging around. The
> > main
> > > question is, why are so many chunks hanging around and what threads are
> > > they associated with. A Jmap dump and analysis should allow you to do
> > > determine which thread owns the excess chunks. My guess would be the
> RPC
> > > pool since those are long lasting (as opposed to the WorkManager pool,
> > > which is contracting).
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > adeneche@maprtech.com>
> > > wrote:
> > >
> > > > When running a set of, mostly window function, queries concurrently
> on
> > a
> > > > single drillbit with a 8GB max direct memory. We are seeing a
> > continuous
> > > > increase of direct memory allocation.
> > > >
> > > > We repeat the following steps multiple times:
> > > > - we launch in "iteration" of tests that will run all queries in a
> > random
> > > > order, 10 queries at a time
> > > > - after the iteration finishes, we wait for a couple of minute to
> give
> > > > Drill time to release the memory being held by the finishing
> fragments
> > > >
> > > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > > snapshots of how memory was internally used by Netty, we only focused
> > on
> > > > the number of allocated chunks, if we take this number and multiply
> it
> > by
> > > > 16MB (netty's chunk size) we get approximately the same value
> reported
> > by
> > > > Drill's direct memory allocation.
> > > > Here is a graph that shows the evolution of the number of allocated
> > > chunks
> > > > on a 500 iterations run (I'm working on improving the plots) :
> > > >
> > > > http://bit.ly/1JL6Kp3
> > > >
> > > > In this specific case, after the first iteration Drill was allocating
> > > ~2GB
> > > > of direct memory, this number kept rising after each iteration to
> ~6GB.
> > > We
> > > > suspect this caused one of our previous runs to crash the JVM.
> > > >
> > > > If we only focus on the log lines between iterations (when Drill's
> > memory
> > > > usage is below 10MB) then all allocated chunks are at most 2% usage.
> At
> > > > some point we end up with 288 nearly empty chunks, yet the next
> > iteration
> > > > will cause more chunks to be allocated!!!
> > > >
> > > > is this expected ?
> > > >
> > > > PS: I am running more tests and will update this thread with more
> > > > informations.
> > > >
> > > > --
> > > >
> > > > Abdelhakim Deneche
> > > >
> > > > Software Engineer
> > > >
> > > > <http://www.mapr.com/>
> > > >
> > > >
> > > > Now Available - Free Hadoop On-Demand Training
> > > > <
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Jacques Nadeau <ja...@dremio.com>.
For the memory leak, not the jmap issue.
On Jul 31, 2015 9:50 PM, "Jacques Nadeau" <ja...@dremio.com> wrote:
> Can you give me a single node repro?
> On Jul 31, 2015 9:20 PM, "Abdel Hakim Deneche" <ad...@maprtech.com>
> wrote:
>
>> I tried getting a jmap dump multiple times without success, each time it
>> crashes the jvm with the following exception:
>>
>> Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
>> > ...
>> > Exception in thread "main" java.io.IOException: Premature EOF
>> > at
>> >
>> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
>> > at
>> >
>> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
>> > at
>> >
>> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
>> > at
>> >
>> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
>> > at sun.tools.jmap.JMap.dump(JMap.java:242)
>> > at sun.tools.jmap.JMap.main(JMap.java:140)
>>
>>
>> On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com>
>> wrote:
>>
>> > A allocate -> release cycle all on the same thread goes into a per
>> thread
>> > cache.
>> >
>> > A bunch of Netty arena settings are configurable. The big issue I
>> believe
>> > is that the limits are soft limits implemented by the allocation-time
>> > release mechanism. As such, if you allocate a bunch of memory, then
>> > release it all, that won't necessarily trigger any actual chunk
>> releases.
>> >
>> > --
>> > Jacques Nadeau
>> > CTO and Co-Founder, Dremio
>> >
>> > On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
>> > adeneche@maprtech.com
>> > > wrote:
>> >
>> > > @Jacques, my understanding is that chunks are not owned by specific a
>> > > thread but they are part of a specific memory arena which is in turn
>> only
>> > > accessed by specific threads. Do you want me to find which threads are
>> > > associated with the same arena where we have hanging chunks ?
>> > >
>> > >
>> > > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
>> > > wrote:
>> > >
>> > > > It sounds like your statement is that we're cacheing too many unused
>> > > > chunks. Hanifi and I previously discussed implementing a separate
>> > > flushing
>> > > > mechanism to release unallocated chunks that are hanging around.
>> The
>> > > main
>> > > > question is, why are so many chunks hanging around and what threads
>> are
>> > > > they associated with. A Jmap dump and analysis should allow you to
>> do
>> > > > determine which thread owns the excess chunks. My guess would be
>> the
>> > RPC
>> > > > pool since those are long lasting (as opposed to the WorkManager
>> pool,
>> > > > which is contracting).
>> > > >
>> > > > --
>> > > > Jacques Nadeau
>> > > > CTO and Co-Founder, Dremio
>> > > >
>> > > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
>> > > > adeneche@maprtech.com>
>> > > > wrote:
>> > > >
>> > > > > When running a set of, mostly window function, queries
>> concurrently
>> > on
>> > > a
>> > > > > single drillbit with a 8GB max direct memory. We are seeing a
>> > > continuous
>> > > > > increase of direct memory allocation.
>> > > > >
>> > > > > We repeat the following steps multiple times:
>> > > > > - we launch in "iteration" of tests that will run all queries in a
>> > > random
>> > > > > order, 10 queries at a time
>> > > > > - after the iteration finishes, we wait for a couple of minute to
>> > give
>> > > > > Drill time to release the memory being held by the finishing
>> > fragments
>> > > > >
>> > > > > Using Drill's memory logger ("drill.allocator") we were able to
>> get
>> > > > > snapshots of how memory was internally used by Netty, we only
>> focused
>> > > on
>> > > > > the number of allocated chunks, if we take this number and
>> multiply
>> > it
>> > > by
>> > > > > 16MB (netty's chunk size) we get approximately the same value
>> > reported
>> > > by
>> > > > > Drill's direct memory allocation.
>> > > > > Here is a graph that shows the evolution of the number of
>> allocated
>> > > > chunks
>> > > > > on a 500 iterations run (I'm working on improving the plots) :
>> > > > >
>> > > > > http://bit.ly/1JL6Kp3
>> > > > >
>> > > > > In this specific case, after the first iteration Drill was
>> allocating
>> > > > ~2GB
>> > > > > of direct memory, this number kept rising after each iteration to
>> > ~6GB.
>> > > > We
>> > > > > suspect this caused one of our previous runs to crash the JVM.
>> > > > >
>> > > > > If we only focus on the log lines between iterations (when Drill's
>> > > memory
>> > > > > usage is below 10MB) then all allocated chunks are at most 2%
>> usage.
>> > At
>> > > > > some point we end up with 288 nearly empty chunks, yet the next
>> > > iteration
>> > > > > will cause more chunks to be allocated!!!
>> > > > >
>> > > > > is this expected ?
>> > > > >
>> > > > > PS: I am running more tests and will update this thread with more
>> > > > > informations.
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Abdelhakim Deneche
>> > > > >
>> > > > > Software Engineer
>> > > > >
>> > > > > <http://www.mapr.com/>
>> > > > >
>> > > > >
>> > > > > Now Available - Free Hadoop On-Demand Training
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Abdelhakim Deneche
>> > >
>> > > Software Engineer
>> > >
>> > > <http://www.mapr.com/>
>> > >
>> > >
>> > > Now Available - Free Hadoop On-Demand Training
>> > > <
>> > >
>> >
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>>
>> Abdelhakim Deneche
>>
>> Software Engineer
>>
>> <http://www.mapr.com/>
>>
>>
>> Now Available - Free Hadoop On-Demand Training
>> <
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>> >
>>
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Jacques Nadeau <ja...@dremio.com>.
Can you give me a single node repro?
On Jul 31, 2015 9:20 PM, "Abdel Hakim Deneche" <ad...@maprtech.com>
wrote:
> I tried getting a jmap dump multiple times without success, each time it
> crashes the jvm with the following exception:
>
> Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> > ...
> > Exception in thread "main" java.io.IOException: Premature EOF
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> > at
> >
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> > at
> >
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> > at sun.tools.jmap.JMap.dump(JMap.java:242)
> > at sun.tools.jmap.JMap.main(JMap.java:140)
>
>
> On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > A allocate -> release cycle all on the same thread goes into a per thread
> > cache.
> >
> > A bunch of Netty arena settings are configurable. The big issue I
> believe
> > is that the limits are soft limits implemented by the allocation-time
> > release mechanism. As such, if you allocate a bunch of memory, then
> > release it all, that won't necessarily trigger any actual chunk releases.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> > adeneche@maprtech.com
> > > wrote:
> >
> > > @Jacques, my understanding is that chunks are not owned by specific a
> > > thread but they are part of a specific memory arena which is in turn
> only
> > > accessed by specific threads. Do you want me to find which threads are
> > > associated with the same arena where we have hanging chunks ?
> > >
> > >
> > > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> > > wrote:
> > >
> > > > It sounds like your statement is that we're cacheing too many unused
> > > > chunks. Hanifi and I previously discussed implementing a separate
> > > flushing
> > > > mechanism to release unallocated chunks that are hanging around. The
> > > main
> > > > question is, why are so many chunks hanging around and what threads
> are
> > > > they associated with. A Jmap dump and analysis should allow you to
> do
> > > > determine which thread owns the excess chunks. My guess would be the
> > RPC
> > > > pool since those are long lasting (as opposed to the WorkManager
> pool,
> > > > which is contracting).
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > > adeneche@maprtech.com>
> > > > wrote:
> > > >
> > > > > When running a set of, mostly window function, queries concurrently
> > on
> > > a
> > > > > single drillbit with a 8GB max direct memory. We are seeing a
> > > continuous
> > > > > increase of direct memory allocation.
> > > > >
> > > > > We repeat the following steps multiple times:
> > > > > - we launch in "iteration" of tests that will run all queries in a
> > > random
> > > > > order, 10 queries at a time
> > > > > - after the iteration finishes, we wait for a couple of minute to
> > give
> > > > > Drill time to release the memory being held by the finishing
> > fragments
> > > > >
> > > > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > > > snapshots of how memory was internally used by Netty, we only
> focused
> > > on
> > > > > the number of allocated chunks, if we take this number and multiply
> > it
> > > by
> > > > > 16MB (netty's chunk size) we get approximately the same value
> > reported
> > > by
> > > > > Drill's direct memory allocation.
> > > > > Here is a graph that shows the evolution of the number of allocated
> > > > chunks
> > > > > on a 500 iterations run (I'm working on improving the plots) :
> > > > >
> > > > > http://bit.ly/1JL6Kp3
> > > > >
> > > > > In this specific case, after the first iteration Drill was
> allocating
> > > > ~2GB
> > > > > of direct memory, this number kept rising after each iteration to
> > ~6GB.
> > > > We
> > > > > suspect this caused one of our previous runs to crash the JVM.
> > > > >
> > > > > If we only focus on the log lines between iterations (when Drill's
> > > memory
> > > > > usage is below 10MB) then all allocated chunks are at most 2%
> usage.
> > At
> > > > > some point we end up with 288 nearly empty chunks, yet the next
> > > iteration
> > > > > will cause more chunks to be allocated!!!
> > > > >
> > > > > is this expected ?
> > > > >
> > > > > PS: I am running more tests and will update this thread with more
> > > > > informations.
> > > > >
> > > > > --
> > > > >
> > > > > Abdelhakim Deneche
> > > > >
> > > > > Software Engineer
> > > > >
> > > > > <http://www.mapr.com/>
> > > > >
> > > > >
> > > > > Now Available - Free Hadoop On-Demand Training
> > > > > <
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
I tried getting a jmap dump multiple times without success, each time it
crashes the jvm with the following exception:
Dumping heap to /home/mapr/private-sql-hadoop-test/framework/myfile.hprof
> ...
> Exception in thread "main" java.io.IOException: Premature EOF
> at
> sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:248)
> at
> sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:199)
> at
> sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:217)
> at
> sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
> at sun.tools.jmap.JMap.dump(JMap.java:242)
> at sun.tools.jmap.JMap.main(JMap.java:140)
On Mon, Jul 27, 2015 at 3:45 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> A allocate -> release cycle all on the same thread goes into a per thread
> cache.
>
> A bunch of Netty arena settings are configurable. The big issue I believe
> is that the limits are soft limits implemented by the allocation-time
> release mechanism. As such, if you allocate a bunch of memory, then
> release it all, that won't necessarily trigger any actual chunk releases.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> > wrote:
>
> > @Jacques, my understanding is that chunks are not owned by specific a
> > thread but they are part of a specific memory arena which is in turn only
> > accessed by specific threads. Do you want me to find which threads are
> > associated with the same arena where we have hanging chunks ?
> >
> >
> > On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > It sounds like your statement is that we're cacheing too many unused
> > > chunks. Hanifi and I previously discussed implementing a separate
> > flushing
> > > mechanism to release unallocated chunks that are hanging around. The
> > main
> > > question is, why are so many chunks hanging around and what threads are
> > > they associated with. A Jmap dump and analysis should allow you to do
> > > determine which thread owns the excess chunks. My guess would be the
> RPC
> > > pool since those are long lasting (as opposed to the WorkManager pool,
> > > which is contracting).
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > > adeneche@maprtech.com>
> > > wrote:
> > >
> > > > When running a set of, mostly window function, queries concurrently
> on
> > a
> > > > single drillbit with a 8GB max direct memory. We are seeing a
> > continuous
> > > > increase of direct memory allocation.
> > > >
> > > > We repeat the following steps multiple times:
> > > > - we launch in "iteration" of tests that will run all queries in a
> > random
> > > > order, 10 queries at a time
> > > > - after the iteration finishes, we wait for a couple of minute to
> give
> > > > Drill time to release the memory being held by the finishing
> fragments
> > > >
> > > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > > snapshots of how memory was internally used by Netty, we only focused
> > on
> > > > the number of allocated chunks, if we take this number and multiply
> it
> > by
> > > > 16MB (netty's chunk size) we get approximately the same value
> reported
> > by
> > > > Drill's direct memory allocation.
> > > > Here is a graph that shows the evolution of the number of allocated
> > > chunks
> > > > on a 500 iterations run (I'm working on improving the plots) :
> > > >
> > > > http://bit.ly/1JL6Kp3
> > > >
> > > > In this specific case, after the first iteration Drill was allocating
> > > ~2GB
> > > > of direct memory, this number kept rising after each iteration to
> ~6GB.
> > > We
> > > > suspect this caused one of our previous runs to crash the JVM.
> > > >
> > > > If we only focus on the log lines between iterations (when Drill's
> > memory
> > > > usage is below 10MB) then all allocated chunks are at most 2% usage.
> At
> > > > some point we end up with 288 nearly empty chunks, yet the next
> > iteration
> > > > will cause more chunks to be allocated!!!
> > > >
> > > > is this expected ?
> > > >
> > > > PS: I am running more tests and will update this thread with more
> > > > informations.
> > > >
> > > > --
> > > >
> > > > Abdelhakim Deneche
> > > >
> > > > Software Engineer
> > > >
> > > > <http://www.mapr.com/>
> > > >
> > > >
> > > > Now Available - Free Hadoop On-Demand Training
> > > > <
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Jacques Nadeau <ja...@dremio.com>.
A allocate -> release cycle all on the same thread goes into a per thread
cache.
A bunch of Netty arena settings are configurable. The big issue I believe
is that the limits are soft limits implemented by the allocation-time
release mechanism. As such, if you allocate a bunch of memory, then
release it all, that won't necessarily trigger any actual chunk releases.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <adeneche@maprtech.com
> wrote:
> @Jacques, my understanding is that chunks are not owned by specific a
> thread but they are part of a specific memory arena which is in turn only
> accessed by specific threads. Do you want me to find which threads are
> associated with the same arena where we have hanging chunks ?
>
>
> On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > It sounds like your statement is that we're cacheing too many unused
> > chunks. Hanifi and I previously discussed implementing a separate
> flushing
> > mechanism to release unallocated chunks that are hanging around. The
> main
> > question is, why are so many chunks hanging around and what threads are
> > they associated with. A Jmap dump and analysis should allow you to do
> > determine which thread owns the excess chunks. My guess would be the RPC
> > pool since those are long lasting (as opposed to the WorkManager pool,
> > which is contracting).
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > adeneche@maprtech.com>
> > wrote:
> >
> > > When running a set of, mostly window function, queries concurrently on
> a
> > > single drillbit with a 8GB max direct memory. We are seeing a
> continuous
> > > increase of direct memory allocation.
> > >
> > > We repeat the following steps multiple times:
> > > - we launch in "iteration" of tests that will run all queries in a
> random
> > > order, 10 queries at a time
> > > - after the iteration finishes, we wait for a couple of minute to give
> > > Drill time to release the memory being held by the finishing fragments
> > >
> > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > snapshots of how memory was internally used by Netty, we only focused
> on
> > > the number of allocated chunks, if we take this number and multiply it
> by
> > > 16MB (netty's chunk size) we get approximately the same value reported
> by
> > > Drill's direct memory allocation.
> > > Here is a graph that shows the evolution of the number of allocated
> > chunks
> > > on a 500 iterations run (I'm working on improving the plots) :
> > >
> > > http://bit.ly/1JL6Kp3
> > >
> > > In this specific case, after the first iteration Drill was allocating
> > ~2GB
> > > of direct memory, this number kept rising after each iteration to ~6GB.
> > We
> > > suspect this caused one of our previous runs to crash the JVM.
> > >
> > > If we only focus on the log lines between iterations (when Drill's
> memory
> > > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > > some point we end up with 288 nearly empty chunks, yet the next
> iteration
> > > will cause more chunks to be allocated!!!
> > >
> > > is this expected ?
> > >
> > > PS: I am running more tests and will update this thread with more
> > > informations.
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Chris Westin <ch...@gmail.com>.
What I was trying to get at was that we might be creating more chunks
because the per-thread caches for all the different threads haven't yet
been filled. Hence my questions about the degree of control we have over
the thread pool Netty is using for RPC threads. But if the asymptotic
observation isn't bearing out, then that probably isn't the cause either.
On Mon, Jul 27, 2015 at 12:47 PM, Abdel Hakim Deneche <adeneche@maprtech.com
> wrote:
> @Jacques, my understanding is that chunks are not owned by specific a
> thread but they are part of a specific memory arena which is in turn only
> accessed by specific threads. Do you want me to find which threads are
> associated with the same arena where we have hanging chunks ?
>
>
> On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > It sounds like your statement is that we're cacheing too many unused
> > chunks. Hanifi and I previously discussed implementing a separate
> flushing
> > mechanism to release unallocated chunks that are hanging around. The
> main
> > question is, why are so many chunks hanging around and what threads are
> > they associated with. A Jmap dump and analysis should allow you to do
> > determine which thread owns the excess chunks. My guess would be the RPC
> > pool since those are long lasting (as opposed to the WorkManager pool,
> > which is contracting).
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > adeneche@maprtech.com>
> > wrote:
> >
> > > When running a set of, mostly window function, queries concurrently on
> a
> > > single drillbit with a 8GB max direct memory. We are seeing a
> continuous
> > > increase of direct memory allocation.
> > >
> > > We repeat the following steps multiple times:
> > > - we launch in "iteration" of tests that will run all queries in a
> random
> > > order, 10 queries at a time
> > > - after the iteration finishes, we wait for a couple of minute to give
> > > Drill time to release the memory being held by the finishing fragments
> > >
> > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > snapshots of how memory was internally used by Netty, we only focused
> on
> > > the number of allocated chunks, if we take this number and multiply it
> by
> > > 16MB (netty's chunk size) we get approximately the same value reported
> by
> > > Drill's direct memory allocation.
> > > Here is a graph that shows the evolution of the number of allocated
> > chunks
> > > on a 500 iterations run (I'm working on improving the plots) :
> > >
> > > http://bit.ly/1JL6Kp3
> > >
> > > In this specific case, after the first iteration Drill was allocating
> > ~2GB
> > > of direct memory, this number kept rising after each iteration to ~6GB.
> > We
> > > suspect this caused one of our previous runs to crash the JVM.
> > >
> > > If we only focus on the log lines between iterations (when Drill's
> memory
> > > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > > some point we end up with 288 nearly empty chunks, yet the next
> iteration
> > > will cause more chunks to be allocated!!!
> > >
> > > is this expected ?
> > >
> > > PS: I am running more tests and will update this thread with more
> > > informations.
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
@Jacques, my understanding is that chunks are not owned by specific a
thread but they are part of a specific memory arena which is in turn only
accessed by specific threads. Do you want me to find which threads are
associated with the same arena where we have hanging chunks ?
On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> It sounds like your statement is that we're cacheing too many unused
> chunks. Hanifi and I previously discussed implementing a separate flushing
> mechanism to release unallocated chunks that are hanging around. The main
> question is, why are so many chunks hanging around and what threads are
> they associated with. A Jmap dump and analysis should allow you to do
> determine which thread owns the excess chunks. My guess would be the RPC
> pool since those are long lasting (as opposed to the WorkManager pool,
> which is contracting).
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > When running a set of, mostly window function, queries concurrently on a
> > single drillbit with a 8GB max direct memory. We are seeing a continuous
> > increase of direct memory allocation.
> >
> > We repeat the following steps multiple times:
> > - we launch in "iteration" of tests that will run all queries in a random
> > order, 10 queries at a time
> > - after the iteration finishes, we wait for a couple of minute to give
> > Drill time to release the memory being held by the finishing fragments
> >
> > Using Drill's memory logger ("drill.allocator") we were able to get
> > snapshots of how memory was internally used by Netty, we only focused on
> > the number of allocated chunks, if we take this number and multiply it by
> > 16MB (netty's chunk size) we get approximately the same value reported by
> > Drill's direct memory allocation.
> > Here is a graph that shows the evolution of the number of allocated
> chunks
> > on a 500 iterations run (I'm working on improving the plots) :
> >
> > http://bit.ly/1JL6Kp3
> >
> > In this specific case, after the first iteration Drill was allocating
> ~2GB
> > of direct memory, this number kept rising after each iteration to ~6GB.
> We
> > suspect this caused one of our previous runs to crash the JVM.
> >
> > If we only focus on the log lines between iterations (when Drill's memory
> > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > some point we end up with 288 nearly empty chunks, yet the next iteration
> > will cause more chunks to be allocated!!!
> >
> > is this expected ?
> >
> > PS: I am running more tests and will update this thread with more
> > informations.
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
On Mon, Jul 27, 2015 at 12:22 PM, Chris Westin <ch...@gmail.com>
wrote:
> The graph does look like it's reached some kind of asymptote -- is that
> true, does it stop increasing at that point?
>
I will need to run the tests one more time to be sure, but if I remember
correctly, further runs got actually higher.
> If it is, there's still the question of why it's so high.
>
Yes especially that the queries we are running use relatively small
datasets, I think 10k rows split among lot's of (small ?) parquet files.
> Parth: yes, we're handing these buffers off from the RPC thread that
> receives them to a worker thread that works on them. As Jacques mentions,
> or pools should be dying off, but its the RPC pool that we may need to look
> at more closely. What are its characteristics? If Netty hangs on to buffers
> to recycle them, there's a pool per thread, which might explain the
> apparent asymptotic approach I see in this graph: we've finally reached the
> limit that can be cached per-thread for all the threads. Can we control the
> size of that pool, possibly reducing it in size, or at least making it
> smaller but more elastic (so it shrinks back down when the threads aren't
> in use)? That might be an easy experiment to do to see how the graph is
> affected.
>
Netty caches released buffers in a per thread cache, and trims those caches
every 8192 allocations, so it makes sense to have cached buffer at the end
of each query in the RPC threads (all other threads are eventually
destroyed and their caches cleared). But those buffers and cached should
reused later on, so why do we need to create that much more new chunks
>
> On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > It sounds like your statement is that we're cacheing too many unused
> > chunks. Hanifi and I previously discussed implementing a separate
> flushing
> > mechanism to release unallocated chunks that are hanging around. The
> main
> > question is, why are so many chunks hanging around and what threads are
> > they associated with. A Jmap dump and analysis should allow you to do
> > determine which thread owns the excess chunks. My guess would be the RPC
> > pool since those are long lasting (as opposed to the WorkManager pool,
> > which is contracting).
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> > adeneche@maprtech.com>
> > wrote:
> >
> > > When running a set of, mostly window function, queries concurrently on
> a
> > > single drillbit with a 8GB max direct memory. We are seeing a
> continuous
> > > increase of direct memory allocation.
> > >
> > > We repeat the following steps multiple times:
> > > - we launch in "iteration" of tests that will run all queries in a
> random
> > > order, 10 queries at a time
> > > - after the iteration finishes, we wait for a couple of minute to give
> > > Drill time to release the memory being held by the finishing fragments
> > >
> > > Using Drill's memory logger ("drill.allocator") we were able to get
> > > snapshots of how memory was internally used by Netty, we only focused
> on
> > > the number of allocated chunks, if we take this number and multiply it
> by
> > > 16MB (netty's chunk size) we get approximately the same value reported
> by
> > > Drill's direct memory allocation.
> > > Here is a graph that shows the evolution of the number of allocated
> > chunks
> > > on a 500 iterations run (I'm working on improving the plots) :
> > >
> > > http://bit.ly/1JL6Kp3
> > >
> > > In this specific case, after the first iteration Drill was allocating
> > ~2GB
> > > of direct memory, this number kept rising after each iteration to ~6GB.
> > We
> > > suspect this caused one of our previous runs to crash the JVM.
> > >
> > > If we only focus on the log lines between iterations (when Drill's
> memory
> > > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > > some point we end up with 288 nearly empty chunks, yet the next
> iteration
> > > will cause more chunks to be allocated!!!
> > >
> > > is this expected ?
> > >
> > > PS: I am running more tests and will update this thread with more
> > > informations.
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > > <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Chris Westin <ch...@gmail.com>.
The graph does look like it's reached some kind of asymptote -- is that
true, does it stop increasing at that point?
If it is, there's still the question of why it's so high.
Parth: yes, we're handing these buffers off from the RPC thread that
receives them to a worker thread that works on them. As Jacques mentions,
or pools should be dying off, but its the RPC pool that we may need to look
at more closely. What are its characteristics? If Netty hangs on to buffers
to recycle them, there's a pool per thread, which might explain the
apparent asymptotic approach I see in this graph: we've finally reached the
limit that can be cached per-thread for all the threads. Can we control the
size of that pool, possibly reducing it in size, or at least making it
smaller but more elastic (so it shrinks back down when the threads aren't
in use)? That might be an easy experiment to do to see how the graph is
affected.
On Mon, Jul 27, 2015 at 11:04 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> It sounds like your statement is that we're cacheing too many unused
> chunks. Hanifi and I previously discussed implementing a separate flushing
> mechanism to release unallocated chunks that are hanging around. The main
> question is, why are so many chunks hanging around and what threads are
> they associated with. A Jmap dump and analysis should allow you to do
> determine which thread owns the excess chunks. My guess would be the RPC
> pool since those are long lasting (as opposed to the WorkManager pool,
> which is contracting).
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > When running a set of, mostly window function, queries concurrently on a
> > single drillbit with a 8GB max direct memory. We are seeing a continuous
> > increase of direct memory allocation.
> >
> > We repeat the following steps multiple times:
> > - we launch in "iteration" of tests that will run all queries in a random
> > order, 10 queries at a time
> > - after the iteration finishes, we wait for a couple of minute to give
> > Drill time to release the memory being held by the finishing fragments
> >
> > Using Drill's memory logger ("drill.allocator") we were able to get
> > snapshots of how memory was internally used by Netty, we only focused on
> > the number of allocated chunks, if we take this number and multiply it by
> > 16MB (netty's chunk size) we get approximately the same value reported by
> > Drill's direct memory allocation.
> > Here is a graph that shows the evolution of the number of allocated
> chunks
> > on a 500 iterations run (I'm working on improving the plots) :
> >
> > http://bit.ly/1JL6Kp3
> >
> > In this specific case, after the first iteration Drill was allocating
> ~2GB
> > of direct memory, this number kept rising after each iteration to ~6GB.
> We
> > suspect this caused one of our previous runs to crash the JVM.
> >
> > If we only focus on the log lines between iterations (when Drill's memory
> > usage is below 10MB) then all allocated chunks are at most 2% usage. At
> > some point we end up with 288 nearly empty chunks, yet the next iteration
> > will cause more chunks to be allocated!!!
> >
> > is this expected ?
> >
> > PS: I am running more tests and will update this thread with more
> > informations.
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
Re: Suspicious direct memory consumption when running queries concurrently
Posted by Jacques Nadeau <ja...@dremio.com>.
It sounds like your statement is that we're cacheing too many unused
chunks. Hanifi and I previously discussed implementing a separate flushing
mechanism to release unallocated chunks that are hanging around. The main
question is, why are so many chunks hanging around and what threads are
they associated with. A Jmap dump and analysis should allow you to do
determine which thread owns the excess chunks. My guess would be the RPC
pool since those are long lasting (as opposed to the WorkManager pool,
which is contracting).
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Jul 27, 2015 at 9:53 AM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> When running a set of, mostly window function, queries concurrently on a
> single drillbit with a 8GB max direct memory. We are seeing a continuous
> increase of direct memory allocation.
>
> We repeat the following steps multiple times:
> - we launch in "iteration" of tests that will run all queries in a random
> order, 10 queries at a time
> - after the iteration finishes, we wait for a couple of minute to give
> Drill time to release the memory being held by the finishing fragments
>
> Using Drill's memory logger ("drill.allocator") we were able to get
> snapshots of how memory was internally used by Netty, we only focused on
> the number of allocated chunks, if we take this number and multiply it by
> 16MB (netty's chunk size) we get approximately the same value reported by
> Drill's direct memory allocation.
> Here is a graph that shows the evolution of the number of allocated chunks
> on a 500 iterations run (I'm working on improving the plots) :
>
> http://bit.ly/1JL6Kp3
>
> In this specific case, after the first iteration Drill was allocating ~2GB
> of direct memory, this number kept rising after each iteration to ~6GB. We
> suspect this caused one of our previous runs to crash the JVM.
>
> If we only focus on the log lines between iterations (when Drill's memory
> usage is below 10MB) then all allocated chunks are at most 2% usage. At
> some point we end up with 288 nearly empty chunks, yet the next iteration
> will cause more chunks to be allocated!!!
>
> is this expected ?
>
> PS: I am running more tests and will update this thread with more
> informations.
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>