Question about memory chunk and waiting for memory to become free

[lars hofhansl <la...@apache.org>]

Hi,
just looking at MemoryChunk for various reasons.
What I did not realize is that when no memory is currently available the implementation waits for memory to become available.The default wait time is 10s.

This seems very dangerous to me. This can hang HBase handler threads at various points, especially when Phoenix is "busy" (I realize we're not talking about real memory here, but the effect is still that the HBase server threads would stall).
HBase was designed with the assumption that individual RPCs do not tie up a handler thread for long. (The worst offender is a scan RPC that could scan up to a region's worth if a highly selective filter was passed). It's also possible that this is mostly used in asynchronous processing, although a quick scan of the code suggests that's not the case.

It's possible that I'm missing something (in which case someone enlighten me please). I feel with wait time should be set to 0. I.e. we check whether there's memory available right now, if not we just fail the request and free the handler thread. I'd even argue we can remove the wait/synchronization code completely from MemoryChunk.

Comments?
-- Lars

Re: Question about memory chunk and waiting for memory to become free

[lars hofhansl <la...@apache.org>]

Thanks James.
I'll file a jira.

      From: James Taylor <ja...@apache.org>
 To: dev@phoenix.apache.org; lars hofhansl <la...@apache.org> 
 Sent: Monday, September 4, 2017 9:11 AM
 Subject: Re: Question about memory chunk and waiting for memory to become free
   
Sure, Lars. As long as we throw an exception that'll cause a retry, it
seems fine to just fail it. We'll need some kind of synchronization around
the read/write of available memory member variable (or we can just use an
atomic long). Please update the docs here and indicate that
phoenix.query.maxGlobalMemoryWaitMs is deprecated as of 4.12:
http://phoenix.apache.org/tuning.html

On Sun, Sep 3, 2017 at 11:28 AM lars hofhansl <la...@apache.org> wrote:

> Hi,
> just looking at MemoryChunk for various reasons.
> What I did not realize is that when no memory is currently available the
> implementation waits for memory to become available.The default wait time
> is 10s.
>
> This seems very dangerous to me. This can hang HBase handler threads at
> various points, especially when Phoenix is "busy" (I realize we're not
> talking about real memory here, but the effect is still that the HBase
> server threads would stall).
> HBase was designed with the assumption that individual RPCs do not tie up
> a handler thread for long. (The worst offender is a scan RPC that could
> scan up to a region's worth if a highly selective filter was passed). It's
> also possible that this is mostly used in asynchronous processing, although
> a quick scan of the code suggests that's not the case.
>
> It's possible that I'm missing something (in which case someone enlighten
> me please). I feel with wait time should be set to 0. I.e. we check whether
> there's memory available right now, if not we just fail the request and
> free the handler thread. I'd even argue we can remove the
> wait/synchronization code completely from MemoryChunk.
>
> Comments?
> -- Lars
>


   

Re: Question about memory chunk and waiting for memory to become free

[James Taylor <ja...@apache.org>]

Sure, Lars. As long as we throw an exception that'll cause a retry, it
seems fine to just fail it. We'll need some kind of synchronization around
the read/write of available memory member variable (or we can just use an
atomic long). Please update the docs here and indicate that
phoenix.query.maxGlobalMemoryWaitMs is deprecated as of 4.12:
http://phoenix.apache.org/tuning.html

On Sun, Sep 3, 2017 at 11:28 AM lars hofhansl <la...@apache.org> wrote:

> Hi,
> just looking at MemoryChunk for various reasons.
> What I did not realize is that when no memory is currently available the
> implementation waits for memory to become available.The default wait time
> is 10s.
>
> This seems very dangerous to me. This can hang HBase handler threads at
> various points, especially when Phoenix is "busy" (I realize we're not
> talking about real memory here, but the effect is still that the HBase
> server threads would stall).
> HBase was designed with the assumption that individual RPCs do not tie up
> a handler thread for long. (The worst offender is a scan RPC that could
> scan up to a region's worth if a highly selective filter was passed). It's
> also possible that this is mostly used in asynchronous processing, although
> a quick scan of the code suggests that's not the case.
>
> It's possible that I'm missing something (in which case someone enlighten
> me please). I feel with wait time should be set to 0. I.e. we check whether
> there's memory available right now, if not we just fail the request and
> free the handler thread. I'd even argue we can remove the
> wait/synchronization code completely from MemoryChunk.
>
> Comments?
> -- Lars
>