Constant temp offloading to disk

[Alex Newman <al...@wandisco.com>]

>From one of our clients, I’m curious if y’all have any suggestions?


---
- Phoenix has this nifty little feature where it creates a temp file (in /var/tmp with the prefix ResultSpooler) to store query results if the query results are too big to fit in memory.
- The default threshold for the size of the query result is 20mb, meaning results under 20mb should use memory and results over 20mb should use the temp file.
- In theory, this temp file should get cleaned up, regardless of whether it was used to store the result or not
- Our problem is that we are getting tons of 53mb temp files that are never getting cleaned up, which is killing our client
- Phoenix seems to be having issues following the threshold for the result size, and having issues with cleaning up the temp files


As far as I've been able to tell while Googling this problem, the Apache folks fixed this in version 3.0 and 4.0 of Phoenix (we're using 4.0), but we're using the HWX Phoenix jar, and they aren't being particularly helpful with potential solutions.


We have tried:
- Modifying the Phoenix jar on the client so that it only creates temp files when absolutely necessary, and modifying the threshold (unsuccessful)
- Using the most recent Apache Phoenix jar on both client and server (serious compatibility issues with the HWX HDP stuff, so we rolled back)


-Alex

Sent from Alex Newman on a small glass box.
404.507.6749
-- 


Listed on the London Stock Exchange: WAND 
<http://www.bloomberg.com/quote/WAND:LN>

THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE 
PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its 
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. 
 If you are not the intended recipient, please notify us immediately and 
destroy the message without disclosing its contents to anyone.  Any 
distribution, use or copying of this e-mail or the information it contains 
by other than an intended recipient is unauthorized.  The views and 
opinions expressed in this e-mail message are the author's own and may not 
reflect the views and opinions of WANdisco, unless the author is authorized 
by WANdisco to express such views or opinions on its behalf.  All email 
sent to or from this address is subject to electronic storage and review by 
WANdisco.  Although WANdisco operates anti-virus programs, it does not 
accept responsibility for any damage whatsoever caused by viruses being 
passed.

Re: Constant temp offloading to disk

[Jeffrey Zhong <jz...@hortonworks.com>]

You can check if resultset & connection are closed after use in your
application code. That¹s one possible area of leaking spill files. There are
two outstanding JIRAs relating to this as PHOENIX-1395 & PHOENIX-1396. You
can post possible repro steps in PHOENIX-1395 so that it can be addressed
there.

Thanks,
-Jeffrey   

From:  Alex Newman <al...@wandisco.com>
Reply-To:  <us...@phoenix.apache.org>
Date:  Tuesday, November 4, 2014 at 8:47 AM
To:  <us...@phoenix.apache.org>
Subject:  Constant temp offloading to disk

>From one of our clients, I¹m curious if y¹all have any suggestions?

---
- Phoenix has this nifty little feature where it creates a temp file (in
/var/tmp with the prefix ResultSpooler) to store query results if the query
results are too big to fit in memory.
- The default threshold for the size of the query result is 20mb, meaning
results under 20mb should use memory and results over 20mb should use the
temp file.
- In theory, this temp file should get cleaned up, regardless of whether it
was used to store the result or not
- Our problem is that we are getting tons of 53mb temp files that are never
getting cleaned up, which is killing our client
- Phoenix seems to be having issues following the threshold for the result
size, and having issues with cleaning up the temp files

As far as I've been able to tell while Googling this problem, the Apache
folks fixed this in version 3.0 and 4.0 of Phoenix (we're using 4.0), but
we're using the HWX Phoenix jar, and they aren't being particularly helpful
with potential solutions.

We have tried:
- Modifying the Phoenix jar on the client so that it only creates temp files
when absolutely necessary, and modifying the threshold (unsuccessful)
- Using the most recent Apache Phoenix jar on both client and server
(serious compatibility issues with the HWX HDP stuff, so we rolled back)

-Alex

Sent from Alex Newman on a small glass box.
404.507.6749

Listed on the London Stock Exchange: WAND
<http://www.bloomberg.com/quote/WAND:LN>

THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
If you are not the intended recipient, please notify us immediately and
destroy the message without disclosing its contents to anyone.  Any
distribution, use or copying of this e-mail or the information it contains
by other than an intended recipient is unauthorized.  The views and opinions
expressed in this e-mail message are the author's own and may not reflect
the views and opinions of WANdisco, unless the author is authorized by
WANdisco to express such views or opinions on its behalf.  All email sent to
or from this address is subject to electronic storage and review by
WANdisco.  Although WANdisco operates anti-virus programs, it does not
accept responsibility for any damage whatsoever caused by viruses being
passed.



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.