You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by "Banerjee, Tapas" <Ta...@fmr.com> on 2014/09/25 18:13:40 UTC
Help! - SXSSF APi writing files under /tmp - not cleaning them.
Hello,
We are using Apache SXSSF POI SXSSF APi for efficient operation of writing into an Excel file.
We plan to write up to 200000 rows.
The problem we are facing that API writing into /tmp javasharedresources (for Websphere App server under Linux) dir - not cleaning it, making /tmp file system full after couple of large export to excel.
Any Idea how we can prevent the filling the file system full?
Note: we are doing workbook.close(); - to ensure file system are cleaned.
Thanks in advance,
-Tapas
Tapas.banerjee@fmr.com<ma...@fmr.com>
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Banerjee, Tapas" <Ta...@fmr.com>.
Nick,
We tried with poi 3.11 beta 2, still no luck.
We have set workbook window size to -1 // -1 turn off auto-flushing and accumulate all rows in memory
Called workbook.dispose();
Also called workbook.setCompressTempFiles(true);
When we ran a query of 146 K excel rows, it created huge xml file at /tmp to make FS full.
Any other ideas?
Thanks,
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:57 PM
To: POI Developers List
Subject: RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> I believe so, we are using poi-3.10-FINAL-20120208.jar
Try 3.11 beta 2 - there were sxssf fixes in betas 1 and 2
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
Posted by Nick Burch <ap...@gagravarr.org>.
On Fri, 26 Sep 2014, Banerjee, Tapas wrote:
> But still we see huge file (few GB) created at /tmp dir - when we tried to write ~ 142 K Excel rows.
> Causing the /tmp FS to full.
>
> Any idea, how we can fix this problem?
Make sure you're on 3.11 beta 2. With that, once you dispose of the temp
files, they should be properly deleted
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Banerjee, Tapas" <Ta...@fmr.com>.
To alleviate disk fill tissue:
We have set workbook window size to -1 // -1 turn off auto-flushing and accumulate all rows in memory
Called workbook.dispose();
Also called workbook.setCompressTempFiles(true);
But still we see huge file (few GB) created at /tmp dir - when we tried to write ~ 142 K Excel rows.
Causing the /tmp FS to full.
Any idea, how we can fix this problem?
Thanks
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:57 PM
To: POI Developers List
Subject: RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> I believe so, we are using poi-3.10-FINAL-20120208.jar
Try 3.11 beta 2 - there were sxssf fixes in betas 1 and 2
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
Posted by Nick Burch <ap...@gagravarr.org>.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> I believe so, we are using poi-3.10-FINAL-20120208.jar
Try 3.11 beta 2 - there were sxssf fixes in betas 1 and 2
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Banerjee, Tapas" <Ta...@fmr.com>.
I believe so, we are using poi-3.10-FINAL-20120208.jar
Below is the example file that gets created.
/tmp/javasharedresources
-rw-r--r-- 1 orwsaaa1 orwsaaa1 52428800 Sep 25 12:14 C240D2A64P_webspherev70_orwsaaa1_G08
While researching, I see some details this space in this article
http://www-01.ibm.com/support/docview.wss?uid=swg21298333
Thanks
-Tapas
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:32 PM
To: POI Developers List
Subject: Re: Help! - SXSSF APi writing files under /tmp - not cleaning them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> The problem we are facing that API writing into /tmp javasharedresources
> (for Websphere App server under Linux) dir - not cleaning it, making
> /tmp file system full after couple of large export to excel.
Did you make sure you're using the latest version of Apache POI?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Banerjee, Tapas" <Ta...@fmr.com>.
We already called dispose.
We kind of bypass the disk space issue by increasing window size as well as calling workbook.setCompressTempFiles(true);
-----Original Message-----
From: Kish, Robert [mailto:Robert.Kish@ncogroup.com]
Sent: Thursday, September 25, 2014 2:10 PM
To: POI Developers List
Subject: RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
You have to call the dispose method.
https://poi.apache.org/apidocs/org/apache/poi/xssf/streaming/SXSSFWorkbo
ok.html#dispose%28%29
-----Original Message-----
From: Banerjee, Tapas [mailto:Tapas.Banerjee@fmr.com]
Sent: Thursday, September 25, 2014 01:00 PM
To: POI Developers List
Subject: RE: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
Just to add on this. Below is a sample XML files are written into /tmp
-rw-r--r-- 1 xxxx xxxx 3046 Sep 25 12:40
poi-sxssf-sheet8160933773697041289.xml
For large export - the bigger xml files are created, and they don't
clean up automatically, causing the disk to be full.
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:32 PM
To: POI Developers List
Subject: Re: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> The problem we are facing that API writing into /tmp
> javasharedresources (for Websphere App server under Linux) dir - not
> cleaning it, making /tmp file system full after couple of large export
to excel.
Did you make sure you're using the latest version of Apache POI?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org For additional
commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org For additional
commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Kish, Robert" <Ro...@ncogroup.com>.
You have to call the dispose method.
https://poi.apache.org/apidocs/org/apache/poi/xssf/streaming/SXSSFWorkbo
ok.html#dispose%28%29
-----Original Message-----
From: Banerjee, Tapas [mailto:Tapas.Banerjee@fmr.com]
Sent: Thursday, September 25, 2014 01:00 PM
To: POI Developers List
Subject: RE: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
Just to add on this. Below is a sample XML files are written into /tmp
-rw-r--r-- 1 xxxx xxxx 3046 Sep 25 12:40
poi-sxssf-sheet8160933773697041289.xml
For large export - the bigger xml files are created, and they don't
clean up automatically, causing the disk to be full.
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:32 PM
To: POI Developers List
Subject: Re: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> The problem we are facing that API writing into /tmp
> javasharedresources (for Websphere App server under Linux) dir - not
> cleaning it, making /tmp file system full after couple of large export
to excel.
Did you make sure you're using the latest version of Apache POI?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org For additional
commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org For additional
commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Help! - SXSSF APi writing files under /tmp - not cleaning them.
Posted by "Banerjee, Tapas" <Ta...@fmr.com>.
Just to add on this. Below is a sample XML files are written into /tmp
-rw-r--r-- 1 xxxx xxxx 3046 Sep 25 12:40 poi-sxssf-sheet8160933773697041289.xml
For large export - the bigger xml files are created, and they don't clean up automatically, causing the disk to be full.
-----Original Message-----
From: Nick Burch [mailto:apache@gagravarr.org]
Sent: Thursday, September 25, 2014 12:32 PM
To: POI Developers List
Subject: Re: Help! - SXSSF APi writing files under /tmp - not cleaning them.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> The problem we are facing that API writing into /tmp javasharedresources
> (for Websphere App server under Linux) dir - not cleaning it, making
> /tmp file system full after couple of large export to excel.
Did you make sure you're using the latest version of Apache POI?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
Re: Help! - SXSSF APi writing files under /tmp - not cleaning
them.
Posted by Nick Burch <ap...@gagravarr.org>.
On Thu, 25 Sep 2014, Banerjee, Tapas wrote:
> The problem we are facing that API writing into /tmp javasharedresources
> (for Websphere App server under Linux) dir - not cleaning it, making
> /tmp file system full after couple of large export to excel.
Did you make sure you're using the latest version of Apache POI?
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org