You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by Faisal Maqsood <fm...@i2cinc.com> on 2008/05/06 10:05:42 UTC
Does POI Support Streaming?
Hello Team,
We found this API useful but it is putting us in out of memory problem using
for excel files. Below are some statistics of memory consumed. Can any of
you guide us how to use this API in streaming i.e. a way that instead of
loading complete file, it load first n records and on accessing (n+1)th
record, it unload n loaded records from the memory and load n+1 to 2n
records and so on.
File Type
Records
API Used for Excel Manipulation
Operation
File Size (MB)
Memory Consumed (MB)
Excel
25,000
Apache POI
Load Only
2.35 MB
Approx 60 MB
I will appreciate if I some sample code or useful links in the reply.
Regards,
Faisal Maqsood
RE: Does POI Support Streaming?
Posted by Nick Burch <ni...@torchbox.com>.
On Tue, 6 May 2008, Faisal Maqsood wrote:
> There is no my own code inside. It is just the example code sent in the
> link.
>From your own trace:
79.6% - fm.poi.EventExample.processRecord
that tells us that ~80% of the time is in _your own_ code. No idea where
the rest is, as your profiler doesn't appear to be able to add up...
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Does POI Support Streaming?
Posted by Faisal Maqsood <fm...@i2cinc.com>.
There is no my own code inside. It is just the example code sent in the
link.
But yes by removing the "System.out.println..." lines from the code, it is
much faster (around 8-9 times fast).
Regards,
Faisal Maqsood
-----Original Message-----
From: Nick Burch [mailto:nick@torchbox.com]
Sent: Tuesday, May 06, 2008 6:30 PM
To: Faisal Maqsood
Cc: 'POI Developers List'
Subject: RE: Does POI Support Streaming?
On Tue, 6 May 2008, Faisal Maqsood wrote:
> Thanks for your time and your comments can help to make our views better
> about POI API but it seems profiler is showing different statistics. A
> zip file is attached containing profiler snap shot and two images. If
> you have not installed JProfiler4, images for CPU and VM view to show
> where the time is consumed.
Looking at that, at least 80% of the time is spent in your code, and
around 60% of your time is spent printing stuff out
Seems to me that the poi component of your cpu time is pretty negligable.
You'll need to optimise your own code
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Does POI Support Streaming?
Posted by Nick Burch <ni...@torchbox.com>.
On Tue, 6 May 2008, Faisal Maqsood wrote:
> Thanks for your time and your comments can help to make our views better
> about POI API but it seems profiler is showing different statistics. A
> zip file is attached containing profiler snap shot and two images. If
> you have not installed JProfiler4, images for CPU and VM view to show
> where the time is consumed.
Looking at that, at least 80% of the time is spent in your code, and
around 60% of your time is spent printing stuff out
Seems to me that the poi component of your cpu time is pretty negligable.
You'll need to optimise your own code
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Does POI Support Streaming?
Posted by Faisal Maqsood <fm...@i2cinc.com>.
Thanks for your time and your comments can help to make our views better
about POI API but it seems profiler is showing different statistics. A zip
file is attached containing profiler snap shot and two images. If you have
not installed JProfiler4, images for CPU and VM view to show where the time
is consumed.
I used 38MB file but to save time I stopped the profiler explicitly after 1+
minute.
Regards,
Faisal Maqsood
-----Original Message-----
From: Nick Burch [mailto:nick@torchbox.com]
Sent: Tuesday, May 06, 2008 4:44 PM
To: Faisal Maqsood
Cc: 'POI Developers List'
Subject: RE: Does POI Support Streaming?
On Tue, 6 May 2008, Faisal Maqsood wrote:
> I have following observations with this sample code:
>
> *** Whether parsing 38MB file or 6MB file, the example code consume around
> 50MB of heap memory
Sounds about right
> *** It took around 6.5minutes for 6MB while for 38MB it took around 20
> minutes in parsing and displaying the contents
Run a profiler and see where your time is being spent. I have a hunch it
won't be in poi, as I've never noticed the event usermodel code take more
than about a second
Nick
RE: Does POI Support Streaming?
Posted by Nick Burch <ni...@torchbox.com>.
On Tue, 6 May 2008, Faisal Maqsood wrote:
> I have following observations with this sample code:
>
> *** Whether parsing 38MB file or 6MB file, the example code consume around
> 50MB of heap memory
Sounds about right
> *** It took around 6.5minutes for 6MB while for 38MB it took around 20
> minutes in parsing and displaying the contents
Run a profiler and see where your time is being spent. I have a hunch it
won't be in poi, as I've never noticed the event usermodel code take more
than about a second
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
RE: Does POI Support Streaming?
Posted by Faisal Maqsood <fm...@i2cinc.com>.
Thank you for instant reply and to the point example code.
I have following observations with this sample code:
*** Whether parsing 38MB file or 6MB file, the example code consume around
50MB of heap memory
*** It took around 6.5minutes for 6MB while for 38MB it took around 20
minutes in parsing and displaying the contents
Your comments for how to improve these statistics will be highly
appreciated.
Optional: If there is any way to compute how much time will it need to
complete the file parsing?
Regards,
Faisal Maqsood
-----Original Message-----
From: Nick Burch [mailto:nick@torchbox.com]
Sent: Tuesday, May 06, 2008 3:02 PM
To: POI Developers List
Cc: 'Faisal Maqsood'
Subject: Re: Does POI Support Streaming?
On Tue, 6 May 2008, Faisal Maqsood wrote:
> Can any of you guide us how to use this API in streaming i.e. a way that
> instead of loading complete file
http://poi.apache.org/hssf/how-to.html#event_api
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
Re: Does POI Support Streaming?
Posted by Nick Burch <ni...@torchbox.com>.
On Tue, 6 May 2008, Faisal Maqsood wrote:
> Can any of you guide us how to use this API in streaming i.e. a way that
> instead of loading complete file
http://poi.apache.org/hssf/how-to.html#event_api
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org