You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@poi.apache.org by Brian Trezise <br...@intellidata.net> on 2010/08/26 19:41:04 UTC
I need to read only (not write) large xls and xlsx files
No formulas, hidden cells, or anything fancy. Anywhere between 5 and 40
columns, and sometimes our datasets approach the 1,000,000 row limit of
.xlsx. I have been searching for a way to read and iterate over the rows of
these very large data files on and off for a month now with no luck. Has
anybody come up with a solution for this? Again, all I need to do is read
and iterate over the rows, I don't need write capability.
Thanks,
-----------------------------
Brian Trezise
IntelliData, Inc.
Staff Software Engineer
brian.trezise@intellidata.net
t: 720.524.4864
m: 303.809.9782
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: I need to read only (not write) large xls and xlsx files
Posted by Brian Trezise <br...@intellidata.net>.
Thanks nick. That's what I understood from all my googling but it's nice to
at least have it confirmed.
~Brian
On Fri, Aug 27, 2010 at 08:18, Nick Burch <ni...@alfresco.com> wrote:
> On Fri, 27 Aug 2010, Brian Trezise wrote:
>
>> Is there anything similar for writing of large .xls files as well?
>>
>
> Alas not. The biff8 file format has loads of back-and-forth references
> between different records. This makes at almost impossible to do a streaming
> write, because you need to know about things later in the file to write
> earlier bits...
>
>
> Nick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
>
Re: I need to read only (not write) large xls and xlsx files
Posted by Nick Burch <ni...@alfresco.com>.
On Fri, 27 Aug 2010, Brian Trezise wrote:
> Is there anything similar for writing of large .xls files as well?
Alas not. The biff8 file format has loads of back-and-forth references
between different records. This makes at almost impossible to do a
streaming write, because you need to know about things later in the file
to write earlier bits...
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: I need to read only (not write) large xls and xlsx files
Posted by Brian Trezise <br...@intellidata.net>.
Is there anything similar for writing of large .xls files as well?
On Thu, Aug 26, 2010 at 17:33, Nick Burch <ni...@alfresco.com> wrote:
> On Thu, 26 Aug 2010, Brian Trezise wrote:
>
>> Thanks Nick. Looks like that'll read the stuff (if a bit awkwardly... not
>> seeing an easy way to convert the example into an iterator for external
>> processing of one row at a time).
>>
>
> You'd probably have to buffer the row up, then block on the sax parsing
> until the row was taken, or something icky like that
>
>
> However, that aside, I'm now being asked if it is possible to write files
>> with the event model. My understanding from looking at this is that it
>> won't go both ways, but please correct me if I'm wrong? (Preferably with
>> some kind of example)
>>
>
> Yegor's BigGridDemo is an example of low memory writing of a big file, it's
> under xssf/usermodel/examples
>
>
> Nick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
>
RE: I need to read only (not write) large xls and xlsx files
Posted by Nick Burch <ni...@alfresco.com>.
On Thu, 26 Aug 2010, Brian Trezise wrote:
> Thanks Nick. Looks like that'll read the stuff (if a bit awkwardly...
> not seeing an easy way to convert the example into an iterator for
> external processing of one row at a time).
You'd probably have to buffer the row up, then block on the sax parsing
until the row was taken, or something icky like that
> However, that aside, I'm now being asked if it is possible to write
> files with the event model. My understanding from looking at this is
> that it won't go both ways, but please correct me if I'm wrong?
> (Preferably with some kind of example)
Yegor's BigGridDemo is an example of low memory writing of a big file,
it's under xssf/usermodel/examples
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
RE: I need to read only (not write) large xls and xlsx files
Posted by Brian Trezise <br...@intellidata.net>.
Thanks Nick. Looks like that'll read the stuff (if a bit awkwardly... not
seeing an easy way to convert the example into an iterator for external
processing of one row at a time). However, that aside, I'm now being asked
if it is possible to write files with the event model. My understanding
from looking at this is that it won't go both ways, but please correct me if
I'm wrong? (Preferably with some kind of example)
Thanks,
-----------------------------
Brian Trezise
IntelliData, Inc.
Staff Software Engineer
brian.trezise@intellidata.net
t: 720.524.4864
m: 303.809.9782
-----Original Message-----
From: Nick Burch [mailto:nick.burch@alfresco.com]
Sent: Thursday, August 26, 2010 14:00
To: POI Users List
Subject: Re: I need to read only (not write) large xls and xlsx files
On Thu, 26 Aug 2010, Brian Trezise wrote:
> No formulas, hidden cells, or anything fancy. Anywhere between 5 and 40
> columns, and sometimes our datasets approach the 1,000,000 row limit of
> .xlsx. I have been searching for a way to read and iterate over the
> rows of these very large data files on and off for a month now with no
> luck.
You want SAX parsing of the file, that's the quickest and lowest memory
way to do it. (Requires more work on your part though). See
http://poi.apache.org/spreadsheet/how-to.html#xssf_sax_api
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: I need to read only (not write) large xls and xlsx files
Posted by Nick Burch <ni...@alfresco.com>.
On Thu, 26 Aug 2010, Brian Trezise wrote:
> No formulas, hidden cells, or anything fancy. Anywhere between 5 and 40
> columns, and sometimes our datasets approach the 1,000,000 row limit of
> .xlsx. I have been searching for a way to read and iterate over the
> rows of these very large data files on and off for a month now with no
> luck.
You want SAX parsing of the file, that's the quickest and lowest memory
way to do it. (Requires more work on your part though). See
http://poi.apache.org/spreadsheet/how-to.html#xssf_sax_api
Nick
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org