You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@poi.apache.org by "dokia@126.com" <do...@126.com> on 2016/12/02 08:10:13 UTC
Too much memory is used when reading a xlsx-file whose size is just 7.3M
Hi!
I am trying to read a xlsx-file whose size is 7.3M with POI version 3.15, here is the code：
InputStream is = new FileInputStream(new File(path));//line 1
Workbook book = new XSSFWorkbook(in);// line 2
When running to line 2，the memory of javaw rised violently from 700M to 1200M，cpu from 10% to 90%，the following is the screenshot of memory dump:
In the picture, the object of Xobj$AttrXobj has more than 5 millions so that it occupy the most of memory，but the file size is just 7.3M.
Can someone help me?
Thank you!
dokia@126.com
RE: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Ha! I just ran into this with .docx as well [1][2]. Given that we only need to extract contents on Tika, the experimental SAX parser [3] is way more efficient, esp on very large docs. The gc with our current DOM was killing performance, esp multithreaded.
There are portions of the docx SAX parser that I think will fit well within POI as a parallel to xssf's eventusermodel. I hope to submit a patch for review sometime next week (?) (or in open-source time, January?)...
Cheers,
Tim
[1] https://issues.apache.org/jira/browse/TIKA-1321
[2] https://issues.apache.org/jira/browse/TIKA-2180
[3] Admittedly, the experimental SAX parser doesn't include all of the features that our current DOM parser does! More work remains...
-----Original Message-----
From: Javen O'Neal [mailto:javenoneal@gmail.com]
Sent: Friday, December 2, 2016 2:21 PM
To: POI Users List <us...@poi.apache.org>
Subject: Re: Too much memory is used when reading a xlsx-file whose size is just 7.3M
Those numbers sound about right. I'm used to 4 MB balloning to 1 GB.
We could significantly reduce memory consumption if we didn't maintain the XML DOM in memory, but replacing that requires thousands of hours of work.
Re: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by Blake Watson <bl...@pnmac.com>.
Yeah, I'm not complaining. =)
What I do wouldn't be possible without POI. We can splurge on RAM...
RE: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by "Murphy, Mark" <mu...@metalexmfg.com>.
If we replaced that, and stored the document as Java objects only, we could then conceivably write the document as either XLS or XLSX, or whatever format comes down the pike simply by selecting a write format. Someone would have to create the writers. Expanding on that, we could also read whatever format we wanted by creating an appropriate reader. However that would lose the ability for a user to just use the CT classes to support a bit of the standard that hasn't yet been surfaced by POI. But, it would make swapping out XMLBeans for something else easier.
Still thousands of hours of work.
-----Original Message-----
From: Javen O'Neal [mailto:javenoneal@gmail.com]
Sent: Friday, December 02, 2016 2:21 PM
To: POI Users List <us...@poi.apache.org>
Subject: Re: Too much memory is used when reading a xlsx-file whose size is just 7.3M
Those numbers sound about right. I'm used to 4 MB balloning to 1 GB.
We could significantly reduce memory consumption if we didn't maintain the XML DOM in memory, but replacing that requires thousands of hours of work.
On Dec 2, 2016 09:44, "Blake Watson" <bl...@pnmac.com> wrote:
> Well, this is interesting: I've also seen a ballooning of RAM usage
> (though I wouldn't call it "violent" =P) when loading a spreadsheet
> and I didn't know about the input-stream vs. file distinction.
> Although, come to think of it, I'm pulling out of a database so that might not help me.
>
> The excess usage is only temporary, though, right? My small (<100K)
> spreadsheets seem to take about 50MB, and my large ones (5MB) seem to
> take about 500MB.
>
> On Fri, Dec 2, 2016 at 7:54 AM, Gordon Pettey <pe...@gmail.com>
> wrote:
>
> > On Fri, Dec 2, 2016 at 2:41 AM, Javen O'Neal <ja...@gmail.com>
> wrote:
> >
> > > CPU pegged due the the Java garbage collector trying to clean up
> > > unused memory. The out of memory error is raised when the garbage
> > > collector
> > cannot
> > > free the necessary memory for new objects to be allocated.
> > >
> > > Have you read the FAQ yet? https://urldefense.proofpoint.
> > com/v2/url?u=https-3A__poi.apache.org_faq.html-23faq-
> 2DN10165&d=DgIFaQ&c=
> > dmLomitc30UP5j2qU8E1rg&r=p42pHJHEwFZOHtVFHKJUdL2fYbroN3
> > 3stXXb3Psthjw&m=K-mZqLKcau3v-2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=_
> > gk9WyMdApng32mJ3IDsTgjct7dpw_BenPZP-wdE0fA&e=
> > >
> > > On Dec 2, 2016 00:11, "dokia@126.com" <do...@126.com> wrote:
> > >
> > > > Hi!
> > > >
> > > > I am trying to read a xlsx-file whose size is 7.3M with POI
> > > > version
> > 3.15,
> > > > here is the code：
> > > >
> > > >
> > > > *InputStream is = new FileInputStream(new File(path));//line
> 1Workbook
> > > book = new XSSFWorkbook(in);// line 2*
> > > > When running to line 2，the memory of javaw rised violently from
> > > > 700M to 1200M，cpu from 10% to 90%，the following is the
> > > > screenshot of memory dump:
> > > >
> > >
> >
> > See https://urldefense.proofpoint.com/v2/url?u=http-3A__poi.
> > apache.org_spreadsheet_quick-2Dguide.html-23Files-2Bvs-
> > 2BInputStreams&d=DgIFaQ&c=dmLomitc30UP5j2qU8E1rg&r=
> > p42pHJHEwFZOHtVFHKJUdL2fYbroN33stXXb3Psthjw&m=K-mZqLKcau3v-
> > 2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=Q2dr3Xc2iSpx8HHAAj0kQkGXZYpdBK
> > 9yZFifR-mQo64&e= ,
> > also.
> >
>
>
>
> --
>
> *Blake Watson*
>
> *PNMAC*
> Application Development Manager
> 5898 Condor Drive
> Moorpark, CA 93021
> (805) 330.4911 x7742
> blake.watson@pnmac.com <me...@pnmac.com>
> www.PennyMacUSA.com <http://www.pennymacusa.com/>
>
Re: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by Javen O'Neal <ja...@gmail.com>.
Those numbers sound about right. I'm used to 4 MB balloning to 1 GB.
We could significantly reduce memory consumption if we didn't maintain the
XML DOM in memory, but replacing that requires thousands of hours of work.
On Dec 2, 2016 09:44, "Blake Watson" <bl...@pnmac.com> wrote:
> Well, this is interesting: I've also seen a ballooning of RAM usage (though
> I wouldn't call it "violent" =P) when loading a spreadsheet and I didn't
> know about the input-stream vs. file distinction. Although, come to think
> of it, I'm pulling out of a database so that might not help me.
>
> The excess usage is only temporary, though, right? My small (<100K)
> spreadsheets seem to take about 50MB, and my large ones (5MB) seem to take
> about 500MB.
>
> On Fri, Dec 2, 2016 at 7:54 AM, Gordon Pettey <pe...@gmail.com>
> wrote:
>
> > On Fri, Dec 2, 2016 at 2:41 AM, Javen O'Neal <ja...@gmail.com>
> wrote:
> >
> > > CPU pegged due the the Java garbage collector trying to clean up unused
> > > memory. The out of memory error is raised when the garbage collector
> > cannot
> > > free the necessary memory for new objects to be allocated.
> > >
> > > Have you read the FAQ yet? https://urldefense.proofpoint.
> > com/v2/url?u=https-3A__poi.apache.org_faq.html-23faq-
> 2DN10165&d=DgIFaQ&c=
> > dmLomitc30UP5j2qU8E1rg&r=p42pHJHEwFZOHtVFHKJUdL2fYbroN3
> > 3stXXb3Psthjw&m=K-mZqLKcau3v-2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=_
> > gk9WyMdApng32mJ3IDsTgjct7dpw_BenPZP-wdE0fA&e=
> > >
> > > On Dec 2, 2016 00:11, "dokia@126.com" <do...@126.com> wrote:
> > >
> > > > Hi!
> > > >
> > > > I am trying to read a xlsx-file whose size is 7.3M with POI version
> > 3.15,
> > > > here is the code：
> > > >
> > > >
> > > > *InputStream is = new FileInputStream(new File(path));//line
> 1Workbook
> > > book = new XSSFWorkbook(in);// line 2*
> > > > When running to line 2，the memory of javaw rised
> > > > violently from 700M to 1200M，cpu from 10% to 90%，the following
> > > > is the screenshot of memory dump:
> > > >
> > >
> >
> > See https://urldefense.proofpoint.com/v2/url?u=http-3A__poi.
> > apache.org_spreadsheet_quick-2Dguide.html-23Files-2Bvs-
> > 2BInputStreams&d=DgIFaQ&c=dmLomitc30UP5j2qU8E1rg&r=
> > p42pHJHEwFZOHtVFHKJUdL2fYbroN33stXXb3Psthjw&m=K-mZqLKcau3v-
> > 2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=Q2dr3Xc2iSpx8HHAAj0kQkGXZYpdBK
> > 9yZFifR-mQo64&e= ,
> > also.
> >
>
>
>
> --
>
> *Blake Watson*
>
> *PNMAC*
> Application Development Manager
> 5898 Condor Drive
> Moorpark, CA 93021
> (805) 330.4911 x7742
> blake.watson@pnmac.com <me...@pnmac.com>
> www.PennyMacUSA.com <http://www.pennymacusa.com/>
>
Re: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by Blake Watson <bl...@pnmac.com>.
Well, this is interesting: I've also seen a ballooning of RAM usage (though
I wouldn't call it "violent" =P) when loading a spreadsheet and I didn't
know about the input-stream vs. file distinction. Although, come to think
of it, I'm pulling out of a database so that might not help me.
The excess usage is only temporary, though, right? My small (<100K)
spreadsheets seem to take about 50MB, and my large ones (5MB) seem to take
about 500MB.
On Fri, Dec 2, 2016 at 7:54 AM, Gordon Pettey <pe...@gmail.com> wrote:
> On Fri, Dec 2, 2016 at 2:41 AM, Javen O'Neal <ja...@gmail.com> wrote:
>
> > CPU pegged due the the Java garbage collector trying to clean up unused
> > memory. The out of memory error is raised when the garbage collector
> cannot
> > free the necessary memory for new objects to be allocated.
> >
> > Have you read the FAQ yet? https://urldefense.proofpoint.
> com/v2/url?u=https-3A__poi.apache.org_faq.html-23faq-2DN10165&d=DgIFaQ&c=
> dmLomitc30UP5j2qU8E1rg&r=p42pHJHEwFZOHtVFHKJUdL2fYbroN3
> 3stXXb3Psthjw&m=K-mZqLKcau3v-2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=_
> gk9WyMdApng32mJ3IDsTgjct7dpw_BenPZP-wdE0fA&e=
> >
> > On Dec 2, 2016 00:11, "dokia@126.com" <do...@126.com> wrote:
> >
> > > Hi!
> > >
> > > I am trying to read a xlsx-file whose size is 7.3M with POI version
> 3.15,
> > > here is the code：
> > >
> > >
> > > *InputStream is = new FileInputStream(new File(path));//line 1Workbook
> > book = new XSSFWorkbook(in);// line 2*
> > > When running to line 2，the memory of javaw rised
> > > violently from 700M to 1200M，cpu from 10% to 90%，the following
> > > is the screenshot of memory dump:
> > >
> >
>
> See https://urldefense.proofpoint.com/v2/url?u=http-3A__poi.
> apache.org_spreadsheet_quick-2Dguide.html-23Files-2Bvs-
> 2BInputStreams&d=DgIFaQ&c=dmLomitc30UP5j2qU8E1rg&r=
> p42pHJHEwFZOHtVFHKJUdL2fYbroN33stXXb3Psthjw&m=K-mZqLKcau3v-
> 2mO1e7Es8pmREXI9_WkDMs3UxiVEsE&s=Q2dr3Xc2iSpx8HHAAj0kQkGXZYpdBK
> 9yZFifR-mQo64&e= ,
> also.
>
--
*Blake Watson*
*PNMAC*
Application Development Manager
5898 Condor Drive
Moorpark, CA 93021
(805) 330.4911 x7742
blake.watson@pnmac.com <me...@pnmac.com>
www.PennyMacUSA.com <http://www.pennymacusa.com/>
Re: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by Gordon Pettey <pe...@gmail.com>.
On Fri, Dec 2, 2016 at 2:41 AM, Javen O'Neal <ja...@gmail.com> wrote:
> CPU pegged due the the Java garbage collector trying to clean up unused
> memory. The out of memory error is raised when the garbage collector cannot
> free the necessary memory for new objects to be allocated.
>
> Have you read the FAQ yet? https://poi.apache.org/faq.html#faq-N10165
>
> On Dec 2, 2016 00:11, "dokia@126.com" <do...@126.com> wrote:
>
> > Hi!
> >
> > I am trying to read a xlsx-file whose size is 7.3M with POI version 3.15,
> > here is the code：
> >
> >
> > *InputStream is = new FileInputStream(new File(path));//line 1Workbook
> book = new XSSFWorkbook(in);// line 2*
> > When running to line 2，the memory of javaw rised
> > violently from 700M to 1200M，cpu from 10% to 90%，the following
> > is the screenshot of memory dump:
> >
>
See http://poi.apache.org/spreadsheet/quick-guide.html#Files+vs+InputStreams,
also.
Re: Too much memory is used when reading a xlsx-file whose size is
just 7.3M
Posted by Javen O'Neal <ja...@gmail.com>.
CPU pegged due the the Java garbage collector trying to clean up unused
memory. The out of memory error is raised when the garbage collector cannot
free the necessary memory for new objects to be allocated.
Have you read the FAQ yet? https://poi.apache.org/faq.html#faq-N10165
On Dec 2, 2016 00:11, "dokia@126.com" <do...@126.com> wrote:
> Hi!
>
> I am trying to read a xlsx-file whose size is 7.3M with POI version 3.15,
> here is the code：
>
>
> *InputStream is = new FileInputStream(new File(path));//line 1Workbook book = new XSSFWorkbook(in);// line 2*
> When running to line 2，the memory of javaw rised
> violently from 700M to 1200M，cpu from 10% to 90%，the following
> is the screenshot of memory dump:
>
> In the picture, the object of Xobj$AttrXobj has more than 5 millions so
> that it occupy the most of memory，but the file size is just 7.3M.
> Can someone help me?
>
>
>
>
> Thank you!
>
> ------------------------------
> dokia@126.com
>