You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@poi.apache.org by Moditha Hewasinghage <mo...@gmail.com> on 2021/08/06 09:44:37 UTC
Appending sheets to xlsx memory issues.
Hi everyone,
I am trying to append a sheet into an existing xlsx and encountering memory issues. However, If I do not open an existing xlsx I can write far bigger files. I have an example code snippet here:
https://snippet.host/epsv
With 2GB of memory I only manage to append 3 sheets before I run out of memory in wb.write().
But I can easily create a new workbook and append 10 sheets without any issue. Could you please tell me if I am doing a mistake somewhere or if there is an alternative for this.
Thank you,
Regards
Moditha
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: Appending sheets to xlsx memory issues.
Posted by Dominik Stadler <do...@gmx.at>.
I could not easily reproduce this, can you provide the original xlsx file
as well and attach it to a bug-entry at https://bz.apache.org/bugzilla/ so
we can track this there.
Thanks... Dominik.
On Wed, Aug 11, 2021 at 12:43 PM Moditha Hewasinghage <
modithadharmasiri@gmail.com> wrote:
> Hi Dominik,
>
> Yes I do have a loop of 100 just to see the failing point. Unfortunately
> it does not reach 100 iterations but fails after 3 (with 2GB memory). So I
> can have only 3 sheets in the "file" case. For "memory" I can have 10
> sheets.
>
> Moditha
>
> On 2021/08/11 08:30:44, Dominik Stadler <do...@gmx.at> wrote:
> > Didn't take a very close look, but it seems for the "file" case you run a
> > loop of 100 and continually overwrite the file with the enlarged one and
> > thus you add more and more sheets over time to the file. For "memory" you
> > do it differently with a loop of 10 and thus there is much less data in
> the
> > resulting workbook, or?
> >
> > Dominik.
> >
> > On Sat, Aug 7, 2021 at 9:21 PM Moditha Hewasinghage <
> > modithadharmasiri@gmail.com> wrote:
> >
> > > Hi Rob,
> > >
> > > As you can see in the example I am appending data sheets in both cases
> > > (same random data)
> > >
> > > On 2021/08/06 14:35:36, Rob Sargent <ro...@gmail.com> wrote:
> > > >
> > > >
> > > > > On Aug 6, 2021, at 3:44 AM, Moditha Hewasinghage <
> > > modithadharmasiri@gmail.com> wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I am trying to append a sheet into an existing xlsx and
> encountering
> > > memory issues. However, If I do not open an existing xlsx I can write
> far
> > > bigger files. I have an example code snippet here:
> > > > >
> > > > > https://snippet.host/epsv
> > > > >
> > > > > With 2GB of memory I only manage to append 3 sheets before I run
> out
> > > of memory in wb.write().
> > > > >
> > > > > But I can easily create a new workbook and append 10 sheets without
> > > any issue. Could you please tell me if I am doing a mistake somewhere
> or if
> > > there is an alternative for this.
> > > > >
> > > > Are you adding empty sheets in the latter and large, with-data
> > > sheets(either the existing or the appended) in the former?
> > > >
> > > > > Thank you,
> > > > > Regards
> > > > > Moditha
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > > > For additional commands, e-mail: user-help@poi.apache.org
> > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > > For additional commands, e-mail: user-help@poi.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > For additional commands, e-mail: user-help@poi.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
>
Re: Appending sheets to xlsx memory issues.
Posted by Moditha Hewasinghage <mo...@gmail.com>.
Hi Dominik,
Yes I do have a loop of 100 just to see the failing point. Unfortunately it does not reach 100 iterations but fails after 3 (with 2GB memory). So I can have only 3 sheets in the "file" case. For "memory" I can have 10 sheets.
Moditha
On 2021/08/11 08:30:44, Dominik Stadler <do...@gmx.at> wrote:
> Didn't take a very close look, but it seems for the "file" case you run a
> loop of 100 and continually overwrite the file with the enlarged one and
> thus you add more and more sheets over time to the file. For "memory" you
> do it differently with a loop of 10 and thus there is much less data in the
> resulting workbook, or?
>
> Dominik.
>
> On Sat, Aug 7, 2021 at 9:21 PM Moditha Hewasinghage <
> modithadharmasiri@gmail.com> wrote:
>
> > Hi Rob,
> >
> > As you can see in the example I am appending data sheets in both cases
> > (same random data)
> >
> > On 2021/08/06 14:35:36, Rob Sargent <ro...@gmail.com> wrote:
> > >
> > >
> > > > On Aug 6, 2021, at 3:44 AM, Moditha Hewasinghage <
> > modithadharmasiri@gmail.com> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I am trying to append a sheet into an existing xlsx and encountering
> > memory issues. However, If I do not open an existing xlsx I can write far
> > bigger files. I have an example code snippet here:
> > > >
> > > > https://snippet.host/epsv
> > > >
> > > > With 2GB of memory I only manage to append 3 sheets before I run out
> > of memory in wb.write().
> > > >
> > > > But I can easily create a new workbook and append 10 sheets without
> > any issue. Could you please tell me if I am doing a mistake somewhere or if
> > there is an alternative for this.
> > > >
> > > Are you adding empty sheets in the latter and large, with-data
> > sheets(either the existing or the appended) in the former?
> > >
> > > > Thank you,
> > > > Regards
> > > > Moditha
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > > For additional commands, e-mail: user-help@poi.apache.org
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > For additional commands, e-mail: user-help@poi.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > For additional commands, e-mail: user-help@poi.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: Appending sheets to xlsx memory issues.
Posted by Dominik Stadler <do...@gmx.at>.
Didn't take a very close look, but it seems for the "file" case you run a
loop of 100 and continually overwrite the file with the enlarged one and
thus you add more and more sheets over time to the file. For "memory" you
do it differently with a loop of 10 and thus there is much less data in the
resulting workbook, or?
Dominik.
On Sat, Aug 7, 2021 at 9:21 PM Moditha Hewasinghage <
modithadharmasiri@gmail.com> wrote:
> Hi Rob,
>
> As you can see in the example I am appending data sheets in both cases
> (same random data)
>
> On 2021/08/06 14:35:36, Rob Sargent <ro...@gmail.com> wrote:
> >
> >
> > > On Aug 6, 2021, at 3:44 AM, Moditha Hewasinghage <
> modithadharmasiri@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > I am trying to append a sheet into an existing xlsx and encountering
> memory issues. However, If I do not open an existing xlsx I can write far
> bigger files. I have an example code snippet here:
> > >
> > > https://snippet.host/epsv
> > >
> > > With 2GB of memory I only manage to append 3 sheets before I run out
> of memory in wb.write().
> > >
> > > But I can easily create a new workbook and append 10 sheets without
> any issue. Could you please tell me if I am doing a mistake somewhere or if
> there is an alternative for this.
> > >
> > Are you adding empty sheets in the latter and large, with-data
> sheets(either the existing or the appended) in the former?
> >
> > > Thank you,
> > > Regards
> > > Moditha
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > > For additional commands, e-mail: user-help@poi.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > For additional commands, e-mail: user-help@poi.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
>
Re: Appending sheets to xlsx memory issues.
Posted by Moditha Hewasinghage <mo...@gmail.com>.
Hi Rob,
As you can see in the example I am appending data sheets in both cases (same random data)
On 2021/08/06 14:35:36, Rob Sargent <ro...@gmail.com> wrote:
>
>
> > On Aug 6, 2021, at 3:44 AM, Moditha Hewasinghage <mo...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > I am trying to append a sheet into an existing xlsx and encountering memory issues. However, If I do not open an existing xlsx I can write far bigger files. I have an example code snippet here:
> >
> > https://snippet.host/epsv
> >
> > With 2GB of memory I only manage to append 3 sheets before I run out of memory in wb.write().
> >
> > But I can easily create a new workbook and append 10 sheets without any issue. Could you please tell me if I am doing a mistake somewhere or if there is an alternative for this.
> >
> Are you adding empty sheets in the latter and large, with-data sheets(either the existing or the appended) in the former?
>
> > Thank you,
> > Regards
> > Moditha
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> > For additional commands, e-mail: user-help@poi.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org
Re: Appending sheets to xlsx memory issues.
Posted by Rob Sargent <ro...@gmail.com>.
> On Aug 6, 2021, at 3:44 AM, Moditha Hewasinghage <mo...@gmail.com> wrote:
>
> Hi everyone,
>
> I am trying to append a sheet into an existing xlsx and encountering memory issues. However, If I do not open an existing xlsx I can write far bigger files. I have an example code snippet here:
>
> https://snippet.host/epsv
>
> With 2GB of memory I only manage to append 3 sheets before I run out of memory in wb.write().
>
> But I can easily create a new workbook and append 10 sheets without any issue. Could you please tell me if I am doing a mistake somewhere or if there is an alternative for this.
>
Are you adding empty sheets in the latter and large, with-data sheets(either the existing or the appended) in the former?
> Thank you,
> Regards
> Moditha
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org