You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ofbiz.apache.org by Pushpraj Singh Jadoun <pu...@gmail.com> on 2024/11/23 08:40:06 UTC
Re: OutOfMemoryError in Webtools When Fetching Large Dataset
Hello Deepak,
I believe this issue occurs because all the records are being fetched from
the database, and handling such a large dataset in OFBiz is causing an
*OutOfMemoryError*.
I also tried unchecking the "Ignore Case" option, but the error persists.
The *executeFind* service is responsible for querying results from the
database. I noticed that it uses the *maxRows* method to limit the number
of records fetched. For example, on the first page, 20 records are shown by
default, so only 20 records are fetched from the database using the
*maxRows* method. However, when navigating to the next page, 40 records are
fetched instead of just the next 20, because the *maxRows* value passed is
40. Similarly, when navigating to the last page of a dataset containing 3
million records, the *maxRows* value passed is 3 million, causing all 3
million records to be fetched from the database.
Would it be possible to use *offset* and *limit* to fetch only the
necessary records from the database, rather than retrieving unnecessary
ones?
Thanks & regards
Pushpraj SIngh Jadoun
On Thu, 17 Oct 2024 at 12:40, Deepak Dixit <de...@apache.org> wrote:
> Hi Puspraj,
>
> When enabled, the "Ignore Case" option converts field names to uppercase
> before querying. If there's any case difference in the field names, the
> database index may not function as expected, which causes the query to
> traverse the entire table to filter results. This can significantly impact
> performance.
>
> By default ignore cases should be unchecked!!
>
>
> Thanks & Regards
> --
> Deepak Dixit
> ofbiz.apache.org
>
>
> On Wed, Jun 5, 2024 at 3:44 PM Pushpraj Singh Jadoun <
> pushprajsinghjadoun123@gmail.com> wrote:
>
> > Hello Nicolas,
> >
> > I tested it with the following configuration:
> > org.gradle.jvmargs=-Xms128m -Xmx8192m -XX:+CMSClassUnloadingEnabled
> >
> > Kind regards,
> > Pushpraj SIngh Jadoun
> >
> > On Wed, 5 Jun 2024 at 13:32, Nicolas Malin <ni...@nereide.fr>
> > wrote:
> >
> >> Hello Pushpraj,
> >>
> >> How memory have you set at XMX value ?
> >>
> >> Nicolas
> >>
> >> Le 17/05/2024 à 17:08, Pushpraj Singh Jadoun a écrit :
> >>
> >> Hello all,
> >>
> >> I am facing an issue while fetching a large number of datasets (over 20
> >> million records) from an entity. Specifically, I receive the following
> >> error:
> >>
> >> "Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded"
> >>
> >>
> >> This error occurs when I attempt to navigate to the last page using the
> >> entity engine search page in Webtools. I have attached a screenshot for
> you
> >> to look over.
> >> [image: image-18- (1).png]
> >>
> >>
> >>
> >> [image: MAIL.png]
> >>
> >>
> >> Has anyone experienced a similar issue or can you offer advice on
> >> resolving this?
> >>
> >> Kind regards,
> >> Pushpraj SIngh Jadoun
> >>
> >>
> >>
>
Re: OutOfMemoryError in Webtools When Fetching Large Dataset
Posted by Pushpraj Singh Jadoun <pu...@gmail.com>.
Hello all,
To address this issue, I have created the Jira ticket OFBIZ-13366
<https://issues.apache.org/jira/browse/OFBIZ-13366>. All the relevant
details, analysis, and the proposed patch are available in the Jira ticket
itself.
The issue occurs because the executeFind service fetches records using the
maxRows value instead of applying proper pagination. As a result, when
navigating to later pages, an increasingly large number of records are
fetched from the database. For example, navigating to the last page of a
dataset containing millions of records causes all records to be loaded into
memory, eventually leading to an OutOfMemoryError.
To resolve this issue, we have updated the implementation to use LIMIT and
OFFSET so that only the required records are fetched from the database for
each page request instead of loading unnecessary records into memory.
Thanks & regards,
Pushpraj Singh Jadoun
On Sat, 23 Nov 2024 at 14:10, Pushpraj Singh Jadoun <
pushprajsinghjadoun123@gmail.com> wrote:
> Hello Deepak,
>
> I believe this issue occurs because all the records are being fetched from
> the database, and handling such a large dataset in OFBiz is causing an
> *OutOfMemoryError*.
>
> I also tried unchecking the "Ignore Case" option, but the error persists.
>
> The *executeFind* service is responsible for querying results from the
> database. I noticed that it uses the *maxRows* method to limit the number
> of records fetched. For example, on the first page, 20 records are shown by
> default, so only 20 records are fetched from the database using the
> *maxRows* method. However, when navigating to the next page, 40 records
> are fetched instead of just the next 20, because the *maxRows* value
> passed is 40. Similarly, when navigating to the last page of a dataset
> containing 3 million records, the *maxRows* value passed is 3 million,
> causing all 3 million records to be fetched from the database.
>
> Would it be possible to use *offset* and *limit* to fetch only the
> necessary records from the database, rather than retrieving unnecessary
> ones?
>
> Thanks & regards
> Pushpraj SIngh Jadoun
>
> On Thu, 17 Oct 2024 at 12:40, Deepak Dixit <de...@apache.org> wrote:
>
>> Hi Puspraj,
>>
>> When enabled, the "Ignore Case" option converts field names to uppercase
>> before querying. If there's any case difference in the field names, the
>> database index may not function as expected, which causes the query to
>> traverse the entire table to filter results. This can significantly impact
>> performance.
>>
>> By default ignore cases should be unchecked!!
>>
>>
>> Thanks & Regards
>> --
>> Deepak Dixit
>> ofbiz.apache.org
>>
>>
>> On Wed, Jun 5, 2024 at 3:44 PM Pushpraj Singh Jadoun <
>> pushprajsinghjadoun123@gmail.com> wrote:
>>
>> > Hello Nicolas,
>> >
>> > I tested it with the following configuration:
>> > org.gradle.jvmargs=-Xms128m -Xmx8192m -XX:+CMSClassUnloadingEnabled
>> >
>> > Kind regards,
>> > Pushpraj SIngh Jadoun
>> >
>> > On Wed, 5 Jun 2024 at 13:32, Nicolas Malin <ni...@nereide.fr>
>> > wrote:
>> >
>> >> Hello Pushpraj,
>> >>
>> >> How memory have you set at XMX value ?
>> >>
>> >> Nicolas
>> >>
>> >> Le 17/05/2024 à 17:08, Pushpraj Singh Jadoun a écrit :
>> >>
>> >> Hello all,
>> >>
>> >> I am facing an issue while fetching a large number of datasets (over 20
>> >> million records) from an entity. Specifically, I receive the following
>> >> error:
>> >>
>> >> "Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded"
>> >>
>> >>
>> >> This error occurs when I attempt to navigate to the last page using the
>> >> entity engine search page in Webtools. I have attached a screenshot
>> for you
>> >> to look over.
>> >> [image: image-18- (1).png]
>> >>
>> >>
>> >>
>> >> [image: MAIL.png]
>> >>
>> >>
>> >> Has anyone experienced a similar issue or can you offer advice on
>> >> resolving this?
>> >>
>> >> Kind regards,
>> >> Pushpraj SIngh Jadoun
>> >>
>> >>
>> >>
>>
>