You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Tony Giaccone <to...@giaccone.org> on 2018/12/04 20:57:51 UTC
Increasing memory leads to OOM error
We're processing a request and generating about 130,000 new objects. Once
these objects are created and saved we'll never need to access them again
in this process. The heap steadily grows as we do this. Even thought after
each insert we invalidate the object we just inserted.
What should I do to minimize the amount of memory that's held by Cayenne as
we do this large number of inserts? Am I missing something? Are there
snapshots for each of these objects? If there are is there an way to purge
those?
Tony
Re: Increasing memory leads to OOM error
Posted by Tony <to...@giaccone.org>.
I think the memory growth now is not related to Cayenne or Bootique.
Tony Giaccone
> On Dec 5, 2018, at 7:58 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
>
> Invalidation goes all the way down from the context to the shared snapshot cache. So maybe this is the shared cache to blame. It is only 10K snapshots max by default though, so not sure why would that be an issue (unless you increased it explicitly via 'cayenne.DataRowStore.snapshot.size' property).
>
> Andrus
>
>> On Dec 5, 2018, at 3:46 PM, Tony Giaccone <an...@nytimes.com> wrote:
>>
>> In the end we invalidated all the objects we inserted and that solved the memory problem. The memory still climbs slightly but nothing like what was happening before the invalidation.
>>
>> There are other features that have PRs ready that are likely to cause the memory footprint to grow. But that’s a totally different problem. :-)
>>
>>
>> Tony Giaccone
>>
>>> On Dec 5, 2018, at 2:00 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
>>>
>>> While this wouldn't hurt, technically it should not be needed, as ObjectContext has weak references to "untouched" objects, so they will be GC'd sooner or later.
>>>
>>> I'd say the shared snapshot cache may be the culprit, but Tony says the following:
>>>
>>>>> Even thought after each insert we invalidate the object we just inserted.
>>>
>>> So this is something else. So I'd suggest doing memory profiling to see which code retains the objects. This may not be Cayenne.
>>>
>>> Andrus
>>>
>>>
>>>> On Dec 5, 2018, at 1:13 AM, John Huss <jo...@gmail.com> wrote:
>>>>
>>>> I would try creating a new ObjectContext every so often, not just
>>>> invalidating the objects.
>>>>
>>>>> On Tue, Dec 4, 2018 at 2:58 PM Tony Giaccone <to...@giaccone.org> wrote:
>>>>>
>>>>> We're processing a request and generating about 130,000 new objects. Once
>>>>> these objects are created and saved we'll never need to access them again
>>>>> in this process. The heap steadily grows as we do this. Even thought after
>>>>> each insert we invalidate the object we just inserted.
>>>>>
>>>>> What should I do to minimize the amount of memory that's held by Cayenne as
>>>>> we do this large number of inserts? Am I missing something? Are there
>>>>> snapshots for each of these objects? If there are is there an way to purge
>>>>> those?
>>>>>
>>>>>
>>>>> Tony
>>>>>
>>>
>
Re: Increasing memory leads to OOM error
Posted by Andrus Adamchik <an...@objectstyle.org>.
Invalidation goes all the way down from the context to the shared snapshot cache. So maybe this is the shared cache to blame. It is only 10K snapshots max by default though, so not sure why would that be an issue (unless you increased it explicitly via 'cayenne.DataRowStore.snapshot.size' property).
Andrus
> On Dec 5, 2018, at 3:46 PM, Tony Giaccone <an...@nytimes.com> wrote:
>
> In the end we invalidated all the objects we inserted and that solved the memory problem. The memory still climbs slightly but nothing like what was happening before the invalidation.
>
> There are other features that have PRs ready that are likely to cause the memory footprint to grow. But that’s a totally different problem. :-)
>
>
> Tony Giaccone
>
>> On Dec 5, 2018, at 2:00 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
>>
>> While this wouldn't hurt, technically it should not be needed, as ObjectContext has weak references to "untouched" objects, so they will be GC'd sooner or later.
>>
>> I'd say the shared snapshot cache may be the culprit, but Tony says the following:
>>
>>>> Even thought after each insert we invalidate the object we just inserted.
>>
>> So this is something else. So I'd suggest doing memory profiling to see which code retains the objects. This may not be Cayenne.
>>
>> Andrus
>>
>>
>>> On Dec 5, 2018, at 1:13 AM, John Huss <jo...@gmail.com> wrote:
>>>
>>> I would try creating a new ObjectContext every so often, not just
>>> invalidating the objects.
>>>
>>>> On Tue, Dec 4, 2018 at 2:58 PM Tony Giaccone <to...@giaccone.org> wrote:
>>>>
>>>> We're processing a request and generating about 130,000 new objects. Once
>>>> these objects are created and saved we'll never need to access them again
>>>> in this process. The heap steadily grows as we do this. Even thought after
>>>> each insert we invalidate the object we just inserted.
>>>>
>>>> What should I do to minimize the amount of memory that's held by Cayenne as
>>>> we do this large number of inserts? Am I missing something? Are there
>>>> snapshots for each of these objects? If there are is there an way to purge
>>>> those?
>>>>
>>>>
>>>> Tony
>>>>
>>
Re: Increasing memory leads to OOM error
Posted by Tony Giaccone <an...@nytimes.com>.
In the end we invalidated all the objects we inserted and that solved the memory problem. The memory still climbs slightly but nothing like what was happening before the invalidation.
There are other features that have PRs ready that are likely to cause the memory footprint to grow. But that’s a totally different problem. :-)
Tony Giaccone
> On Dec 5, 2018, at 2:00 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
>
> While this wouldn't hurt, technically it should not be needed, as ObjectContext has weak references to "untouched" objects, so they will be GC'd sooner or later.
>
> I'd say the shared snapshot cache may be the culprit, but Tony says the following:
>
>>> Even thought after each insert we invalidate the object we just inserted.
>
> So this is something else. So I'd suggest doing memory profiling to see which code retains the objects. This may not be Cayenne.
>
> Andrus
>
>
>> On Dec 5, 2018, at 1:13 AM, John Huss <jo...@gmail.com> wrote:
>>
>> I would try creating a new ObjectContext every so often, not just
>> invalidating the objects.
>>
>>> On Tue, Dec 4, 2018 at 2:58 PM Tony Giaccone <to...@giaccone.org> wrote:
>>>
>>> We're processing a request and generating about 130,000 new objects. Once
>>> these objects are created and saved we'll never need to access them again
>>> in this process. The heap steadily grows as we do this. Even thought after
>>> each insert we invalidate the object we just inserted.
>>>
>>> What should I do to minimize the amount of memory that's held by Cayenne as
>>> we do this large number of inserts? Am I missing something? Are there
>>> snapshots for each of these objects? If there are is there an way to purge
>>> those?
>>>
>>>
>>> Tony
>>>
>
Re: Increasing memory leads to OOM error
Posted by Andrus Adamchik <an...@objectstyle.org>.
While this wouldn't hurt, technically it should not be needed, as ObjectContext has weak references to "untouched" objects, so they will be GC'd sooner or later.
I'd say the shared snapshot cache may be the culprit, but Tony says the following:
>> Even thought after each insert we invalidate the object we just inserted.
So this is something else. So I'd suggest doing memory profiling to see which code retains the objects. This may not be Cayenne.
Andrus
> On Dec 5, 2018, at 1:13 AM, John Huss <jo...@gmail.com> wrote:
>
> I would try creating a new ObjectContext every so often, not just
> invalidating the objects.
>
> On Tue, Dec 4, 2018 at 2:58 PM Tony Giaccone <to...@giaccone.org> wrote:
>
>> We're processing a request and generating about 130,000 new objects. Once
>> these objects are created and saved we'll never need to access them again
>> in this process. The heap steadily grows as we do this. Even thought after
>> each insert we invalidate the object we just inserted.
>>
>> What should I do to minimize the amount of memory that's held by Cayenne as
>> we do this large number of inserts? Am I missing something? Are there
>> snapshots for each of these objects? If there are is there an way to purge
>> those?
>>
>>
>> Tony
>>
Re: Increasing memory leads to OOM error
Posted by John Huss <jo...@gmail.com>.
I would try creating a new ObjectContext every so often, not just
invalidating the objects.
On Tue, Dec 4, 2018 at 2:58 PM Tony Giaccone <to...@giaccone.org> wrote:
> We're processing a request and generating about 130,000 new objects. Once
> these objects are created and saved we'll never need to access them again
> in this process. The heap steadily grows as we do this. Even thought after
> each insert we invalidate the object we just inserted.
>
> What should I do to minimize the amount of memory that's held by Cayenne as
> we do this large number of inserts? Am I missing something? Are there
> snapshots for each of these objects? If there are is there an way to purge
> those?
>
>
> Tony
>