You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by "Kevan Jahanshahi (Jira)" <ji...@apache.org> on 2023/03/10 16:00:00 UTC
[jira] [Comment Edited] (UNOMI-748) Unomi merge system is exposed to OOM
[ https://issues.apache.org/jira/browse/UNOMI-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699002#comment-17699002 ]
Kevan Jahanshahi edited comment on UNOMI-748 at 3/10/23 3:59 PM:
-----------------------------------------------------------------
Other impacted area: [https://github.com/apache/unomi/blob/master/extensions/lists-extension/services/src/main/java/org/apache/unomi/services/UserListServiceImpl.java#L88-L97]
loading all profiles in RAM:
{code:java}
List<Profile> profiles = persistenceService.query(query, null, Profile.class);{code}
Do asynchronous update using bulkProcessor:
{code:java}
persistenceService.update(p, Profile.class, "systemProperties", profileSystemProperties);{code}
{{{}{}}}It’s actually really dangerous, because the bulk processor is retaining the action, so the profile could be added to a user list in the mean time.
And once the bulk processor would execute it’s retain action, the profile system properties will be overwritten.
2 risks identified here:
* OOM: in case a lot of profiles in the user list
* inconsistency due to asynchronous bulkprocessor overwritting the profiles systemProperties
was (Author: jkevan):
Other impacted area: [https://github.com/apache/unomi/blob/master/extensions/lists-extension/services/src/main/java/org/apache/unomi/services/UserListServiceImpl.java#L88-L97]
loading all profiles in RAM:
{code:java}
List<Profile> profiles = persistenceService.query(query, null, Profile.class);{code}
{{{}{}}}Do asynchronous update using bulkProcessor:
{{}}
{code:java}
{code}
{{persistenceService.update(p, Profile.class, "systemProperties", profileSystemProperties);}}
{{{}{}}}It’s actually really dangerous, because the bulk processor is retaining the action, so the profile could be added to a user list in the mean time.
And once the bulk processor would execute it’s retain action, the profile system properties will be overwritten.
2 risks identified here:
* OOM: in case a lot of profiles in the user list
* inconsistency due to asynchronous bulkprocessor overwritting the profiles systemProperties
> Unomi merge system is exposed to OOM
> ------------------------------------
>
> Key: UNOMI-748
> URL: https://issues.apache.org/jira/browse/UNOMI-748
> Project: Apache Unomi
> Issue Type: Improvement
> Affects Versions: unomi-2.1.0
> Reporter: Kevan Jahanshahi
> Priority: Major
>
> currently the sessions/events *update* is using bulkProcessor and it is asynchronous, we never know when the bulk will be perform.
> * t{+}he benefit{+}: fast merge requests, the merge request is fast as nothing is retain, bulk processor will do the job in a separate thread.
> * {+}the cons{+}: {*}all previous sessions/events are first loaded in memory{*}, so in case of merging active profiles that contains a lot of past events/sessions, {{{}we could be exposed to OOM{}}}. {_}(We already had similar case with the purge that was loading all profiles in memory.{_})
> If we replace the *update(one item at a time)* by using {*}updateByQuery{*}, the request will loose it’s asynchronous nature provided by the so called: BulkProcessor.
> * {+}the benefit{+}: sessions, events not load in memory, no OOM possible
> * {+}the cons{+}: request will be synchron and {{{}we expose merge requests to timeout on client side{}}}. merge is actually trigger by the login on jExp side adding extra timing here could have bad impacts and side effects.
>
> Since none of this solution seem’s ok, the perfect solution should be a mix of both strength: * use *{{updateByQuery}}* in a separate thread to avoid retaining merge request
> *
> ** We have the OOM protection by not loading all the past events/sessions
> ** We have the asynchronous execution done in a separate thread/job to free the current request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)