[jira] [Created] (UNOMI-748) Unomi merge system is exposed to OOM

["Kevan Jahanshahi (Jira)" <ji...@apache.org>]

Kevan Jahanshahi created UNOMI-748:
--------------------------------------

             Summary: Unomi merge system is exposed to OOM
                 Key: UNOMI-748
                 URL: https://issues.apache.org/jira/browse/UNOMI-748
             Project: Apache Unomi
          Issue Type: Improvement
    Affects Versions: unomi-2.1.0
            Reporter: Kevan Jahanshahi


currently the sessions/events *update* is using bulkProcessor and it is asynchronous, we never know when the bulk will be perform.
 * t{+}he benefit{+}: fast merge requests, the merge request is fast as nothing is retain, bulk processor will do the job in a separate thread.
 * {+}the cons{+}: {*}all previous sessions/events are loaded in memory{*}, so in case of merging active profiles that contains a lot of past events/sessions, {{{}we are exposed to OOM here{}}}. {_}(We already had similar case with the purge that was loading all profiles in memory.{_})

If we replace the *update(one item at a time)* by using {*}updateByQuery{*}, the request will loose it’s asynchronous nature provided by the so called: BulkProcessor.
 * {+}the benefit{+}: sessions, events not load in memory, no OOM possible
 * {+}the cons{+}: request will be synchron and {{{}we expose merge requests to timeout on client side{}}}. merge is actually trigger by the login on jExp side adding extra timing here could have bad impacts and side effects.

 
Since none of this solution seem’s ok, the perfect solution should be a mix of both strength: * use *{{updateByQuery}}* in a separate thread to avoid retaining merge request
 ** We have the OOM protection by not loading all the past events/sessions
 ** We have the asynchronous execution done in a separate thread/job to free the current request.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)