[jira] [Created] (UNOMI-753) Avoid loading items in RAM when possible, use updateByScript or scroll queries

["Kevan Jahanshahi (Jira)" <ji...@apache.org>]

Kevan Jahanshahi created UNOMI-753:
--------------------------------------

             Summary: Avoid loading items in RAM when possible, use updateByScript or scroll queries
                 Key: UNOMI-753
                 URL: https://issues.apache.org/jira/browse/UNOMI-753
             Project: Apache Unomi
          Issue Type: Improvement
            Reporter: Kevan Jahanshahi


There is place in Unomi code base where we load all profiles in RAM, for example:

[https://github.com/apache/unomi/blob/master/extensions/lists-extension/services/src/main/java/org/apache/unomi/services/UserListServiceImpl.java#L88-L97]
{code:java}
List<Profile> profiles = persistenceService.query(query, null, Profile.class);{code}
This can lead to OOM issue depending on the number of profiles in ES. But it can also lead to inconsistencies due to the usage of bulkProcessor update:
{code:java}
persistenceService.update(p, Profile.class, "systemProperties", profileSystemProperties);{code}
the bulk processor is retaining the actions and will push them asynchronously. If a profile is modified in the meantime the bulkprocessor is retaining a profile to be updated, then when the bulk is performed it will overrides profile data, and this can lead to inconsistencies.

2 risks identified here, with this kind of pattern:
 * OOM: in case a lot of profiles in the user list
 * inconsistency due to asynchronous bulkprocessor overwritting the profiles systemProperties

Others places may be impacted, and should be fixed in current ticket.

To identify other places: 
 - check usages of *persistenceService.update()*

One usage have been updated already:

Take a look at *MergeProfilesOnPropertyAction.reassignPersistedBrowsingDatasAsync*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)