[jira] [Commented] (UNOMI-739) Implemenation of purge of sessions / events following rollover usage.

["Kevan Jahanshahi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/UNOMI-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687003#comment-17687003 ] 

Kevan Jahanshahi commented on UNOMI-739:
----------------------------------------

* Since the rollover conf is shared between sessions and events, could make sense to also share the purge conf ? (just a question, I'm also ok for having two diff conf regarding the number of months maximum)
 * We need to be careful to not delete normal empty indices (like a rollover just happen, and not session is created yet, if the purge is triggered it should not delete that index.)

Overal ok with this proposal that is simple and inspired by the existing purge system used for the profiles.

> Implemenation of purge of sessions / events following rollover usage.
> ---------------------------------------------------------------------
>
>                 Key: UNOMI-739
>                 URL: https://issues.apache.org/jira/browse/UNOMI-739
>             Project: Apache Unomi
>          Issue Type: New Feature
>            Reporter: David Griffon
>            Priority: Major
>
> Currently the configuration is set by:
> {code:java}
> # Purge all monthly indexes (sessions/events) that have been created for a specific number of months
> org.apache.unomi.monthly.index.purge.existTime=${env:UNOMI_MONTHLY_INDEX_PURGE_EXISTTIME:-12}
> {code}
> it has to be changed to allow a purge by type: session and event
> {code:java}
> org.apache.unomi.sessions.purge.existTime=${env:UNOMI_SESSIONS_PURGE_EXISTTIME:-}
> org.apache.unomi.events.purge.existTime=${env:UNOMI_EVENTS_PURGE_EXISTTIME:-}
> {code}
> *Old value should be deprecated and use as default if the new one are not set.*
> h2. Implementation
> 2 mechanisms need to be implemented in the purge of sessions / events:
> h3. Deleting old documents:
> There is a job that currently purge monthly indices and profiles, the goal is to replace the [monthly indices purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L402] by a dedicated method to remove events and sessions.
> This method will work as the [profile purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L333] calling {{{}persistenceService.removeByQuery(..){}}}.
> Like in here:
> [https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L365]
> *A log need to be added to mention the number of document removed.*
> h3. Removing empty indices.
> When documents are deleted, it will happen that some indices contain no document, we need to have a dedicated execution to remove the empty indices.
> *A log need to be added to mention the deleted indices and the remaining indices with the number of documents in each.*
> h2. Handle TODOs
> Look for UNOMI-726 todos within the code:
>  - Remove index deletion from the rollup configuration.
>  - Enable back and adapt integration test related to the purge.
>  - Remove the delete actions in the rollover configuration in the migration file of the event/session indexes. (create_rollover_policy_query.json file) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)