You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by "David Griffon (Jira)" <ji...@apache.org> on 2023/02/09 07:53:00 UTC
[jira] [Comment Edited] (UNOMI-726) Session/Event index rollover: purge mechanism proposal
[ https://issues.apache.org/jira/browse/UNOMI-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686046#comment-17686046 ]
David Griffon edited comment on UNOMI-726 at 2/9/23 7:52 AM:
-------------------------------------------------------------
Currently the configuration is set by:
{code}
# Purge all monthly indexes (sessions/events) that have been created for a specific number of months
org.apache.unomi.monthly.index.purge.existTime=${env:UNOMI_MONTHLY_INDEX_PURGE_EXISTTIME:-12}
{code}
it has to be changed to allow a purge by type: session and event
{code}
org.apache.unomi.sessions.purge.existTime=${env:UNOMI_SESSIONS_PURGE_EXISTTIME:-}
org.apache.unomi.events.purge.existTime=${env:UNOMI_EVENTS_PURGE_EXISTTIME:-}
{code}
Old value should be deprecated and use as default if the new one are not set.
2 mechanisms are involved in the purge of sessions / events:
h3. Deleting old documents:
There is a job that currently purge monthly indices and profiles, the goal is to replace the [monthly indices purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L402] by a dedicated method to remove events and sessions.
This method will work as the [profile purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L333] calling {{persistenceService.removeByQuery(..)}}.
Like in here:
https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L365
*A log need to be added to mention the number of document removed.*
h3. Removing empty indices.
When documents are deleted, it will happen that some indices contain no document, we need to have a dedicated execution to remove the empty indices.
*A log need to be added to mention the deleted indices and the remaining indices with the number of documents in each.*
was (Author: dgriffon):
Currently the configuration is set by:
{code}
# Purge all monthly indexes (sessions/events) that have been created for a specific number of months
org.apache.unomi.monthly.index.purge.existTime=${env:UNOMI_MONTHLY_INDEX_PURGE_EXISTTIME:-12}
{code}
it has to be changed to allow a purge by type: session and event
{code}
org.apache.unomi.sessions.purge.existTime=${env:UNOMI_SESSIONS_PURGE_EXISTTIME:-}
org.apache.unomi.events.purge.existTime=${env:UNOMI_EVENTS_PURGE_EXISTTIME:-}
{code}
Old value should be deprecated and use as default if the new one are not set.
2 mechanisms are involved in the purge of sessions / events:
h3. Deleting old documents:
There is a job that currently purge monthly indices and profiles, the goal is to replace the [monthly indices purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L402] by a dedicated method to remove events and sessions.
This method will work as the [profile purge|https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L333] calling {{persistenceService.removeByQuery(..)}}.
Like in here:
https://github.com/apache/unomi/blob/7feff5c5067ff679a079211d9f2eb989898efc75/services/src/main/java/org/apache/unomi/services/impl/profiles/ProfileServiceImpl.java#L365
Log need to be added to mention the number of document removed.
h3. Removing empty indices.
When documents are deleted, it will happen that some indices contain no document, we need to have a dedicated execution to remove the empty indices.
A log need to be added to mention the deleted indices and the remaining indices with the number of documents in each.
> Session/Event index rollover: purge mechanism proposal
> ------------------------------------------------------
>
> Key: UNOMI-726
> URL: https://issues.apache.org/jira/browse/UNOMI-726
> Project: Apache Unomi
> Issue Type: Improvement
> Reporter: Kevan Jahanshahi
> Assignee: David Griffon
> Priority: Major
> Fix For: unomi-2.2.0
>
>
> Purge system was design to match previous monthly indices setup, since we switch to rollover system the entire purge need to be rework to match expected specification:
> ToDo in current ticket:
> * specification for purge system and expected threshold to be configurable to purge events and sessions
> * implementation proposal for the new purge system based on rollover indices
--
This message was sent by Atlassian Jira
(v8.20.10#820010)