[jira] [Resolved] (UNOMI-724) Session/Event index rollover: Implement ElasticSearch rollover recommandation and system to manage current monthly indices

["Kevan Jahanshahi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/UNOMI-724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kevan Jahanshahi resolved UNOMI-724.
------------------------------------
    Resolution: Fixed

> Session/Event index rollover: Implement ElasticSearch rollover recommandation and system to manage current monthly indices
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: UNOMI-724
>                 URL: https://issues.apache.org/jira/browse/UNOMI-724
>             Project: Apache Unomi
>          Issue Type: Improvement
>    Affects Versions: unomi-2.1.0
>            Reporter: Kevan Jahanshahi
>            Assignee: David Griffon
>            Priority: Major
>             Fix For: unomi-2.2.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> A POC have been done to validate the usage of Rollover ES API that combine a lifecycle policy and aliases to replace the current monthly indices rotation.
> The goal behind that is to apply current ElasticSearch recommandation on Index/Shards sizing to reduce costs related to monthly indices not being sized correctly due to unpredictable amount of events/sessions per months.
> POC PR: [https://github.com/apache/unomi/pull/567]
> h2. Prerequisite before starting this ticket:
> Read carefully ES documentation regarding:
>  * Rollover API: [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html] 
> h2. What remain to be done in current ticket ?
>  * Cleanup the POC
>  ** cleanup used old functions from previous implem
>  ** deprecate old param not used like: *dateHint*
>  ** global code review and cleanup of the new implem (if necessary contact me directly to discuss improvement point)
>  * make the rollover configurable using configuration file ({*}max_age, max_size, max_primary_shard_size, max_docs.{*} See documentations regarding the rollover capabilities: [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html] )
>  * Disable/quarantine tests associated with the purge, since these tests will be broken as a result of this story. Create ticket to re-enable the tests once epic is complete and before releasing the next version of Unomi.
> Out of current ticket scope: 
>  - check performance for session loaded using query: https://issues.apache.org/jira/browse/UNOMI-725
>  - fixing the purge system: https://issues.apache.org/jira/browse/UNOMI-726
>  - fixing the merge system: https://issues.apache.org/jira/browse/UNOMI-727
>  - handle migration: https://issues.apache.org/jira/browse/UNOMI-728
> h2. Automated test
> Query Elasticsearch to verify lifecycle policy once the configuration has been applied in Unomi (this way we do not test Elasticsearch rollover mechanism but ensure the configuration is properly applied).
> h2. Manual Test Case: 
>  - Pick one of the configuration (such as max_docs), configure Unomi with a very low value
>  - Create anonymous session
>  - Verify rotation is happening when sessions are created and get above the configured threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)