[jira] [Created] (UNOMI-946) Add search engine cluster health checks and replica enforcement to IT harness before and after migration

["Serge Huber (Jira)" <ji...@apache.org>]

Serge Huber created UNOMI-946:
---------------------------------

             Summary: Add search engine cluster health checks and replica enforcement to IT harness before and after migration
                 Key: UNOMI-946
                 URL: https://issues.apache.org/jira/browse/UNOMI-946
             Project: Apache Unomi
          Issue Type: Sub-task
          Components: unomi(-core)
    Affects Versions: unomi-3.1.0
            Reporter: Serge Huber
             Fix For: unomi-3.1.0


The IT harness can enter a yellow/red cluster state during the test suite due to two root causes:
 # *Replica settings cannot be set via container environment variables.* {{index.number_of_replicas}} is an index-level setting in ES 9.x and OS 3.x; it is explicitly rejected in node config / Docker env vars. The current {{pom.xml}} correctly uses env vars for routing settings (e.g. {{{}cluster.routing.allocation.disk.threshold_enabled{}}}) but has no mechanism to enforce 0 replicas on indices.

 # *ES 9.x creates system data streams with {{number_of_replicas=1}} during test execution.* The cluster starts GREEN (Unomi's own indices have {{{}numberOfReplicas=0{}}}), but as the test suite progresses, ES internally creates background data streams (deprecation log streams, ILM bookkeeping, etc.) with the default replica count. On a single-node container these cannot be allocated, transitioning the cluster to YELLOW.

 # *Snapshot restore brings back 1.6.x indices with their original replica settings.* {{Migrate16xToCurrentVersionIT}} restores a snapshot taken against a cluster with the ES default {{{}number_of_replicas=1{}}}. All restored indices come back with 1 replica, immediately putting the cluster in a YELLOW state that the migration then runs against.

 # *{{minimalClusterState}} asymmetry in BaseIT.* The OpenSearch path in BaseIT already overrides {{minimalClusterState=YELLOW}} (line 619), but the Elasticsearch path has no equivalent override, so {{ElasticSearchHealthCheckProvider}} returns {{UP}} instead of {{LIVE}} and {{HealthCheckIT}} fails.

*Observed symptom:* {{HealthCheckIT.testHealthCheck}} and {{testConcurrentHealthCheck}} fail consistently when run with the Elasticsearch profile because the health endpoint returns HTTP 206 with {{elasticsearch.status=UP}} instead of {{{}LIVE{}}}.
----
*Proposed changes:*

Three check/fix anchor points, all using direct HTTP calls to {{localhost:\{getSearchPort()}}} (container is up, security is disabled, no auth required):

*Point 0 — {{BaseIT.checkSearchEngine()}} (runs first in both migration and non-migration paths)*
 * {{PUT /_cluster/settings}} → {{{"persistent": \{"index.number_of_replicas": "0"}}}} — enforces 0 replicas as the cluster-wide default for all subsequently created indices (Unomi indices, ES system indices, migration reindex targets)
 * {{GET /_cluster/health}} + log baseline cluster status

*Point 1 — {{{}Migrate16xToCurrentVersionIT.waitForStartup(){}}}, immediately after snapshot restore*
 * {{PUT /_all/_settings}} → {{{"index": \{"number_of_replicas": "0"}}}} — retroactively zeros out replicas on all restored 1.6.x indices
 * {{GET /_cluster/health?wait_for_status=green&timeout=30s}} — wait for GREEN before starting the migration; fail fast with diagnosis if not reached

*Point 2 — After migration + {{super.waitForStartup()}} (migration path) and after {{unomi:start}} (non-migration path)*
 * {{GET /_cat/indices?h=index,rep,health&format=json}} — assert no index has {{{}rep > 0{}}}; log any violations
 * {{GET /_cluster/health}} — assert GREEN; surface which indices have unassigned shards if not

Additionally, add the missing {{minimalClusterState=YELLOW}} override for Elasticsearch to BaseIT (mirroring the OpenSearch line) as a belt-and-suspenders fallback.

*Code structure:*
 * {{BaseIT.configureSearchEngineForTesting()}} — reusable method for Point 0, called from {{checkSearchEngine()}}
 * {{BaseIT.assertClusterHealthy(String context)}} — reusable method for Point 2
 * {{Migrate16xToCurrentVersionIT.fixRestoredIndexReplicas()}} — migration-specific Point 1

----
*Acceptance criteria:*
 *  {{HealthCheckIT.testHealthCheck}} and {{testConcurrentHealthCheck}} pass consistently with the Elasticsearch profile on a single-node container
 *  {{Migrate16xToCurrentVersionIT}} restores the 1.6.x snapshot and runs the migration against a GREEN cluster
 *  Any index with {{number_of_replicas > 0}} after startup or migration causes a clear, early diagnostic failure with the index name(s) listed — not a silent YELLOW cluster
 *  Both Elasticsearch and OpenSearch paths are covered
 *  {{configureSearchEngineForTesting()}} and {{assertClusterHealthy()}} are reusable from any future migration IT



--
This message was sent by Atlassian Jira
(v8.20.10#820010)