You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "w3ll1ngt (via GitHub)" <gi...@apache.org> on 2026/05/13 20:22:00 UTC

[PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

w3ll1ngt opened a new pull request, #13130:
URL: https://github.com/apache/ignite/pull/13130

   Thank you for submitting the pull request to the Apache Ignite.
   
   In order to streamline the review of the contribution 
   we ask you to ensure the following steps have been taken:
   
   ### The Contribution Checklist
   - [ ] There is a single JIRA ticket related to the pull request. 
   - [ ] The web-link to the pull request is attached to the JIRA ticket.
   - [ ] The JIRA ticket has the _Patch Available_ state.
   - [ ] The pull request body describes changes that have been made. 
   The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
   - [ ] The pull request title is treated as the final commit message. 
   The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
   - [ ] A reviewer has been mentioned through the JIRA comments 
   (see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers)) 
   - [ ] The pull request has been checked by the Teamcity Bot and 
   the `green visa` attached to the JIRA ticket (see tab `PR Check` at [TC.Bot - Instance 1](https://tcbot2.sbt-ignite-dev.ru/prs.html) or [TC.Bot - Instance 2](https://mtcga.gridgain.com/prs.html))
   
   ### Notes
   - [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
   - [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
   - [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
   - [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
   
   If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603405066


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.

Review Comment:
   confusing stuff was replaced by 'Cache or SQL schema changes can also trigger PME without server node loss. Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.' 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3261137688


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.
+
+When checking whether a cluster is healthy, start with topology and cluster state. The cluster should be in the expected state, usually ACTIVE, and the number of server and client nodes should be stable. If native persistence is enabled, the baseline should also be in the expected shape: for a stable deployment, the nodes that are expected to be online should appear online both in baseline-related metrics and in the SYS.BASELINE_NODES system view. Frequent unexpected topology changes are not normal and should be treated as a sign of node instability or network problems.
+
+Then check data safety and convergence. A healthy cluster does not have lost partitions, and consistency checks such as control.sh --cache idle_verify should not report conflict partitions when the cluster is idle. After a topology event, transient rebalancing is expected, but it should converge: KeysToRebalanceLeft should trend to zero, and partition states should settle back to OWNING rather than remain in MOVING, RENTING, or LOST.
+
+Next, check execution pressure. Communication, discovery, and thread-pool queues may spike under load, but they should not grow continuously. In Ignite, sustained growth of OutboundMessagesQueueSize, MessageWorkerQueueSize, or thread pool queue sizes means that the node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor: temporary backlog can happen, but a persistent backlog or repeating starvation warnings are signs of contention, hot partitions, or blocked internal processing. Use SYS.STRIPED_THREADPOOL_QUEUE, SYS.TRANSACTIONS, and SYS.SQL_QUERIES for a live view of the work that is not draining.
+
+Checkpointing and transactions should also remain bounded. Checkpoint activity can slow the cluster down, so LastCheckpointDuration should be monitored together with dirty pages and disk behavior. Transactions and queries can legitimately take longer during bursts, but healthy steady-state behavior means that lock-holding transactions, long-running transactions, and long-running SQL queries do not accumulate over time. If long transactions repeatedly block partition map exchange, use transaction timeout settings such as TxTimeoutOnPartitionMapExchange and investigate the application path that keeps transactions open.
+
+Finally, check the underlying JVM and critical workers. Ignite treats IgniteOutOfMemoryException, OutOfMemoryError, system worker termination, system worker hangs, and cluster node segmentation as critical failures. A healthy cluster should not emit blocked system-critical worker messages, and JVM resource pools should stay comfortably below exhaustion. In practice, monitor heap usage, direct buffer usage, and open file descriptors continuously, because all three are finite pools and approaching their limits usually means the node is already close to a failure condition rather than merely under benign load.

Review Comment:
   Completely refurbish this phrase. Before that, i meant to think about critical failures as smth that triggers FailureHandler. Thank you 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3239308916


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.
+
+When checking whether a cluster is healthy, start with topology and cluster state. The cluster should be in the expected state, usually ACTIVE, and the number of server and client nodes should be stable. If native persistence is enabled, the baseline should also be in the expected shape: for a stable deployment, the nodes that are expected to be online should appear online both in baseline-related metrics and in the SYS.BASELINE_NODES system view. Frequent unexpected topology changes are not normal and should be treated as a sign of node instability or network problems.
+
+Then check data safety and convergence. A healthy cluster does not have lost partitions, and consistency checks such as control.sh --cache idle_verify should not report conflict partitions when the cluster is idle. After a topology event, transient rebalancing is expected, but it should converge: KeysToRebalanceLeft should trend to zero, and partition states should settle back to OWNING rather than remain in MOVING, RENTING, or LOST.
+
+Next, check execution pressure. Communication, discovery, and thread-pool queues may spike under load, but they should not grow continuously. In Ignite, sustained growth of OutboundMessagesQueueSize, MessageWorkerQueueSize, or thread pool queue sizes means that the node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor: temporary backlog can happen, but a persistent backlog or repeating starvation warnings are signs of contention, hot partitions, or blocked internal processing. Use SYS.STRIPED_THREADPOOL_QUEUE, SYS.TRANSACTIONS, and SYS.SQL_QUERIES for a live view of the work that is not draining.
+
+Checkpointing and transactions should also remain bounded. Checkpoint activity can slow the cluster down, so LastCheckpointDuration should be monitored together with dirty pages and disk behavior. Transactions and queries can legitimately take longer during bursts, but healthy steady-state behavior means that lock-holding transactions, long-running transactions, and long-running SQL queries do not accumulate over time. If long transactions repeatedly block partition map exchange, use transaction timeout settings such as TxTimeoutOnPartitionMapExchange and investigate the application path that keeps transactions open.
+
+Finally, check the underlying JVM and critical workers. Ignite treats IgniteOutOfMemoryException, OutOfMemoryError, system worker termination, system worker hangs, and cluster node segmentation as critical failures. A healthy cluster should not emit blocked system-critical worker messages, and JVM resource pools should stay comfortably below exhaustion. In practice, monitor heap usage, direct buffer usage, and open file descriptors continuously, because all three are finite pools and approaching their limits usually means the node is already close to a failure condition rather than merely under benign load.

Review Comment:
   `as critical failures` - and what does it mean ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626936889


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.

Review Comment:
   thx



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627510164


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole MBeans view showing io/datastorage checkpoint attributes and a persistent dataregion bean]
+
+=== Check Logs and Optional Features
+
+Investigate logs and failure handling for:
+
+* node segmentation;
+* repeated system-worker blockage;
+* failure-handler activations;
+* `OutOfMemoryError` and `IgniteOutOfMemoryException`;
+* repeated striped-pool starvation warnings;
+* unexpected node restarts.
+
+Ignite reports critical failures to the configured failure handler; depending on the handler, the result can be node invalidation, failover handling, node stop, or JVM termination.

Review Comment:
   done with crosslinking



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626897019


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.

Review Comment:
   there will be detailed explanation of this metrics later on in arictle, but here is the first use of it in text. so i've linked it to their doc origins



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627040356


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.

Review Comment:
   i agree. it's not common use at all. From the "newbie hat" perspective it makes no sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627020229


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.

Review Comment:
   crosslinked



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603368485


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.

Review Comment:
   extended



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626757864


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626898295


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.

Review Comment:
   last line removed. finally



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3612347325


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.

Review Comment:
   All this terms need to be covered\documentally extended somehow ? PME - what does it mean ? Long-running transactions - how it can be detected ? and so on ...



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.

Review Comment:
   I misunderstand this section at all ( It gives nothing only confusing a reader, need to be refactored or removed.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.

Review Comment:
   ```suggestion
   Client nodes are not a part of the baseline.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.

Review Comment:
   "repeated membership churn" - i don\`t listen this term before, what does it mean ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.

Review Comment:
   ```suggestion
   The baseline contains server nodes that are intended to store a data.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.

Review Comment:
   'churn' - confusing, may be only for me ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.

Review Comment:
   PME - crosslink or extended description need to be passed



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.

Review Comment:
   Very confusing sentence and very scary, as for me. Plz explain - "If data is still missing" how it can happen ? 
   for example backup = 2 node1 - left the grid, node2 (with second backup) is still in topology, no autoadjust is happens, some writes are going into node2, further - lost node2 and return node1 = miss data, smth like this



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole MBeans view showing io/datastorage checkpoint attributes and a persistent dataregion bean]
+
+=== Check Logs and Optional Features
+
+Investigate logs and failure handling for:
+
+* node segmentation;
+* repeated system-worker blockage;
+* failure-handler activations;
+* `OutOfMemoryError` and `IgniteOutOfMemoryException`;
+* repeated striped-pool starvation warnings;
+* unexpected node restarts.
+
+Ignite reports critical failures to the configured failure handler; depending on the handler, the result can be node invalidation, failover handling, node stop, or JVM termination.

Review Comment:
   Oh ! seems we need to mention failure handler and it\`s configurations policies here ? wdyt ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics

Review Comment:
   ```suggestion
   JConsole view of node-local persistence checkpoint metrics
   ```
   is dot expected here ? i miss smth ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole MBeans view showing io/datastorage checkpoint attributes and a persistent dataregion bean]
+
+=== Check Logs and Optional Features
+
+Investigate logs and failure handling for:
+
+* node segmentation;
+* repeated system-worker blockage;
+* failure-handler activations;
+* `OutOfMemoryError` and `IgniteOutOfMemoryException`;
+* repeated striped-pool starvation warnings;
+* unexpected node restarts.
+
+Ignite reports critical failures to the configured failure handler; depending on the handler, the result can be node invalidation, failover handling, node stop, or JVM termination.
+See link:perf-and-troubleshooting/handling-exceptions[Handling Exceptions] for failure-handler behavior.
+No single local command guarantees the absence of split brain.
+Monitor membership from all expected monitoring points, and use logs and failure handling to detect segmentation and discovery failures.
+
+CDC is a feature-specific check, not part of a universal cluster health definition.
+If CDC is enabled, monitor `ignite-cdc.log`, WAL gaps, and consumption lag.

Review Comment:
   theme is not covered at all )) may be it\`s ok for this doc but if you have more info - plz append it



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.

Review Comment:
   I misunderstand here, what does it mean - " without server node loss"  ? May be i miss smth ? plz explain ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.

Review Comment:
   ```suggestion
   After a topology event raized, such as: a new node join\left, ???transient??? rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
   ```
   
   "converge"  - gives nothing, probably you need to show some metric examples ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.

Review Comment:
   PME metric - crosslink ? how to check this metric ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics

Review Comment:
   ```suggestion
   JConsole view of node-local striped executor queue metrics
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber

Review Comment:
   plz also show "ignite.log" output for checkpoint operations, how it looks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626975704


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.

Review Comment:
   i wanted to say that not only so called churn-change can trigger PME, but schema changes as well like cache start\stop. The user may want to know that there is no need to panic, it is not neccessary server loss or smth



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627506896


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #13130:
URL: https://github.com/apache/ignite/pull/13130#issuecomment-4447972099

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=13130) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=13130&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=13130&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=13130&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=13130&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=13130&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=13130)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3261084907


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.

Review Comment:
   yea, thx. Looks wierd at  second glance, i agree. i was intended to say smth like: "healthy cluster could not bejust defined by some simple memory metrics (number), but rather... whole complex system etc"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3586973582


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.

Review Comment:
   Just give me a clue here - this sentence is correct, but how it can be detected ? I think we need to form this document from position that reader is not deeply understant how it work and need to have some commands\logs how to understand (split brain\PME) situations ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.

Review Comment:
   Not clear, rewrite or expand it plz.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.

Review Comment:
   "deployment plan" - what does it mean ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:

Review Comment:
   When it can be enabled\disabled and what user need to do - if it  disabled ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.

Review Comment:
   From reader side this phrase is confusing, (you and me understand that it\`s true but i don\`t like to show it for reader) we need to remove it at all or extend this comment.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.

Review Comment:
   what does it mean in common ? Probably we need to mention baseline, partition loss ?



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.

Review Comment:
   I know that this is a common practise (in some installations) but, in common, it shows nothing, this command can be executed on client node without cluster nodes interaction.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.

Review Comment:
   What different between: 'Cache start' and 'distributed cache creation' ?
   'SQL table creation', and what about drop ?
   And over all - why this information is helpful in acpect of this document ?
   Probably you want to say smth., but i miss it (



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:

Review Comment:
   Plz append examples: 
   1. with one lost node
   2. with partition loss
    2a. if node will come back to cluster - it still has a partition loss and need to be reset - expand this plz.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.

Review Comment:
   "easy" - not an engeneering term ) what does it mean ? I\`d prefer to say like : Baseline change procedure will provide 'rebalance' which means that partitions will be rebased according to new affinity distribution.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.

Review Comment:
   control.**sh** - it\`s a linux specific



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).

Review Comment:
   'idle_verify' need to be cross-linked with appropriate chapter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603370192


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.

Review Comment:
   completely refurbished this line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3619384327


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics

Review Comment:
   dot is expected due eto ascii doc formatting (.Title for nearest image::)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627482754


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.

Review Comment:
   added on with some explanarory examples 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan merged PR #13130:
URL: https://github.com/apache/ignite/pull/13130


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603375836


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.

Review Comment:
   extended to fulfil this thought 



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603383461


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.

Review Comment:
   extended with condition-and-action structure. i think it is the clearest method of explanation  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3626784108


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.

Review Comment:
   churn-change describes sitations that happens in the set of participating nodes participating nodes due to joins, graceful leaves, and failures. For example, a high churn rate can increase costs or decrease service quality. [src](https://www.academia.edu/100244898/Minimizing_churn_in_distributed_systems): 
   
   as for this article, i'll replace it with more common wording



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627540127


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when operational constraints allow it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole MBeans view showing io/datastorage checkpoint attributes and a persistent dataregion bean]
+
+=== Check Logs and Optional Features
+
+Investigate logs and failure handling for:
+
+* node segmentation;
+* repeated system-worker blockage;
+* failure-handler activations;
+* `OutOfMemoryError` and `IgniteOutOfMemoryException`;
+* repeated striped-pool starvation warnings;
+* unexpected node restarts.
+
+Ignite reports critical failures to the configured failure handler; depending on the handler, the result can be node invalidation, failover handling, node stop, or JVM termination.
+See link:perf-and-troubleshooting/handling-exceptions[Handling Exceptions] for failure-handler behavior.
+No single local command guarantees the absence of split brain.
+Monitor membership from all expected monitoring points, and use logs and failure handling to detect segmentation and discovery failures.
+
+CDC is a feature-specific check, not part of a universal cluster health definition.
+If CDC is enabled, monitor `ignite-cdc.log`, WAL gaps, and consumption lag.

Review Comment:
   appended



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627471064


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603365504


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3643287974


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,460 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex set of metrics and states that need to be analyzed together. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* link:clustering/baseline-topology[Baseline topology], when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, with no unexpected or repeated node `JOIN`, `LEFT`, or `FAIL` events and no reported node segmentation.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* The number of link:tools/control-script#transaction-management[long-running transactions] and their durations do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+In the examples below, `control.(sh|bat)` is documentation shorthand.
+Run `control.sh` on Unix-like systems or `control.bat` on Windows.
+
+Run the link:tools/control-script#getting-cluster-state[cluster-state command]:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when the cluster has been intentionally placed in read-only mode.
+`INACTIVE` is acceptable only when it matches the current operation, for example, planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run the link:tools/control-script#getting-nodes-registered-in-baseline-topology[baseline-topology command]:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+When a server node joins, leaves, or fails, cluster membership changes and triggers a link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchange (PME)].
+A dynamic cache start or stop can also trigger PME even when no server node joins, leaves, or fails.
+Therefore, a topology version or link:monitoring-metrics/new-metrics#partition-map-exchange[PME metric] change is useful only when interpreted together with maintenance actions and node logs.
+Use these metrics to establish historical distributions; there is no universal duration threshold.
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated `JOIN`, `LEFT`, `FAIL`, or segmentation events and exchange-worker messages.
+Investigate unexpected repeated `JOIN`, `LEFT`, or `FAIL` events, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.

Review Comment:
   ```suggestion
   Run it only on idle cluster i.e. without active data updates.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3239303641


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.
+
+When checking whether a cluster is healthy, start with topology and cluster state. The cluster should be in the expected state, usually ACTIVE, and the number of server and client nodes should be stable. If native persistence is enabled, the baseline should also be in the expected shape: for a stable deployment, the nodes that are expected to be online should appear online both in baseline-related metrics and in the SYS.BASELINE_NODES system view. Frequent unexpected topology changes are not normal and should be treated as a sign of node instability or network problems.

Review Comment:
   usually ACTIVE - need cross link to ACTIVE



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3640895628


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.

Review Comment:
   everything removed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "nizhikov (via GitHub)" <gi...@apache.org>.
nizhikov commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3346882312


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like

Review Comment:
   Let's add examples for `control.sh` commands and JMX beans screenshot that demonstrates your article.
   
   Like the following:
   
   ```
   $ ./control.sh --cluster state
   ... example output for ACTIVE cluster state here ...
   
   $ ./control.sh --baseline
   ... example output for healthy baseline when all nodes are active here ...
   ... and so on ...
   ```
   
   For those checks that must be in graph form let's provide simplest jconsole examples.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3619385833


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the command only clears the `LOST` state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics

Review Comment:
   wont fix, ascii formatting issue



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3586795227


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+./control.sh --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check checkpoint pressure when persistence is enabled

Review Comment:
   The section heading capitalization is inconsistent with other headings in this document. Consider using title case to match the surrounding style.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.

Review Comment:
   The document uses `*` for unordered lists earlier in this file, but this new list uses `-`, which is inconsistent and can render differently depending on AsciiDoc settings. Consider switching these items to `*` to match the surrounding style.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.

Review Comment:
   "As it was said" is unidiomatic English in this context and reads like a mistranslation. Consider rephrasing to a more standard form (for example, "As mentioned above").



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries

Review Comment:
   The section heading capitalization is inconsistent with the other `=== Check ...` headings in this document (most use title case). Consider capitalizing this heading for consistency.



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is expected, but it should converge. Rebalance progress should move toward completion.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+./control.sh --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+=== Check checkpoint pressure when persistence is enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to `DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high `UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint metrics];
+* use faster production storage devices, as discussed in link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate directories on the same disk; see link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page write throttling] configuration;
+* distribute write load or add server nodes when the deployment plan allows it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole MBeans view showing io/datastorage checkpoint attributes and a persistent dataregion bean]
+
+=== Check logs and optional features

Review Comment:
   The section heading capitalization is inconsistent with other headings in this document. Consider using title case to match the surrounding style.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #13130:
URL: https://github.com/apache/ignite/pull/13130#issuecomment-4553566627

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=13130) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=13130&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=13130&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=13130&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=13130&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=13130&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=13130)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3261086342


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.
+
+When checking whether a cluster is healthy, start with topology and cluster state. The cluster should be in the expected state, usually ACTIVE, and the number of server and client nodes should be stable. If native persistence is enabled, the baseline should also be in the expected shape: for a stable deployment, the nodes that are expected to be online should appear online both in baseline-related metrics and in the SYS.BASELINE_NODES system view. Frequent unexpected topology changes are not normal and should be treated as a sign of node instability or network problems.

Review Comment:
   thank you, links added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3239299083


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,20 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== What healthy cluster behavior looks like
+
+A healthy Ignite cluster is not defined by a single latency, CPU, or memory number. In practice, it is a cluster whose topology is stable, whose cluster state and baseline match the intended deployment, whose partitions are not lost or divergent, whose rebalancing and checkpointing complete in bounded time, and whose execution queues and memory pools return to a steady level after short-lived spikes. Ignite exposes these signals through built-in metrics, system views, and the control script rather than through a single aggregate health score.

Review Comment:
   `A healthy Ignite cluster is not defined by a single latency, CPU, or memory number.` - very strange comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "zstan (via GitHub)" <gi...@apache.org>.
zstan commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3631950517


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,458 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no unexpected repeated node `JOIN`, `LEFT`, or `FAIL` events or node segmentation are reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level.
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store a data.
+Client nodes are not a part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PME)].
+A dynamic cache start or stop can also trigger PME even when no server node joins, leaves, or fails.
+Therefore, a topology version or link:monitoring-metrics/new-metrics#partition-map-exchange[PME metric] PME metric change is useful only when interpreted together with maintenance actions and node logs.
+For example, these metrics should be used for historical distributions; there is no universal duration threshold.
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated `JOIN`, `LEFT`, or `FAIL` events, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+Immediately after an intended topology or cache event, `MOVING` and `RENTING` counts can be non-zero.
+After the topology becomes stable, repeat the query and confirm that both counts decrease and eventually disappear.
+A `LOST` count greater than zero is not a normal transient rebalance state.
+
+[source,sql]
+----
+SELECT STATE, COUNT(*) AS PARTITION_COUNT
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+GROUP BY STATE
+ORDER BY STATE;
+----
+
+It is also possible to track some node-local cache-group metrics, such as:
+
+* LocalNodeMovingPartitionsCount;
+* LocalNodeRentingPartitionsCount;
+* LocalNodeRentingEntriesCount.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.

Review Comment:
   ```suggestion
   In stable cluster state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,458 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no unexpected repeated node `JOIN`, `LEFT`, or `FAIL` events or node segmentation are reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level.

Review Comment:
   ```suggestion
   * The number and timeout of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,458 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no unexpected repeated node `JOIN`, `LEFT`, or `FAIL` events or node segmentation are reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level.
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store a data.
+Client nodes are not a part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PME)].
+A dynamic cache start or stop can also trigger PME even when no server node joins, leaves, or fails.
+Therefore, a topology version or link:monitoring-metrics/new-metrics#partition-map-exchange[PME metric] PME metric change is useful only when interpreted together with maintenance actions and node logs.
+For example, these metrics should be used for historical distributions; there is no universal duration threshold.
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated `JOIN`, `LEFT`, or `FAIL` events, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.

Review Comment:
   ```suggestion
   If updates are active, the command can report false positive conflicts because copies are changing while hashes are being calculated.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,458 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.

Review Comment:
   ```suggestion
   Cluster health is a complex set of metrics and states that need to be analyzed together. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
   ```



##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,458 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes.
+* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no unexpected repeated node `JOIN`, `LEFT`, or `FAIL` events or node segmentation are reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level.
+There is no single command or metric that proves cluster health for every deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=OFFLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause partition loss.
+To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store a data.
+Client nodes are not a part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and triggers link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PME)].
+A dynamic cache start or stop can also trigger PME even when no server node joins, leaves, or fails.
+Therefore, a topology version or link:monitoring-metrics/new-metrics#partition-map-exchange[PME metric] PME metric change is useful only when interpreted together with maintenance actions and node logs.
+For example, these metrics should be used for historical distributions; there is no universal duration threshold.
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and exchange-worker messages.
+Investigate unexpected repeated `JOIN`, `LEFT`, or `FAIL` events, node segmentation, network failures, or a PME that does not finish.
+For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be incomplete.
+A successful `idle_verify` result is an important confirmation of consistency, but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+Immediately after an intended topology or cache event, `MOVING` and `RENTING` counts can be non-zero.
+After the topology becomes stable, repeat the query and confirm that both counts decrease and eventually disappear.
+A `LOST` count greater than zero is not a normal transient rebalance state.
+
+[source,sql]
+----
+SELECT STATE, COUNT(*) AS PARTITION_COUNT
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+GROUP BY STATE
+ORDER BY STATE;
+----
+
+It is also possible to track some node-local cache-group metrics, such as:
+
+* LocalNodeMovingPartitionsCount;
+* LocalNodeRentingPartitionsCount;
+* LocalNodeRentingEntriesCount.
+
+Use the link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner; this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID                              | STATE | IS_PRIMARY
+1544803905     | 5            | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST  | true
+----
+
+If the query returns `LOST`, follow the link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery procedure.
+If a failed node returns, its persistent data may become available again, but the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions.
+Before resetting lost partitions, ensure that at least one complete and up-to-date copy of each lost partition is available.
+
+For a persistent cluster, follow the link:configuring-caches/partition-loss-policy#clusters-with-persistence[recovery procedure for clusters with persistence].
+Return all nodes in the baseline topology before resetting lost partitions, or stop the cluster, start all nodes including the failed nodes, and activate the cluster.
+If some nodes cannot be returned, exclude them from the baseline topology only after determining whether their unavailable partition copies contain data that still has to be recovered.
+
+For example, assume that a partition has one primary and one backup copy on nodes A and B. If A leaves, B can continue accepting writes while it remains available.
+If B subsequently fails and only A returns, A can contain an older copy that does not include the writes accepted by B after A left.
+
+After restoring a complete and up-to-date partition copy, or after explicitly accepting that the unavailable updates cannot be recovered, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+`reset_lost_partitions` only clears the `LOST` state. It does not reconstruct updates that are absent from every currently available copy.
+
+[#check-execution-queues]
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks from a shared queue. These queues may grow for a short time under load, but they should not grow continuously. Sustained queue growth means that a node is not keeping up with the workload or that message processing is impaired. The same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between independent stripes: tasks in the same stripe run sequentially, while different stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+   OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As mentioned above, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated `DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name `communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`; the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group `threadPools`, bean name `StripedExecutor`; the attributes include `TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see link:monitoring-metrics/new-metrics-system#jmx[JMX] and link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+[#check-transactions-and-sql-queries]
+=== Check Transactions and SQL Queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+control.(sh|bat) --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+    'tx.OwnerTransactionsNumber',
+    'tx.TransactionsHoldingLockNumber',
+    'tx.LockedKeysNumber',
+    'pme.Duration',
+    'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and failure to return to the usual range after the workload drops.
+For view definitions and metrics, see link:monitoring-metrics/system-views#transactions[TRANSACTIONS], link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES], link:monitoring-metrics/new-metrics#transactions[transaction metrics], and link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchange metrics].
+
+[#check-partition-map-exchange]
+llink:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] synchronizes partition distribution after topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is described in link:key-value-api/transactions#long-running-transactions-termination[Long Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where appropriate; see link:key-value-api/transactions#handling-failed-transactions[Handling Failed Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a justified safety margin, and test application behavior when rollback happens.
+
+[#check-checkpoint-pressure]
+=== Check Checkpoint Pressure When Persistence Is Enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update threads.
+If the checkpoint buffer is exhausted, update processing can stop until the checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+It looks like this in ignite.log

Review Comment:
   ```suggestion
   Corresponding records in <SOME_formatting ? bold ? italic ?>ignite.log looks like as follows:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603371712


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.

Review Comment:
   fixed overwhere, thx



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-28671 Describe healthy cluster behavior in general tips guide [ignite]

Posted by "w3ll1ngt (via GitHub)" <gi...@apache.org>.
w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3603421426


##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant performance benefits.
 
 * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes.
 
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is reachable, but it does not check user partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check, but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for example planned maintenance; an inactive cluster does not serve the data workload.
+The criterion is whether the actual state matches the state that was intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+    ConsistentId=node-1, State=ONLINE, Order=1
+    ConsistentId=node-2, State=ONLINE, Order=2
+    ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+    ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the deployment plan, then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node instability, network issues, or misconfigured discovery. However, a temporary topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other affinity-changing events can change the minor affinity topology version without a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network failures, or a partition map exchange that does not finish.

Review Comment:
   Commands added, could be expanded with logs anyway.. 
   
   Should we expand split-brain situations? It is kinda vague and misty in this doc from my point of view now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org