[jira] [Created] (IGNITE-29102) Add a metric to track B+Tree cursor corruption failures during SQL query execution

["Egor Baranov (Jira)" <ji...@apache.org>]

Egor Baranov created IGNITE-29102:
-------------------------------------

             Summary: Add a metric to track B+Tree cursor corruption failures during SQL query execution
                 Key: IGNITE-29102
                 URL: https://issues.apache.org/jira/browse/IGNITE-29102
             Project: Ignite
          Issue Type: Task
            Reporter: Egor Baranov
            Assignee: Egor Baranov


{*}Problem{*}{*}{*}

 

    When a B+Tree cursor encounters a broken invariant (typically an AssertionError in AbstractForwardCursor.find() or AbstractForwardCursor.fillFromBuffer()), the exception is wrapped

    into a CorruptedTreeException and propagated up the call chain to GridMapQueryExecutor. There, it is caught in a generic catch (Throwable e) block and logged as "Failed to execute 

    local query" at ERROR/WARN level.

 

    There is currently *no metric* that tracks how often such corruption-related query failures occur. Operators can only detect this problem by parsing log files, which is impractical for:

     - Large clusters with high query throughput

     - Intermittent corruption events that are hard to spot in log noise

     - Automated alerting on infrastructure degradation

 

    Example stack trace:

 

     1 java.lang.AssertionError: null

     2     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)

     3     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)

     4     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)

     5     ...

     6     at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)

 

    {*}Solution{*}{*}{*}

 

    Add a dedicated long counter metric in the map query execution path that is incremented whenever a query fails due to a CorruptedTreeException. This follows the same pattern as

    FailureProcessor metrics (IGNITE-29047) and the existing sql.queries.user metrics.

 

    {*}Changes{*}{*}{*}

 

     1. *`GridMapQueryExecutor`* (modules/indexing):

        - Register a new metric registry sql.queries.treeCorruption with a single long metric count.

        - Increment the counter in both error-handling paths (onQueryRequest0 and onNextPageRequest) when the caught throwable's cause chain contains a CorruptedTreeException.

 

     2. *Documentation* (docs/_docs/monitoring-metrics/new-metrics.adoc):

        - Document the new registry and metric.

 

     3. *Test* (TreeCorruptionMetricTest):

        - Verify the metric is registered and initially zero.

        - Simulate B+Tree corruption via BPlusTree.testHndWrapper, execute a query, and assert the counter is incremented.

 

    {*}Resulting Metric{*}{*}{*}

 

 

    ┌────────────────────────────┬───────┬──────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐

    │ *Registry*                   │ *Name*  │ *Type* │ *Description*                                                                                                  │

    ├────────────────────────────┼───────┼──────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤

    │ sql.queries.treeCorruption │ count │ long │ Number of queries that failed due to B+Tree cursor corruption (AssertionError during B+Tree page iteration). │

    └────────────────────────────┴───────┴──────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

 

 

    This metric enables:

     - Automated alerting on tree corruption frequency

     - Correlation of corruption events with specific operations (rebalance, compaction, etc.)

     - Quantification of data reliability without log scraping

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)