You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Rushabh Shah (Jira)" <ji...@apache.org> on 2023/07/24 21:15:00 UTC

[jira] [Created] (PHOENIX-7002) Insufficient logging in phoenix client when server throws StaleRegionBoundaryCacheException.

Rushabh Shah created PHOENIX-7002:
-------------------------------------

             Summary: Insufficient logging in phoenix client when server throws StaleRegionBoundaryCacheException.
                 Key: PHOENIX-7002
                 URL: https://issues.apache.org/jira/browse/PHOENIX-7002
             Project: Phoenix
          Issue Type: Bug
            Reporter: Rushabh Shah
            Assignee: Rushabh Shah


Saw an incident in production cluster where phoenix returned result outside of the range provided by the customer. There were hbck repair runs going on while the query was running. During the start of the query, there were region holes in the table (no way to confirm) and while the query was still running we ran hbck repair operation and that caused region overlaps (This is confirmed since overlap continued after the query). 
But the sad part is there were absolutely no exceptions/errors/stack trace on the client or server side.
After the query is run we log the execution time, number of exception encountered as a log line. There we see this query encountered [StaleRegionBoundaryCacheException|https://github.com/apache/phoenix/blob/4.16/phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricType.java#L57].

There is some logic in BaseResultIterators where we adjust the start and end key range for the scan. See [here|https://github.com/apache/phoenix/blob/4.16/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L688-L730]

Without knowing the state of meta known or exception encountered, it is very difficult to debug why this happened.

At the very least, we would want to log all the exceptions on the phoenix client side.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)