You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "matanper (via GitHub)" <gi...@apache.org> on 2026/10/11 05:32:26 UTC

[I] [Bug] Meta-service retries KV_TXN_CONFLICT with 0.5–4 s exponential sleeps, so hot-partition commits stall and fail [doris]

matanper opened a new issue, #68909:
URL: https://github.com/apache/doris/issues/68909

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   ### Version
   
   4.1.4 (cloud mode, arm64). master is the same: `MetaServiceProxy::call_impl` (`cloud/src/meta-service/meta_service.h:1103-1157`) retries `KV_TXN_CONFLICT` with the generic txn-store policy, sleeping `(1 << retry_times) * txn_store_retry_base_intervals_ms + drift` (`:1147`), with `txn_store_retry_times=4` and `txn_store_retry_base_intervals_ms=500` (`cloud/src/common/config.h:274-275`).
   
   ### What's Wrong?
   
   Concurrent commits to the same hot partition conflict on the partition version key in FDB. That's inherent (versions must stay contiguous), and each conflict is resolved quickly by retrying. But the meta-service treats `KV_TXN_CONFLICT` like a slow, retryable storage error: it waits 500 ms, 1 s, 2 s, 4 s (+ up to 500 ms drift), then gives up with `KV_TXN_CONFLICT_RETRY_EXCEEDED_MAX_TIMES`.
   
   So a short-lived conflict costs at least 0.5 s of commit latency, the long sleeps leave the partition idle while committers wait, and with many concurrent committers some loads exhaust the 4 retries and fail.
   
   With many concurrent Stream Loads into one partition on 4 BEs, 18% of commits conflicted (2,248 of 12,528 in 6.5 minutes), 15 commits failed with `KV_TXN_CONFLICT_RETRY_EXCEEDED_MAX_TIMES`, and commit p95 was 1.7 s.
   
   ### What You Expected?
   
   Conflict retries should use a short backoff and more attempts, so a hot partition's commits are serialized quickly instead of sleeping for seconds or failing.
   
   ### How to Reproduce?
   
   Cloud mode. Start many concurrent Stream Loads (dozens in flight) into a single partition of one table, and count `KV_TXN_CONFLICT` warnings ("sleep ... ms before next round") in the meta-service log and `KV_TXN_CONFLICT_RETRY_EXCEEDED_MAX_TIMES` load failures.
   
   ### Anything Else?
   
   What we're doing (unit-tested; production comparison still pending): give `KV_TXN_CONFLICT` its own mutable retry policy, e.g. `txn_conflict_retry_times=10`, `txn_conflict_retry_base_intervals_ms=20`, `txn_conflict_retry_max_intervals_ms=500`, with each wait uniform in `[ceil/2, ceil]` (full jitter, ~3.1 s worst case in total, still under the BE's 10 s RPC timeout). A negative base keeps the current behaviour. Add bvars for conflict retries and exhausted retries.
   
   Retrying the load's label after an exceeded conflict is safe: the FDB transaction didn't commit, so the txn stays PREPARED, is aborted, and the label can be reused.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] Meta-service retries KV_TXN_CONFLICT with 0.5–4 s exponential sleeps, so hot-partition commits stall and fail [doris]

Posted by "Doris-Breakwater (via GitHub)" <gi...@apache.org>.
Doris-Breakwater commented on issue #68909:
URL: https://github.com/apache/doris/issues/68909#issuecomment-6105921626

   Preliminary triage: the source supports a cloud meta-service retry latency/reliability issue, and a separate bounded conflict policy is a reasonable PR direction. The expensive backoff is verified; the exact conflicting keys, the reported performance impact, and the benefit of the proposed defaults still need workload evidence. This issue currently has no labels; cloud/meta-service and ingestion/performance would be appropriate categories.
   
   I checked the public `4.1.4` tag (commit `ad35a140c7fd0b842f18c23300bac581f7d04326`) through the configured local repository and current upstream `master` (`dd456ab7a894f54c6553415223c6d5c9c7d4e035`). This was source inspection and retry-budget calculation; I did not run the workload or the proposed patch's tests.
   
   Verified behavior:
   
   - With both retry switches enabled, `KV_TXN_CONFLICT` uses the same counter and exponential delay as the other retryable store errors. Four retries mean **five handler attempts**, with sleeps of `500+d`, `1000+d`, `2000+d`, and `4000+d` ms. The drift `d` is sampled once per RPC in `[0,500]` and reused. Consequently, exhausting the budget costs **7.5–9.5 seconds of sleep alone**, before handler execution, FDB latency, and scheduling. Exhaustion is converted to `KV_TXN_CONFLICT_RETRY_EXCEEDED_MAX_TIMES`. Both checked revisions retain this behavior. See the [4.1.4 retry loop](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_service.h#L1075-L1129), [defaults](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/common/config.h#L269-L277), and [current master loop](https://github.com/apache/doris/blob/dd456ab7a894f54c6553415223c6d5c9c7d4e035/cloud/src/meta-service/meta_service.h#L1103-L1156).
   - The immediate commit path reads partition versions with a non-snapshot batch read, writes `version+1`, and commits the metadata atomically. Overlapping commits that read the same version can therefore conflict; a proxy retry starts a fresh handler invocation and transaction. This supports the proposed mechanism, but the warning count alone does not establish which key caused each conflict. See [partition version reads](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_service_txn.cpp#L1455-L1507), [version writes](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_service_txn.cpp#L1797-L1819), and [commit error handling](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_service_txn.cpp#L2027-L2040). Long sleeps delay those callers; partition-wide idle periods require a timeline to demonstrate.
   - Exhaustion explains a terminal load error: the direct BE RPC helper retries raw `KV_TXN_CONFLICT`, but returns an error for the exhausted code. The FE commit loop likewise stops on the exhausted code. Stream Load can commit directly from BE or through FE depending on table type and configuration. See [BE retry handling](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_meta_mgr.cpp#L577-L607), [FE commit handling](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L818-L871), and [Stream Load routing](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_stream_load_executor.cpp#L105-L149).
   
   A related public fix, [#64647](https://github.com/apache/doris/pull/64647), removed unnecessary conflicts from a **table-version cache-hint read**. The checked `4.1.4` code already uses that snapshot read. It does not remove the partition-version read/modify/write conflict, and making that partition read a snapshot would require a separate correctness design.
   
   Two qualifications to the proposal:
   
   - With ten retries, base 20 ms, and cap 500 ms, the ceilings are `20,40,80,160,320,500,500,500,500,500`: maximum sleep is **3,120 ms**, across eleven handler attempts. That is promising for short contention, but does not guarantee completion within a 10-second RPC deadline. Include all eleven attempts' work, queueing, and any outer retries. The direct BE path defaults to 10 seconds; FE-to-MS defaults to 30 seconds. Please identify the actual route and effective timeouts. See [BE timeout use](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_meta_mgr.cpp#L562-L568), [BE defaults](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/config.cpp#L30-L37), and [FE default](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L3879-L3881).
   - An FDB conflict means that particular FDB transaction did not commit; it does not by itself prove the Doris transaction is now aborted. Stream Load attempts rollback after a terminal failure, but rollback can fail. `begin_txn` permits label reuse after `ABORTED`, subject to its aborted-transaction limit, while active or committed transactions retain the label. Thus retrying the same load label should follow a **confirmed abort**. Do not extend that conclusion to RPC timeouts, ambiguous commit results, or already committed lazy/2PC transactions. See [label checks](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_service_txn.cpp#L263-L291), [rollback handling](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_stream_load_executor.cpp#L137-L155), and [abort state checks](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-service/meta_serv
 ice_txn.cpp#L3444-L3461).
   
   To complete the diagnosis, please attach:
   
   1. Exact FE/BE/MS build hashes, FDB version, effective retry/timeout settings, table DDL including key model/MoW and bucket count, and lazy-commit/2PC settings.
   2. A runnable load generator with concurrency, payload size, request rate, headers, and label/retry behavior, plus complete JSON responses for representative failed loads.
   3. Correlated MS/BE/FE logs for one recovered conflict and one exhausted load, including transaction ID, RPC method, each attempt/sleep, and abort outcome. For a controlled reproduction, MS's existing `enable_logging_conflict_keys` can collect conflicting ranges; [the implementation](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/cloud/src/meta-store/txn_kv.cpp#L1175-L1235) can help distinguish partition-version contention from other metadata keys. Redact identifiers as needed.
   4. Clarify whether 2,248 counts unique affected commits or retry warnings: one RPC can generate multiple warnings, and this proxy covers multiple methods. Provide per-method attempt counts, unique conflicted/exhausted requests, and the definition of commit p95.
   
   Suggested next step: submit the proposed PR with deterministic tests for success after conflict, exact attempt limits/exhaustion, jitter bounds and cap, negative-base legacy fallback, disabled retry switches, and mixed conflict/store-error sequences. Keep non-conflict error behavior and terminal-code compatibility explicit; validate mutable settings and use overflow-safe capped arithmetic. Separate per-method retry attempts, exhausted requests, and cumulative sleep in the new metrics.
   
   Then compare legacy and proposed policies under identical single-partition load, with low-concurrency and multiple-partition controls. Report throughput, commit p50/p95/p99, failures/timeouts, attempts per request, and MS/FDB CPU and latency. Shorter waits reduce the verified sleep penalty; more attempts may also increase contention and service work, so the proposed defaults need that comparison before their production benefit is established.
   
   Breakwater-GitHub-Analysis-Slot: slot_aadfaed8635f
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org