[I] [Bug] PD gRPC workers stay held while the leader has no quorum: KV writes, id reads and KV reads wait on the shared hg-grpc pool [hugegraph]

["bitflicker64 (via GitHub)" <gi...@apache.org>]

bitflicker64 opened a new issue, #3389:
URL: https://github.com/apache/hugegraph/issues/3389

   ### Bug Type (问题类型)
   
   logic (逻辑设计问题)
   
   ### Before submit
   
   - [x] I have confirmed and searched that there are no similar problems in the historical issue and documents
   
   ### Environment (环境信息)
   
   - Server Version: PD on `master` at `ebb8f4e17` (1.8.0-SNAPSHOT), and the KV read change of #3383 at its head `db0d771f7`
   - Backend: PD's raft metadata store (jraft-core 1.3.9)
   - OS: any; found by reading the code on 2026-10-22, see "How this was found" below
   - Data Size: N/A
   
   ### Expected & Actual behavior (期望与实际表现)
   
   While a PD leader cannot reach a raft quorum, a request that needs the quorum should fail or be redirected without holding a gRPC worker for long, so the RPCs that need no quorum keep being served.
   
   What happens instead: three request paths wait on a worker of the one shared gRPC executor (`hg-grpc`) until raft answers, and the executor has a fixed number of workers in front of an unbounded queue.
   
   1. **The executor.** `GRpcServerConfig.configure` (lines 37 to 41) installs `HgExecutorUtil.createExecutor("hg-grpc", core, max, queue)` for the whole PD gRPC server. `PDConfig.ThreadPoolGrpc` defaults to `thread.pool.grpc.core=600`, `max=1000` and `queue=Integer.MAX_VALUE`, which becomes a `LinkedBlockingQueue` of that capacity (`HgExecutorUtil.java` lines 101 to 107). A `ThreadPoolExecutor` adds threads above the core size only when the queue is full, so in practice the server has 600 workers, and every request past them waits in the queue.
   2. **KV writes wait without a timeout.** Every write of `RaftKVStore` blocks on `applyOperation(...).get()` with no deadline (`RaftKVStore.java` lines 76, 99, 109, 119, 128, 138, 157 and 210). Without a quorum the entry does not commit, and the future completes when jraft fails the pending task at step-down (`NodeImpl.stepDown` calls `ballotBox.clearPendingTasks()`, which runs each closure with `EPERM "Leader stepped down"`).
   3. **Id reads wait for a ReadIndex.** `IdMetaStore.inLeaderTerm` (line 108) calls `waitReadIndexInTerm`, added by #3368, which waits up to `raft.rpc-timeout` (10 s by default; `RaftEngine.waitReadIndex`, lines 663 and 690 to 713).
   4. **KV `get` and `scanPrefix` (#3383) wait for the same ReadIndex**, also up to `raft.rpc-timeout` (`KvServiceGrpcImpl.java` lines 166 and 270 at `db0d771f7`).
   
   How long a worker is held: PD sets no election timeout, so jraft's default of 1000 ms applies, with a leader lease of 90% of it. A leader that loses its quorum steps down about 1 s later. After that, new requests see `isLeader() == false` and are redirected without waiting. A write that was waiting is released at the step-down. A ReadIndex that was already waiting (paths 3 and 4) holds its worker until it fails or reaches its 10 s deadline.
   
   So a burst of about 600 requests that arrive in the second before the step-down can occupy every worker. RPCs that need no quorum then queue behind them, for about a second for writes and up to 10 s for reads. Nothing is lost, since the queue is unbounded, but the delay is real. This was raised in the review of #3383 for the KV reads, and paths 2 and 3 show the same holds on master.
   
   One possible direction, for the reads: let the handler return at once and answer from a small executor once the ReadIndex completes. `RaftEngine` already builds a `CompletableFuture` for the wait, so it could return that future instead of blocking on it. The write path would need the same treatment, or a deadline. Other directions are a bound on the number of concurrent waits, or a separate executor for the RPCs that need raft.
   
   ### How this was found
   
   By reading the code at `ebb8f4e17` and the jraft-core 1.3.9 sources, on 2026-10-22. **No quorum-loss test was run**, so the counts and times above come from the configuration defaults and the code paths, not from a measurement.
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   N/A
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@hugegraph.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org