You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "BohanZhang0222 (via GitHub)" <gi...@apache.org> on 2026/03/20 10:24:32 UTC
[I] [Bug] Intermittent OperationHandle 404 when fetching results under concurrent load [kyuubi]
BohanZhang0222 opened a new issue, #7363:
URL: https://github.com/apache/kyuubi/issues/7363
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
I am performing a concurrency stress test on Kyuubi and encountered intermittent 404 errors when fetching operation results.
The issue appears to occur when calling get result row set for an operation that has already transitioned to CLOSED_STATE, even though the client did not explicitly close the operation.
**Test Setup**
Goal
Validate the maximum concurrent Operation capacity supported by 4 Kyuubi nodes under the same environment configuration.
Load Testing Method
Using Locust to simulate client behavior with the following workflow:
Create operation
Poll operation status
Fetch operation result
Repeat continuously
Test Parameters
Concurrency: 60
Duration: 10 minutes continuous execution
**Observed Behavior**
Intermittent failures when fetching results:
`Error getting result row set for operation handle <operation_id>
`
Example error:
`org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
`
This corresponds to a 404-like scenario where the operation handle is no longer valid.
### Affects Version(s)
1.9.3
### Kyuubi Server Log Output
```logtalk
From Kyuubi server logs, the lifecycle of a failed operation is as follows:
2026-03-20 17:36:33.524 INFO OperationLog - Creating operation log file ...
2026-03-20 17:42:53.342 INFO ExecuteStatement - PENDING_STATE -> RUNNING_STATE
2026-03-20 17:42:54.572 INFO ExecuteStatement - RUNNING_STATE -> FINISHED_STATE
2026-03-20 17:44:42.814 INFO ExecuteStatement - FINISHED_STATE -> CLOSED_STATE
2026-03-20 17:44:42.814 ERROR ApiUtils - Error getting result row set for operation handle a0f5cbc3-08a4-4a54-8e38-e944174fa940
org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.operation.OperationManager.getOperation(OperationManager.scala:106)
at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$fetchResults$1(BackendServiceMetric.scala:211)
at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:112)
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults(BackendServiceMetric.scala:186)
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults$(BackendServiceMetric.scala:181)
at org.apache.kyuubi.server.KyuubiServer$$anon$1.fetchResults(KyuubiServer.scala:179)
at org.apache.kyuubi.server.api.v1.OperationsResource.getNextRowSet(OperationsResource.scala:186)
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
Not all operations are affected。
<img width="2730" height="414" alt="Image" src="https://github.com/user-attachments/assets/bd168fff-0775-47a4-a015-9c7f3fe72771" />
<img width="2746" height="1176" alt="Image" src="https://github.com/user-attachments/assets/eefe565b-0a44-45ac-8b2d-640396ba0db4" />
<img width="1358" height="664" alt="Image" src="https://github.com/user-attachments/assets/b5284d6b-ee94-40d5-bdc3-4cb699c0d89c" />
<img width="1364" height="522" alt="Image" src="https://github.com/user-attachments/assets/39d0f8fd-08d6-4af5-92db-8439218dfbab" />
<img width="2664" height="512" alt="Image" src="https://github.com/user-attachments/assets/b2029896-1744-472d-8dea-b25721b444c4" />
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Intermittent OperationHandle 404 when fetching results under concurrent load [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7363:
URL: https://github.com/apache/kyuubi/issues/7363#issuecomment-4170009830
> Using Locust to simulate client behavior with the following workflow:
>
> Create operation
>
> Poll operation status
>
> Fetch operation result
Which exactly APIs? General descriptions do not help understand problems
> ```
> 2026-03-20 17:44:42.814 INFO ExecuteStatement - Processing zhou's query[a0f5cbc3-08a4-4a54-8e38-e944174fa940]: FINISHED_STATE -> CLOSED_STATE, time taken: 109.472 seconds
> ```
this is clear that the op was closed explicitly, so the question is where the signal comes from, two suggestions:
- follow the latest `conf/log4j2.xml.template` to enable `restAudit` logs, then check if it is closed by REST API request.
- tune the code of `ExecuteStatement` where prints the above logs, print the callsite
your screenshot indicates you are using batch operation? this does not match what you described above (I assume you are using interactive sessions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Intermittent OperationHandle 404 when fetching results under concurrent load [kyuubi]
Posted by "BohanZhang0222 (via GitHub)" <gi...@apache.org>.
BohanZhang0222 commented on issue #7363:
URL: https://github.com/apache/kyuubi/issues/7363#issuecomment-4198015007
> > Using Locust to simulate client behavior with the following workflow:
> > Create operation
> > Poll operation status
> > Fetch operation result
>
> Which exactly APIs? General descriptions do not help understand problems
>
> your screenshot indicates you are using batch operation? this does not match what you described above (I assume you are using interactive sessions)
1. POST /sessions
2. POST /sessions/${sessionHandle}/operations/statement
3. GET /operations/${operationHandle}/event
4. Multiple GET requests to /operations/{operationHandle}/rowset
I am indeed using operations, not batch.
From the monitoring screenshot, you can see that interactive queries are being submitted, while the number of batch submissions is 0.
<img width="1638" height="960" alt="Image" src="https://github.com/user-attachments/assets/8b43decd-02fb-4447-9aae-27efdeca5f85" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org