You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sampreeth Reddy B (Jira)" <ji...@apache.org> on 2026/10/12 20:34:00 UTC
[jira] [Updated] (CASSSIDECAR-509) Live-migration download hangs indefinitely when source returns a non-200 for an invalid file path
[ https://issues.apache.org/jira/browse/CASSSIDECAR-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sampreeth Reddy B updated CASSSIDECAR-509:
------------------------------------------
Description:
*Problem*
When a live-migration client issues a {{GET /api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against the sidecar and the server-side path-validation code throws an unexpected exception, the HTTP response is never closed on the server side. The client's underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits for a body it will never receive, blocking the live-migration task's thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}} fires (default: operation-wide timeout, often an hour).
The hang is caused by a missing {{.onFailure(...)}} handler on the {{executorPools.service().executeBlocking(...)}} chain in [{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
*Steps to Reproduce*
# Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any valid Cassandra data directory.
# Issue a {{GET}} for an obviously-invalid path through either the Vert.x client or {{{}curl{}}}:
{{curl -v http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The server correctly returns {{HTTP/1.1 404 Not Found}} .
# Repeat the request via the Vert.x client:
{{sidecarClient.liveMigrationStreamFileAsync(instance, "this_path_does_not_exist.db", "/tmp/dest.db")
.get() this never completes}}
# Observed: the {{CompletableFuture}} returned by {{liveMigrationStreamFileAsync}} never completes (success or failure). A thread dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
{*}RCA{*}:
LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch chained to the {{executeBlocking(...)}} future. Vert.x's documented contract is that unhandled Future failures bubble up through {{UndeliverableException}} and are logged, but the {{RoutingContext}} is not auto-failed by the framework the handler is responsible for calling {{rc.fail(cause)}} or ending the response.
h4. Proposed Fix
chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or {{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using the project's existing error-wrapping convention {{{}HttpExceptions.wrapHttpException(...){}}}):
was:
*Problem*
**
When a live-migration client issues a {{GET /api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against the sidecar and the server-side path-validation code throws an unexpected exception, the HTTP response is never closed on the server side. The client's underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits for a body it will never receive, blocking the live-migration task's thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}} fires (default: operation-wide timeout, often an hour).
The hang is caused by a missing {{.onFailure(...)}} handler on the {{executorPools.service().executeBlocking(...)}} chain in [{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
*Steps to Reproduce*
# Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any valid Cassandra data directory.
# Issue a {{GET}} for an obviously-invalid path through either the Vert.x client or {{{}curl{}}}:
{{curl -v http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The server correctly returns {{HTTP/1.1 404 Not Found}} .
# Repeat the request via the Vert.x client:
{{sidecarClient.liveMigrationStreamFileAsync(instance, "this_path_does_not_exist.db", "/tmp/dest.db")
.get() this never completes}}
# Observed: the {{CompletableFuture}} returned by {{liveMigrationStreamFileAsync}} never completes (success or failure). A thread dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
{*}RCA{*}:
LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch chained to the {{executeBlocking(...)}} future. Vert.x's documented contract is that unhandled Future failures bubble up through {{UndeliverableException}} and are logged, but the {{RoutingContext}} is not auto-failed by the framework the handler is responsible for calling {{rc.fail(cause)}} or ending the response.
h4. Proposed Fix
chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or {{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using the project's existing error-wrapping convention {{{}HttpExceptions.wrapHttpException(...){}}}):
> Live-migration download hangs indefinitely when source returns a non-200 for an invalid file path
> -------------------------------------------------------------------------------------------------
>
> Key: CASSSIDECAR-509
> URL: https://issues.apache.org/jira/browse/CASSSIDECAR-509
> Project: Sidecar for Apache Cassandra
> Issue Type: Bug
> Components: Rest API
> Reporter: Sampreeth Reddy B
> Priority: Major
>
> *Problem*
> When a live-migration client issues a {{GET /api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against the sidecar and the server-side path-validation code throws an unexpected exception, the HTTP response is never closed on the server side. The client's underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits for a body it will never receive, blocking the live-migration task's thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}} fires (default: operation-wide timeout, often an hour).
> The hang is caused by a missing {{.onFailure(...)}} handler on the {{executorPools.service().executeBlocking(...)}} chain in [{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
> *Steps to Reproduce*
> # Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any valid Cassandra data directory.
> # Issue a {{GET}} for an obviously-invalid path through either the Vert.x client or {{{}curl{}}}:
> {{curl -v http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The server correctly returns {{HTTP/1.1 404 Not Found}} .
> # Repeat the request via the Vert.x client:
> {{sidecarClient.liveMigrationStreamFileAsync(instance, "this_path_does_not_exist.db", "/tmp/dest.db")
> .get() this never completes}}
> # Observed: the {{CompletableFuture}} returned by {{liveMigrationStreamFileAsync}} never completes (success or failure). A thread dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
> {*}RCA{*}:
> LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch chained to the {{executeBlocking(...)}} future. Vert.x's documented contract is that unhandled Future failures bubble up through {{UndeliverableException}} and are logged, but the {{RoutingContext}} is not auto-failed by the framework the handler is responsible for calling {{rc.fail(cause)}} or ending the response.
> h4. Proposed Fix
> chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or {{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using the project's existing error-wrapping convention {{{}HttpExceptions.wrapHttpException(...){}}}):
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org