You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "macdoor (via GitHub)" <gi...@apache.org> on 2026/04/18 04:17:02 UTC
[PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: abort unfinished GetObject on seek/reopen [flink]
macdoor opened a new pull request, #27965:
URL: https://github.com/apache/flink/pull/27965
## Purpose
Fix `ConnectionClosedException: Premature end of Content-Length delimited message body` when reading large Parquet files from S3-compatible storage (e.g. MinIO) via `flink-s3-fs-native`, especially with `seek()` / ranged GETs.
## Root cause
Closing a partially consumed `GetObject` response via `BufferedInputStream.close()` lets Apache HttpClient drain the remaining body for connection reuse. If the connection ends early, draining throws. AWS SDK v2 recommends `ResponseInputStream.abort()` when the rest of the body will not be read.
## Change
- On reopen and on `close()` when `position < contentLength`, call `abort()` on the `ResponseInputStream` instead of relying on drain-on-close.
## Testing
- Local `mvn` compile was not run in this environment (Flink 2.4-SNAPSHOT reactor deps not resolvable against the configured Maven mirror). Relying on Azure Pipelines for `flink-s3-fs-native`.
JIRA: https://issues.apache.org/jira/browse/FLINK-39484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: abort unfinished GetObject on seek/reopen [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27965:
URL: https://github.com/apache/flink/pull/27965#issuecomment-4272703329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91883a7057728d58c67448a78b025c97e3445b73",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91883a7057728d58c67448a78b025c97e3445b73",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91883a7057728d58c67448a78b025c97e3445b73 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #27965:
URL: https://github.com/apache/flink/pull/27965#issuecomment-4276053731
@macdoor spotless is failing for the build, also there are some changes in the license header which I believe in unintended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27965:
URL: https://github.com/apache/flink/pull/27965#issuecomment-4278005257
Thank you @macdoor for raising the patch.
i have a fundamental query associate with this change.
is the problem true for AWS S3?
As per my analysis , Premature end of Content-Length delimited message body can only occur when the server closes the TCP connection before delivering all Content-Length bytes. This is an HTTP/1.1 protocol violation.
AWS S3 does not do this. S3 always delivers exactly the number of bytes promised in Content-Length. The drain loop in Apache `HttpClient's ContentLengthInputStream.close()` will always succeed against real S3. It may be slow (reading GBs to discard), but it will never hit a premature EOF.
The only valid concern for real S3 is performance. I see probable improvement is draining large amounts of unread data on close() just to reuse a connection.
The SDK JavaDoc says it plainly:
> "If it is not desired to read remaining data from the stream, you can explicitly abort the connection via abort() instead. This will close the underlying connection and require establishing a new HTTP connection on subsequent requests which may outweigh the cost of reading the additional data."
reference : https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/ResponseInputStream.java#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor closed pull request #27965: [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse
URL: https://github.com/apache/flink/pull/27965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor commented on PR #27965:
URL: https://github.com/apache/flink/pull/27965#issuecomment-4276100418
@spuru9 Thanks — Spotless is fixed (spotless:apply + spotless:check), and the license header in NativeS3InputStream.java is aligned with the standard ASF template.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39484][filesystem/s3] NativeS3InputStream: graceful abort on early close; close() first for connection reuse [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor commented on PR #27965:
URL: https://github.com/apache/flink/pull/27965#issuecomment-4276101345
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org