You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "macdoor (via GitHub)" <gi...@apache.org> on 2026/04/18 04:13:45 UTC
[PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
macdoor opened a new pull request, #27964:
URL: https://github.com/apache/flink/pull/27964
## What
`NativeS3FileSystem.getFileStatus` always issued `HeadObject` with the key from `NativeS3AccessHelper.extractKey`. For a bucket root path such as `s3://my-bucket`, the key is empty and AWS SDK v2 rejects `HeadObject` with `Key cannot be empty`, breaking catalog creation / existence checks.
## Change
* For an empty key, use `HeadBucket` and return a directory `FileStatus` when the bucket exists.
* In `getDirectoryStatus`, use list prefix `""` for an empty key instead of `"/"` so bucket root listing matches actual object keys.
## Testing
* `mvn spotless:check` (module `flink-s3-fs-native`)
JIRA: https://issues.apache.org/jira/browse/FLINK-39485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4273099054
Hi, @Samrat002 Could you help take a look ? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor closed pull request #27964: [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key
URL: https://github.com/apache/flink/pull/27964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4280357146
Is this a bug for the vanilla s3 integration or just minio support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4278041842
thank you, @macdoor, for confirming. 🙌🏻
I think, before adding this patch, let's first discuss with the community on
1. Shall we continue 100% support for Minio as part of native-s3-fs, given that Minio has moved away from the Apache 2.0 license ?
2. Adding support s3 compitable filesystem in native-s3-fs adds code complexity are we willing to keep such maintainance overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #27964:
URL: https://github.com/apache/flink/pull/27964#discussion_r3104645687
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:
##########
@@ -257,14 +266,37 @@ public FileStatus getFileStatus(Path path) throws IOException {
}
}
+ /**
+ * Resolves the bucket root path (empty object key) to a directory {@link FileStatus} by
+ * verifying the bucket exists via {@link S3Client#headBucket}.
+ */
+ private FileStatus getBucketRootFileStatus(S3Client s3Client, Path path) throws IOException {
+ try {
+ s3Client.headBucket(HeadBucketRequest.builder().bucket(bucketName).build());
+ LOG.debug("HeadBucket successful for s3://{}", bucketName);
+ return S3FileStatus.withDirectory(path);
+ } catch (NoSuchBucketException e) {
+ throw new FileNotFoundException("Bucket not found: " + path);
+ } catch (S3Exception e) {
+ LOG.error(
+ "S3 error checking bucket root s3://{} - StatusCode: {}, ErrorCode: {}, Message: {}",
+ bucketName,
+ e.statusCode(),
+ S3ExceptionUtils.errorCode(e),
+ S3ExceptionUtils.errorMessage(e));
+ throw S3ExceptionUtils.toIOException(
+ String.format("Failed to get file status for s3://%s", bucketName), e);
+ }
+ }
+
/**
* Checks if the given key represents a directory by listing objects with that prefix. Returns a
* directory {@link FileStatus} if objects exist under the prefix, otherwise throws {@link
* FileNotFoundException}.
*/
private FileStatus getDirectoryStatus(S3Client s3Client, String key, Path path)
throws FileNotFoundException {
- final String prefix = key.endsWith("/") ? key : key + "/";
+ final String prefix = key.isEmpty() ? "" : (key.endsWith("/") ? key : key + "/");
Review Comment:
nit: is this change need? if `key.isEmpty()`, getFileStatus returns at L202 and getDirectoryStatus is not called in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4276315371
I have checked and tried with the S3 bucket. I was not able to reproduce the issue.
@macdoor, can you confirm the issue described is only observable with minio, not S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4273418675
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "macdoor (via GitHub)" <gi...@apache.org>.
macdoor commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4277676486
@Samrat002 You are right. I can reproduce this on MinIO with warehouse = s3://<bucket> (empty key), not S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4280423925
This is not a bug for vanilla S3 integration. I have validated it. This issue is not reproducing with vanilla S3.
It only arises and is reproducible with minio.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4272692121
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3496b6b0143eac698b7881775f3af46bd4b9ff52",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3496b6b0143eac698b7881775f3af46bd4b9ff52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3496b6b0143eac698b7881775f3af46bd4b9ff52 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39485][filesystem/s3] Support bucket root using HeadBucket for empty object key [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #27964:
URL: https://github.com/apache/flink/pull/27964#issuecomment-4272905154
@macdoor Can you alter the PR Description to match as per the default guidelines https://github.com/apache/flink/blob/master/.github/PULL_REQUEST_TEMPLATE.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org