You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "sjgllgh (via GitHub)" <gi...@apache.org> on 2023/12/22 09:50:50 UTC
[PR] Fixed the bug that cannot read the s3 directory tree [linkis]
sjgllgh opened a new pull request, #5055:
URL: https://github.com/apache/linkis/pull/5055
### What is the purpose of the change
When using the S3 file system, it is impossible to correctly traverse directories. This PR fixes this bug and has been verified in our own environment.
### Related issues/PRs
Related issues: #5052
- [x] I have read the [Contributing Guidelines on pull requests](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#pull-requests).
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not, please discuss on the [Linkis mailing list](https://linkis.apache.org/community/how-to-subscribe) first)
- [x] **If this is a code change**: I have written unit tests to fully verify the new behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
Re: [PR] Fixed the bug that cannot read the s3 directory tree [linkis]
Posted by "sjgllgh (via GitHub)" <gi...@apache.org>.
sjgllgh closed pull request #5055: Fixed the bug that cannot read the s3 directory tree
URL: https://github.com/apache/linkis/pull/5055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
Re: [PR] Fixed the bug that cannot read the s3 directory tree [linkis]
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #5055:
URL: https://github.com/apache/linkis/pull/5055#discussion_r1435618756
##########
linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java:
##########
@@ -165,19 +163,29 @@ public List<FsPath> list(FsPath path) throws IOException {
@Override
public FsPathListWithError listPathWithError(FsPath path) throws IOException {
try {
+ List<FsPath> rtn = new ArrayList();
if (!StringUtils.isEmpty(path.getPath())) {
- ListObjectsV2Result listObjectsV2Result = s3Client.listObjectsV2(bucket, path.getPath());
- List<S3ObjectSummary> s3ObjectSummaries = listObjectsV2Result.getObjectSummaries();
- if (s3ObjectSummaries != null) {
- List<FsPath> rtn = new ArrayList();
- String message = "";
- for (S3ObjectSummary summary : s3ObjectSummaries) {
+ ListObjectsV2Request request = new ListObjectsV2Request();
+ request.withBucketName(bucket).withPrefix(buildPrefix(path.getPath())).withDelimiter("/");
+ ListObjectsV2Result result = s3Client.listObjectsV2(request);
+ List<S3ObjectSummary> fileSummaries = result.getObjectSummaries();
+ List<String> dirNames = result.getCommonPrefixes();
+ String message = "";
+ if (fileSummaries != null) {
+ for (S3ObjectSummary summary : fileSummaries) {
if (isDir(summary, path.getPath()) || isInitFile(summary)) continue;
FsPath newPath = new FsPath(buildPath(summary.getKey()));
rtn.add(fillStorageFile(newPath, summary));
}
- return new FsPathListWithError(rtn, message);
}
+ if (CollectionUtils.isNotEmpty(dirNames)) {
Review Comment:
should use isBlank
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
Re: [PR] Fixed the bug that cannot read the s3 directory tree [linkis]
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on PR #5055:
URL: https://github.com/apache/linkis/pull/5055#issuecomment-1874801396
ping ~ @sjgllgh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org