You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2023/11/10 11:43:00 UTC
[jira] [Commented] (HADOOP-18968) S3A FileSystem does not correctly ls empty directories after HADOOP-17199
[ https://issues.apache.org/jira/browse/HADOOP-18968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784901#comment-17784901 ]
Steve Loughran commented on HADOOP-18968:
-----------------------------------------
hmm. I'd have expected that to show up in testing.
going to need some diagnostics here
# can you download the latest release of the cloudstore diagnostics library: https://github.com/steveloughran/cloudstore
# can you run its "storediag" command with a path in the bucket and the -w -debug options set
# and email to me directly at stevel@apache.org. All login credentials are stripped and you can rename the bucket if you want, I just need to know that configuration and what actually comes back from the store.
{code}
hadoop jar cloudstore-1.0.jar storediag -debug -w s3a://testbucket/temp/tempdir > storediag.log 2>&1
hadoop jar cloudstore-1.0.jar listobjects -debug -w s3a://testbucket/dir > listobjects.log 2>&1
{code}
Also, we did ship 3.3.0 a few years back -is there any reason why you can't upgrade? there are lot of improvements when working with S3 stores -and more coming!
Note: this is going to be very much a back-burner piece of work for me, sorry. You may find yourself writing s3a code an tests yourself...
Looking at the code snippet, yes, i'd expect the list probe to do nothing but a list, but the empty dir should come back as a / from the listing because it is an object underneath.
> S3A FileSystem does not correctly ls empty directories after HADOOP-17199
> -------------------------------------------------------------------------
>
> Key: HADOOP-18968
> URL: https://issues.apache.org/jira/browse/HADOOP-18968
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.2.4
> Reporter: Max Xie
> Priority: Major
> Attachments: image-2023-11-10-16-36-53-337.png
>
>
> After HADOOP-17199, s3a filesystem dose not correctly ls empty directories. For example.
> Before HADOOP-17199
> ```
> hdfs dfs -ls s3a://testbucket/dir/empty
>
> // work right
> ```
>
> After HADOOP-17199
> ```
> hdfs dfs -ls s3a://testbucket/dir/empty
> ls: `s3a://testbucket/dir/empty': No such file or directory
>
> // wrong ouput
> ```
>
> After Check the code, s3a empty folder don't have prefixes Or objects, it whil throw FIleNotFouldException.
> !image-2023-11-10-16-36-53-337.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org