[jira] [Created] (HADOOP-18157) S3AFileSystem cannot find a fake dir any more

["JacobZheng (Jira)" <ji...@apache.org>]

JacobZheng created HADOOP-18157:
-----------------------------------

             Summary: S3AFileSystem cannot find a fake dir any more
                 Key: HADOOP-18157
                 URL: https://issues.apache.org/jira/browse/HADOOP-18157
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 3.3.1, 3.2.2
            Reporter: JacobZheng


I am using spark and delta lake for reading and writing datasets. After I upgraded the hadoop version, I found that writing files reported an error, indicating that the folder does not exist.

{code:java}
    java.io.FileNotFoundException: No such file or directory: s3a://table_cache/t5ffe773c987844cbba0aafd/_delta_log
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3356)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4263)
        at org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.listFromInternal(S3SingleDriverLogStore.scala:120)
        at org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.exists(S3SingleDriverLogStore.scala:156)
        at org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.write(S3SingleDriverLogStore.scala:174)
        at org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit(OptimisticTransaction.scala:742)
...
{code}
When this folder has been created, the error is still reported.

The related commit is 49f8ae96, It delete some code that dealing with Fake directory.


{code:java}
// Either a normal file was not found or the probe was skipped.
      // because the key ended in "/" or it was not in the set of probes.
      // Look for the dir marker
      if (probes.contains(StatusProbeEnum.DirMarker)) {
        String newKey = maybeAddTrailingSlash(key);
        try {
          ObjectMetadata meta = getObjectMetadata(newKey);

          if (objectRepresentsDirectory(newKey, meta.getContentLength())) {
            LOG.debug("Found file (with /): fake directory");
            return new S3AFileStatus(Tristate.TRUE, path, username);
          } else {
            LOG.warn("Found file (with /): real file? should not happen: {}",
                key);

            return new S3AFileStatus(meta.getContentLength(),
                    dateToLong(meta.getLastModified()),
                    path,
                    getDefaultBlockSize(path),
                    username,
                    meta.getETag(),
                    meta.getVersionId());
          }
        } catch (AmazonServiceException e) {
          if (e.getStatusCode() != SC_404 || isUnknownBucket(e)) {
            throw translateException("getFileStatus", newKey, e);
          }
        } catch (AmazonClientException e) {
          throw translateException("getFileStatus", newKey, e);
        }
      }
    }
{code}






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org