[jira] [Resolved] (HADOOP-16177) regression: ITestS3AMiniYarnCluster failing on branch-3.2

["Steve Loughran (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-16177.
-------------------------------------
    Resolution: Invalid

some logging shows up the problem: I'd switched my (common) core-site to use the staging committer by default

{code}
  <property>
    <name>fs.s3a.committer.name</name>
    <value>directory</value>
    <description>
      Committer to create for output to S3A, one of:
      "file", "directory", "partitioned", "magic".
    </description>
  </property>
{code}

And that was working -but generating unique filenames, as it does by default.

Simple fix: turn off unique filenames.

{code}
  <property>
    <name>fs.s3a.committer.staging.unique-filenames</name>
    <value>false</value>
  </property
{code}

I was worried about some serious problem which s3guard was either not picking up or somehow creating, but no, just configuration changes.


> regression: ITestS3AMiniYarnCluster failing on branch-3.2
> ---------------------------------------------------------
>
>                 Key: HADOOP-16177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16177
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Priority: Major
>
> The test ITestS3AMiniYarnCluster is failing -the output of the trivial MR job isn't being found.
> {code}
> tS3AMiniYarnCluster
> [ERROR] testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster)  Time elapsed: 8.309 s  <<< ERROR!
> java.io.FileNotFoundException: No such file or directory: s3a://hwdev-steve-ireland-new/test/MiniClusterWordCount/output/part-r-00000
> 	at org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.readStringFromFile(ITestS3AMiniYarnCluster.java:144)
> 	at org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.testWithMiniCluster(ITestS3AMiniYarnCluster.java:103)
> {code}
> In trunk, HADOOP-14556 moved this test to the new committers, and the problem doesn't surface. 
> Thought this was related to S3Guard not on, or the fact that a versioned S3 store was in use (S3 london), but it also arises (at least today) on S3 ireland with s3guard and no versioned store. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org