You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/10/05 15:18:44 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
steveloughran opened a new pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361
This reverts changes in HADOOP-13230 to use S3Guard TTL in choosing when
to issue a HEAD request; fixing tests to compensate.
New operation cost constant, S3GUARD_NONAUTH_FILE_STATUS_PROBE for
use in cost tests.
Change-Id: Ie00279eb94f105d7fc99c7d3342eaf7002fef347
testing in progress: S3 London + S3Guard + markers keep and delete
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-704335369
tests with markers=keep & delete all good other than
* continuous read() under-fulfillment breaking tests which expect their buffers to always be full. Root cause clearly some networking issue, but where?
* ITestS3AInputStreamPerformance stream had to be reopened 3 times, rather than 1. Didn't come back on a rerun.
```
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 386.753 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[ERROR] testDecompressionSequential128K(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance) Time elapsed: 215.757 s <<< FAILURE!
java.lang.AssertionError:
open operations in
null expected:<1> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.assertOpenOperationCount(ITestS3AInputStreamPerformance.java:189)
at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.assertStreamOpenedExactlyOnce(ITestS3AInputStreamPerformance.java:181)
at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testDecompressionSequential128K(ITestS3AInputStreamPerformance.java:324)
```
Assumption: again, network playing up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501703520
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was 1: 0 and th
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was 1: 0 and th
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was originally authMode ? 0 : 1; I'd switched it to 0 : 0 as things changed -but didn't actually deleting, so am reinstating the old code
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2956,55 +2956,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- // If the timestamp of the pm is close to "now", we don't need to
- // bother with a check of S3. that means:
- // one of : status modtime is close to now,
- // or pm.getLastUpdated() == now
-
- // get the time in which a status modtime is considered valid
- // in a non-auth metastore
- long validTime =
- ttlTimeProvider.getNow() - ttlTimeProvider.getMetadataTtl();
- final long msModTime = msStatus.getModificationTime();
-
- if (msModTime < validTime) {
- LOG.debug("Metastore entry of {} is out of date, probing S3", path);
- try {
- S3AFileStatus s3AFileStatus = s3GetFileStatus(path,
- key,
- probes,
- tombstones,
- needEmptyDirectoryFlag);
- // if the new status is more current than that in the metastore,
- // it means S3 has changed and the store needs updating
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if (s3ModTime > msModTime) {
- // there's new data in S3
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- // add to S3Guard
- S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
- } else {
- // the modtime of the data is the same as/older than the s3guard
- // value either an old object has been found, or the existing one
- // was retrieved in both cases -refresh the S3Guard entry so the
- // record's TTL is updated.
- S3Guard.refreshEntry(metadataStore, pm, s3AFileStatus,
- ttlTimeProvider);
- }
- // return the value
- // note that the checks for empty dir status below can be skipped
- // because the call to s3GetFileStatus include the checks there
- return s3AFileStatus;
- } catch (FileNotFoundException fne) {
- // the attempt to refresh the record failed because there was
- // no entry. Either it is a new file not visible, or it
- // has been deleted (and therefore S3Guard is out of sync with S3)
- LOG.warn("Failed to find file {}. Either it is not yet visible, or "
- + "it has been deleted.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path,
+ key,
+ probes,
+ tombstones,
+ needEmptyDirectoryFlag);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
Review comment:
done. logging @ trace so you get the stack when you really, really want it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-703745841
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 27s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 4s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 44s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 6s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 73m 58s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2361 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 83fa2e20c0d4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 43b0c0b0546 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/testReport/ |
| Max. process+thread count | 450 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-703745841
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 27s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 4s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 44s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 6s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 73m 58s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2361 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 83fa2e20c0d4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 43b0c0b0546 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/testReport/ |
| Max. process+thread count | 450 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
bgaborg commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501626796
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2956,55 +2956,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- // If the timestamp of the pm is close to "now", we don't need to
- // bother with a check of S3. that means:
- // one of : status modtime is close to now,
- // or pm.getLastUpdated() == now
-
- // get the time in which a status modtime is considered valid
- // in a non-auth metastore
- long validTime =
- ttlTimeProvider.getNow() - ttlTimeProvider.getMetadataTtl();
- final long msModTime = msStatus.getModificationTime();
-
- if (msModTime < validTime) {
- LOG.debug("Metastore entry of {} is out of date, probing S3", path);
- try {
- S3AFileStatus s3AFileStatus = s3GetFileStatus(path,
- key,
- probes,
- tombstones,
- needEmptyDirectoryFlag);
- // if the new status is more current than that in the metastore,
- // it means S3 has changed and the store needs updating
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if (s3ModTime > msModTime) {
- // there's new data in S3
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- // add to S3Guard
- S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
- } else {
- // the modtime of the data is the same as/older than the s3guard
- // value either an old object has been found, or the existing one
- // was retrieved in both cases -refresh the S3Guard entry so the
- // record's TTL is updated.
- S3Guard.refreshEntry(metadataStore, pm, s3AFileStatus,
- ttlTimeProvider);
- }
- // return the value
- // note that the checks for empty dir status below can be skipped
- // because the call to s3GetFileStatus include the checks there
- return s3AFileStatus;
- } catch (FileNotFoundException fne) {
- // the attempt to refresh the record failed because there was
- // no entry. Either it is a new file not visible, or it
- // has been deleted (and therefore S3Guard is out of sync with S3)
- LOG.warn("Failed to find file {}. Either it is not yet visible, or "
- + "it has been deleted.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path,
+ key,
+ probes,
+ tombstones,
+ needEmptyDirectoryFlag);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
Review comment:
a debug log could be done here about that we received an exception and that's why it's null
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
interesting, why it was 0 : 0? I really haven't noticed it last time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
bgaborg commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705555714
(also, tests run against Ireland was successful)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501703520
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was 1: 0 and th
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
bgaborg commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501626796
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2956,55 +2956,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- // If the timestamp of the pm is close to "now", we don't need to
- // bother with a check of S3. that means:
- // one of : status modtime is close to now,
- // or pm.getLastUpdated() == now
-
- // get the time in which a status modtime is considered valid
- // in a non-auth metastore
- long validTime =
- ttlTimeProvider.getNow() - ttlTimeProvider.getMetadataTtl();
- final long msModTime = msStatus.getModificationTime();
-
- if (msModTime < validTime) {
- LOG.debug("Metastore entry of {} is out of date, probing S3", path);
- try {
- S3AFileStatus s3AFileStatus = s3GetFileStatus(path,
- key,
- probes,
- tombstones,
- needEmptyDirectoryFlag);
- // if the new status is more current than that in the metastore,
- // it means S3 has changed and the store needs updating
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if (s3ModTime > msModTime) {
- // there's new data in S3
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- // add to S3Guard
- S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
- } else {
- // the modtime of the data is the same as/older than the s3guard
- // value either an old object has been found, or the existing one
- // was retrieved in both cases -refresh the S3Guard entry so the
- // record's TTL is updated.
- S3Guard.refreshEntry(metadataStore, pm, s3AFileStatus,
- ttlTimeProvider);
- }
- // return the value
- // note that the checks for empty dir status below can be skipped
- // because the call to s3GetFileStatus include the checks there
- return s3AFileStatus;
- } catch (FileNotFoundException fne) {
- // the attempt to refresh the record failed because there was
- // no entry. Either it is a new file not visible, or it
- // has been deleted (and therefore S3Guard is out of sync with S3)
- LOG.warn("Failed to find file {}. Either it is not yet visible, or "
- + "it has been deleted.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path,
+ key,
+ probes,
+ tombstones,
+ needEmptyDirectoryFlag);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
Review comment:
a debug log could be done here about that we received an exception and that's why it's null
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
interesting, why it was 0 : 0? I really haven't noticed it last time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501703520
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was 1: 0 and th
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-703833708
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 39s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 16s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 8s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 52s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 5s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 32s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 71m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2361 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 990603229bd4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6ece6407e18 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/2/testReport/ |
| Max. process+thread count | 466 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501759422
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
it was originally authMode ? 0 : 1; I'd switched it to 0 : 0 as things changed -but didn't actually deleting, so am reinstating the old code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
bgaborg commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705555714
(also, tests run against Ireland was successful)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501799545
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2956,55 +2956,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- // If the timestamp of the pm is close to "now", we don't need to
- // bother with a check of S3. that means:
- // one of : status modtime is close to now,
- // or pm.getLastUpdated() == now
-
- // get the time in which a status modtime is considered valid
- // in a non-auth metastore
- long validTime =
- ttlTimeProvider.getNow() - ttlTimeProvider.getMetadataTtl();
- final long msModTime = msStatus.getModificationTime();
-
- if (msModTime < validTime) {
- LOG.debug("Metastore entry of {} is out of date, probing S3", path);
- try {
- S3AFileStatus s3AFileStatus = s3GetFileStatus(path,
- key,
- probes,
- tombstones,
- needEmptyDirectoryFlag);
- // if the new status is more current than that in the metastore,
- // it means S3 has changed and the store needs updating
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if (s3ModTime > msModTime) {
- // there's new data in S3
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- // add to S3Guard
- S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
- } else {
- // the modtime of the data is the same as/older than the s3guard
- // value either an old object has been found, or the existing one
- // was retrieved in both cases -refresh the S3Guard entry so the
- // record's TTL is updated.
- S3Guard.refreshEntry(metadataStore, pm, s3AFileStatus,
- ttlTimeProvider);
- }
- // return the value
- // note that the checks for empty dir status below can be skipped
- // because the call to s3GetFileStatus include the checks there
- return s3AFileStatus;
- } catch (FileNotFoundException fne) {
- // the attempt to refresh the record failed because there was
- // no entry. Either it is a new file not visible, or it
- // has been deleted (and therefore S3Guard is out of sync with S3)
- LOG.warn("Failed to find file {}. Either it is not yet visible, or "
- + "it has been deleted.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path,
+ key,
+ probes,
+ tombstones,
+ needEmptyDirectoryFlag);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
Review comment:
done. logging @ trace so you get the stack when you really, really want it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705660591
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 4s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 50s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 4s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 37s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not generate ASF License warnings. |
| | | 77m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2361 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 29813eb6dfb6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 52db86b0bb4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/3/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705619305
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 4s | | https://github.com/apache/hadoop/pull/2361 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/2361 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2361/4/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705636416
patch merged to 3.3.x and trunk, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-703794769
tests run with `-Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Ds3guard -Ddynamo -Dfs.s3a.directory.marker.audit=true -Dscale` found regression in TestS3AEncryptionSSEC which fixed with test changes. Running again with markers = keep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705636416
patch merged to 3.3.x and trunk, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2361: HADOOP-17293. Aways probe S3 when opening/getFileStatus on non-auth paths
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#issuecomment-705619305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org