You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/02/26 21:27:34 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1861:
HADOOP-13230. Optionally retain directory markers
steveloughran opened a new pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861
This allows a bucket to have policies of keep, delete or authoritative,
where the latter means "delete in auth dirs"
Retention policy is implemented in its own little class for testing;
does need a callback to probe for a path being auth or not.
Tests? Not yet, no.
Change-Id: Ibcd86048ffaa4c39bba5bd4ea796a24ea8d7f479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-613446195
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 0s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 9s | trunk passed |
| +1 :green_heart: | compile | 22m 5s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 15s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 25m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | trunk passed |
| +0 :ok: | spotbugs | 1m 28s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 11s | trunk passed |
| -0 :warning: | patch | 1m 52s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | the patch passed |
| +1 :green_heart: | compile | 21m 51s | the patch passed |
| -1 :x: | javac | 21m 51s | root generated 12 new + 1870 unchanged - 0 fixed = 1882 total (was 1870) |
| -0 :warning: | checkstyle | 3m 32s | root: The patch generated 42 new + 59 unchanged - 0 fixed = 101 total (was 59) |
| +1 :green_heart: | mvnsite | 2m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 17m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 31s | the patch passed |
| +1 :green_heart: | findbugs | 3m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 21s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 36s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. |
| | | 151m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux f2b1f66e338a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / aeeebc5 |
| Default Java | 1.8.0_242 |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/10/artifact/out/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/10/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/10/testReport/ |
| Max. process+thread count | 1343 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#discussion_r410651379
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DirectoryPolicyImpl.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+
+import java.util.Locale;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_AUTHORITATIVE;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_DELETE;
+import static org.apache.hadoop.fs.s3a.Constants.DIRECTORY_MARKER_POLICY_KEEP;
+
+/**
+ * Implementation of directory policy.
+ */
+public final class DirectoryPolicyImpl
+ implements DirectoryPolicy {
+
+ public static final String UNKNOWN_MARKER_POLICY = "Unknown value of "
+ + DIRECTORY_MARKER_POLICY + ": ";
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ DirectoryPolicyImpl.class);
+
+ private final MarkerPolicy markerPolicy;
+
+ private final Predicate<Path> authoritativenes;
+
+ public DirectoryPolicyImpl(
+ final Configuration conf,
+ final Predicate<Path> authoritativenes) {
+ this.authoritativenes = authoritativenes;
+ String option = conf.getTrimmed(DIRECTORY_MARKER_POLICY,
+ DEFAULT_DIRECTORY_MARKER_POLICY);
+ MarkerPolicy p;
+ switch (option.toLowerCase(Locale.ENGLISH)) {
+
+ case DIRECTORY_MARKER_POLICY_KEEP:
+ p = MarkerPolicy.Keep;
+ LOG.info("Directory markers will be deleted");
+ break;
+ case DIRECTORY_MARKER_POLICY_AUTHORITATIVE:
+ p = MarkerPolicy.Authoritative;
+ LOG.info("Directory markers will be deleted on authoritative"
+ + " paths");
+ break;
+ case DIRECTORY_MARKER_POLICY_DELETE:
+ p = MarkerPolicy.Delete;
+ break;
+ default:
+ throw new IllegalArgumentException(UNKNOWN_MARKER_POLICY + option);
+ }
+ this.markerPolicy = p;
+ }
+
+ @Override
+ public boolean keepDirectoryMarkers(final Path path) {
+ switch (markerPolicy) {
+ case Keep:
+ return true;
+ case Authoritative:
+ return authoritativenes.test(path);
+ case Delete:
+ default: // which cannot happen
Review comment:
throw unchecked exception so new policy (in future) will fail in an obvious way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#discussion_r410640648
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StatusProbeEnum.java
##########
@@ -33,28 +33,25 @@
/** LIST under the path. */
List;
- /** All probes. */
- public static final Set<StatusProbeEnum> ALL = EnumSet.allOf(
- StatusProbeEnum.class);
-
- /** Skip the HEAD and only look for directories. */
- public static final Set<StatusProbeEnum> DIRECTORIES =
- EnumSet.of(DirMarker, List);
-
- /** We only want the HEAD or dir marker. */
- public static final Set<StatusProbeEnum> HEAD_OR_DIR_MARKER =
- EnumSet.of(Head, DirMarker);
+ /** Look for files and directories. */
+ public static final Set<StatusProbeEnum> ALL =
+ EnumSet.of(Head, List);
/** We only want the HEAD. */
public static final Set<StatusProbeEnum> HEAD_ONLY =
EnumSet.of(Head);
- /** We only want the dir marker. */
- public static final Set<StatusProbeEnum> DIR_MARKER_ONLY =
- EnumSet.of(DirMarker);
-
- /** We only want the dir marker. */
+ /** List operation only. */
public static final Set<StatusProbeEnum> LIST_ONLY =
EnumSet.of(List);
+ /** Look for files and directories. */
+ public static final Set<StatusProbeEnum> FILE =
+ HEAD_ONLY;
+
+ /** Skip the HEAD and only look for directories. */
+ public static final Set<StatusProbeEnum> DIRECTORIES =
+ LIST_ONLY;
+
+
Review comment:
Do we still need `DirMarker` in the `StatusProbeEnum`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-592115890
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 5s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 52s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed |
| +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 57s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed |
| +1 :green_heart: | findbugs | 1m 16s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 42s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 71m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d7d4b8e62a5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cd2c6b1 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-610280135
Operation cost tests failiing; I'm trying to (a) cut down things to a minimum and (b) understand why every head/list call takes place, rather than change the assertions to match what's happening. Hadn't realised until now, for example, that rename(file, dest) has 3 HEAD calls, for example
```
[INFO]
[ERROR] Failures:
[ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:598->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(srcFilePath, destFilePath) [object_delete_requests starting=5 current=8 diff=3, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=31 current=46 diff=15, object_list_requests starting=20 current=23 diff=3, object_metadata_requests starting=18 current=21 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1>
[ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:621->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(s3a://stevel-london/test/testFakeDirectoryDeletion%5Braw-keep-markers%5D/1/2/3/4/5/6/source2.txt, s3a://stevel-london/test/testFakeDirectoryDeletion[raw-keep-markers]/1/2/3/4/5/6/source.txt) [object_delete_requests starting=1 current=2 diff=1, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=0 current=0 diff=0, object_list_requests starting=25 current=28 diff=3, object_metadata_requests starting=25 current=28 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1>
[INFO]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-598392416
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 20s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 24s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 29s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 64m 57s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28818861734a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0a9b3c9 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-604011125
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 4s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-aws: The patch generated 16 new + 32 unchanged - 0 fixed = 48 total (was 32) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 10s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 31s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 65m 21s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux a4b7a71bc7d1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e3fbdcb |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-614321549
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 31s | trunk passed |
| +1 :green_heart: | compile | 17m 50s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 26s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 9s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | the patch passed |
| +1 :green_heart: | compile | 17m 8s | the patch passed |
| +1 :green_heart: | javac | 17m 8s | the patch passed |
| -0 :warning: | checkstyle | 2m 50s | root: The patch generated 39 new + 60 unchanged - 0 fixed = 99 total (was 60) |
| +1 :green_heart: | mvnsite | 2m 8s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 33s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 28s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 16s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. |
| | | 128m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux fbc48ef4a1d4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc5c1da |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/11/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/11/testReport/ |
| Max. process+thread count | 2187 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/11/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-604011125
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 4s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-aws: The patch generated 16 new + 32 unchanged - 0 fixed = 48 total (was 32) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 10s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 31s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 65m 21s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux a4b7a71bc7d1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e3fbdcb |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#discussion_r410650058
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -1411,10 +1440,13 @@ public boolean rename(Path src, Path dst) throws IOException {
LOG.debug("rename: destination path {} not found", dst);
// Parent must exist
Path parent = dst.getParent();
- if (!pathToKey(parent).isEmpty()) {
+ if (!pathToKey(parent).isEmpty()
+ && !parent.equals(src.getParent()) ) {
try {
- S3AFileStatus dstParentStatus = innerGetFileStatus(dst.getParent(),
- false, StatusProbeEnum.ALL);
+ // only look against S3 for directories; saves
+ // a HEAD request on all normal operations.
+ S3AFileStatus dstParentStatus = innerGetFileStatus(parent,
+ false, StatusProbeEnum.DIRECTORIES);
Review comment:
We do not need to fallback to `StatusProbeEnum.FILE` because we know this `innerGetFileStatus` should not throw `FileNotFoundException` - at least dst should show in the listing. Is this correct?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-610044172
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 12 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 31s | trunk passed |
| +1 :green_heart: | compile | 17m 46s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 10s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | the patch passed |
| +1 :green_heart: | compile | 17m 9s | the patch passed |
| -1 :x: | javac | 17m 9s | root generated 12 new + 1870 unchanged - 0 fixed = 1882 total (was 1870) |
| -0 :warning: | checkstyle | 2m 49s | root: The patch generated 35 new + 47 unchanged - 0 fixed = 82 total (was 47) |
| +1 :green_heart: | mvnsite | 2m 10s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 25s | the patch passed |
| +1 :green_heart: | findbugs | 3m 28s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 13s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 127m 48s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.TestFixKerberosTicketOrder |
| | hadoop.security.TestRaceWhenRelogin |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1f9ef040f2e2 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ab7495d |
| Default Java | 1.8.0_242 |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/artifact/out/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/testReport/ |
| Max. process+thread count | 1928 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-608048937
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 36s | trunk passed |
| +1 :green_heart: | compile | 18m 6s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 19s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 28s | trunk passed |
| +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 24s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | the patch passed |
| +1 :green_heart: | compile | 25m 7s | the patch passed |
| +1 :green_heart: | javac | 25m 7s | the patch passed |
| -0 :warning: | checkstyle | 4m 6s | root: The patch generated 40 new + 42 unchanged - 0 fixed = 82 total (was 42) |
| +1 :green_heart: | mvnsite | 3m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 22m 31s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 52s | hadoop-tools_hadoop-aws generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | findbugs | 5m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 13m 25s | hadoop-common in the patch passed. |
| -1 :x: | unit | 2m 25s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | The patch does not generate ASF License warnings. |
| | | 155m 17s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 258a331795e4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 25361b0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/diff-checkstyle-root.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/testReport/ |
| Max. process+thread count | 2831 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-591677163
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 43s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 23s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
| | | 59m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0493a589848 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 2096304 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/testReport/ |
| Max. process+thread count | 453 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-598392416
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 20s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 24s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 29s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 64m 57s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28818861734a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0a9b3c9 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-614824943
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 25m 1s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 49s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 13s | trunk passed |
| +1 :green_heart: | compile | 19m 14s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 36s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 21s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 38s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 15s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | the patch passed |
| +1 :green_heart: | compile | 16m 22s | the patch passed |
| +1 :green_heart: | javac | 16m 22s | the patch passed |
| -0 :warning: | checkstyle | 2m 38s | root: The patch generated 33 new + 60 unchanged - 0 fixed = 93 total (was 60) |
| +1 :green_heart: | mvnsite | 2m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 10s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed |
| +1 :green_heart: | findbugs | 3m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 9s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 150m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux f66cbca97f02 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 8505840 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/12/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/12/testReport/ |
| Max. process+thread count | 2672 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/12/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#discussion_r410645571
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -1265,18 +1282,30 @@ public WriteOperationHelper getWriteOperationHelper() {
* is not a directory.
*/
@Override
- public FSDataOutputStream createNonRecursive(Path path,
+ public FSDataOutputStream createNonRecursive(Path p,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress) throws IOException {
entryPoint(INVOCATION_CREATE_NON_RECURSIVE);
+ final Path path = makeQualified(p);
Path parent = path.getParent();
- if (parent != null) {
- // expect this to raise an exception if there is no parent
- if (!getFileStatus(parent).isDirectory()) {
+ // expect this to raise an exception if there is no parent dir
+ if (parent != null && !parent.isRoot()) {
+ S3AFileStatus status;
+ try {
+ // optimize for the directory existing: Call list first
+ status = innerGetFileStatus(parent, false,
+ StatusProbeEnum.DIRECTORIES);
+ } catch (FileNotFoundException e) {
+ // no dir, fall back to looking for a file
+ // (failure condition if true)
+ status = innerGetFileStatus(parent, false,
+ StatusProbeEnum.HEAD_ONLY);
Review comment:
I know they are the same, but is `StatusProbeEnum.FILE` a bit better here since the status is for heading a file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-611249845
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 12s | trunk passed |
| +1 :green_heart: | compile | 23m 0s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 26m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed |
| +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 8s | trunk passed |
| -0 :warning: | patch | 1m 50s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | the patch passed |
| +1 :green_heart: | compile | 22m 0s | the patch passed |
| -1 :x: | javac | 22m 0s | root generated 12 new + 1870 unchanged - 0 fixed = 1882 total (was 1870) |
| -0 :warning: | checkstyle | 3m 11s | root: The patch generated 39 new + 47 unchanged - 0 fixed = 86 total (was 47) |
| +1 :green_heart: | mvnsite | 2m 34s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 19m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 53s | the patch passed |
| +1 :green_heart: | findbugs | 4m 15s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 54s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. |
| | | 159m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1517b0131fc9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e53d472 |
| Default Java | 1.8.0_242 |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/9/artifact/out/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/9/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/9/testReport/ |
| Max. process+thread count | 2229 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-592115890
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 5s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 52s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed |
| +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 57s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed |
| +1 :green_heart: | findbugs | 1m 16s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 42s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 71m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d7d4b8e62a5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cd2c6b1 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1861:
HADOOP-13230. Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#discussion_r410635387
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -445,6 +456,12 @@ public void initialize(URI name, Configuration originalConf)
DEFAULT_S3GUARD_DISABLED_WARN_LEVEL);
S3Guard.logS3GuardDisabled(LOG, warnLevel, bucket);
}
+ // directory policy, which will look at authoritative paths
+ // if needed
+ directoryPolicy = new DirectoryPolicyImpl(conf,
+ this::allowAuthoritative);
+ LOG.debug("Directory marker retention policy is {}",
Review comment:
This can be `info` level since this is new and client may want to see the message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-591677163
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 43s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 23s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
| | | 59m 16s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1861 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0493a589848 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 2096304 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/testReport/ |
| Max. process+thread count | 453 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-600775261
HADOOP-13230. directory markers
the LIST call asks for two objects when needEmptyDirectoryFlag = true, so can distinguish
dir marker exists
from
dir marker exists + children
moved much of the prefix/object analysis into S3ListResult where I intend to add some unit tests for the result parsing.
Change the enum for all innerGetFileStatus calls from ALL to FILES_AND_DIRECTORIES, as we no longer need to do *any* HEAD / request on a marker; the list finds it after all.
There may be more risk of delayed consistency in listings
Tests: one mocking test fails (as usual); also failures in ITestS3AFileOperationCost, ITestS3GuardOutOfBandOperations, ITestRestrictedReadAccess
```
[ERROR] Failures:
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:161->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=1 current=2 diff=1: object_list_requests expected:<0> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnEmptyDir:159->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=4 current=5 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingFile:180->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=0 current=1 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnMissingSubPath:192->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir:215->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=5 current=6 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCreateCost:511->verifyOperationCount:140->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_metadata_requests starting=2 current=3 diff=1: object_metadata_requests expected:<2> but was:<1>
[ERROR] ITestS3AFileOperationCost.testDirProbes:474 [LIST output is not considered empty]
Expecting:
<S3AFileStatus{path=s3a://stevel-london/fork-0001/test/testEmptyDirHeadProbe; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=TRUE eTag=null versionId=null>
to match 'is empty' predicate.
[ERROR] ITestRestrictedReadAccess.testNoReadAccess:304->checkDeleteOperations:637->accessDenied:680 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : true
[ERROR] ITestRestrictedReadAccess.testNoReadAccess:298->checkBasicFileOperations:413->accessDeniedIf:697 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : [Lorg.apache.hadoop.fs.FileStatus;@1a902257
[ERROR] Errors:
[ERROR] ITestS3GuardOutOfBandOperations.testListingDelete:988->expectExceptionWhenReadingOpenFileAPI:1055 ยป Execution
```
the access ones are failing because LIST is working whereas a HEAD would fail if the caller doesn't have read access to the object. OOB ops may be from me setting up a new bucket. Cost ones: well, our costs have come down. literally
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-615315234
failure in restricted permission test as ls empty dir now succeeds in list (no HEAD, see)
```
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 31.512 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess[ERROR] testNoReadAccess[raw](org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess) Time elapsed: 26.071 s <<< FAILURE! java.lang.AssertionError: Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : [Lorg.apache.hadoop.fs.s3a.S3AFileStatus;@53d4d5a
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.accessDeniedIf(ITestRestrictedReadAccess.java:697)
at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.checkBasicFileOperations(ITestRestrictedReadAccess.java:413)
at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.testNoReadAccess(ITestRestrictedReadAccess.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230.
Optionally retain directory markers
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
URL: https://github.com/apache/hadoop/pull/1861#issuecomment-609773143
```
java.lang.NullPointerException
at org.apache.hadoop.fs.s3a.S3ListResult.getCommonPrefixes(S3ListResult.java:100)
at org.apache.hadoop.fs.s3a.S3ListResult.hasPrefixesOrObjects(S3ListResult.java:143)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3066)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2935)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2795)
at org.apache.hadoop.fs.s3a.TestS3AGetFileStatus.testFakeDirectory(TestS3AGetFileStatus.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org