You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/10/28 15:48:35 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
steveloughran opened a new pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597
Contains MAPREDUCE-7367. Parallelize file moves in FileOutputCommitter v1 job commit
Contributed by Rajesh Balamohan
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958077592
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 26s | | trunk passed |
| +1 :green_heart: | compile | 21m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 48s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 16s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 16s | | the patch passed |
| +1 :green_heart: | compile | 19m 58s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 13 new + 99 unchanged - 0 fixed = 112 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 4s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 5m 58s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 134m 34s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not generate ASF License warnings. |
| | | 369m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bdd11688ebd6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 72bc6fd23f78bbe2f33491ecd075a314d390adf8 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/testReport/ |
| Max. process+thread count | 1325 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958077592
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 26s | | trunk passed |
| +1 :green_heart: | compile | 21m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 48s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 16s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 16s | | the patch passed |
| +1 :green_heart: | compile | 19m 58s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 13 new + 99 unchanged - 0 fixed = 112 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 4s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 5m 58s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 134m 34s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 33s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not generate ASF License warnings. |
| | | 369m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bdd11688ebd6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 72bc6fd23f78bbe2f33491ecd075a314d390adf8 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/testReport/ |
| Max. process+thread count | 1325 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r739119293
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/StoreOperationsThroughFileSystem.java
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.CommonPathCapabilities;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.Trash;
+import org.apache.hadoop.fs.impl.ResilientCommitByRename;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.FileOrDirEntry;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.TaskManifest;
+import org.apache.hadoop.util.JsonSerialization;
+
+import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY;
+
+/**
+ * Implement task and job operations through the filesystem API.
+ */
+public class StoreOperationsThroughFileSystem extends StoreOperations {
Review comment:
This class is not supposed to be a part of this commit right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-954156037
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 31s | | trunk passed |
| +1 :green_heart: | compile | 2m 30s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 9s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 5s | | the patch passed |
| +1 :green_heart: | compile | 2m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 2m 23s | | the patch passed |
| +1 :green_heart: | compile | 2m 2s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 2m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 52s | [/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/1/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 15 new + 38 unchanged - 0 fixed = 53 total (was 38) |
| +1 :green_heart: | mvnsite | 1m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 6m 26s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 142m 30s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 248m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 5e7994d48f2e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a04e45b5f9ccd1ba3ccc6e4c2e8519bf4f1d4a6f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/1/testReport/ |
| Max. process+thread count | 1345 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740782689
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/ResilientCommitByRename.java
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Options;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.statistics.IOStatisticsSource;
+
+/**
+ * This is something internal to make our rename-based job committers
+ * more resilient to failures.
+ * If you are in the hive team: do not use this as it lacks
+ * spec, tests, stability, etc. if we find you using it we will change
+ * the signature just to stop your code compiling.
+ * View this as a proof of concept of the functionality we'd want from a
+ * "modern" rename call, but not the API (which would be builder based,
+ * return a future, etc).
+ */
+@InterfaceAudience.LimitedPrivate({"Filesystems", "hadoop-mapreduce-client-core"})
+@InterfaceStability.Unstable
+public interface ResilientCommitByRename {
+
+ /**
+ * Path capability.
+ * FS Instances which support the operation MUST return
+ * true and implement the method; FileSystem instances which do not
+ * MUST return false.
+ * There's a risk wrapper filesystems may pass the probe
+ * through.
+ * Clients MUST check for both the interface and this
+ * cqpability.
+ */
+ String RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY =
+ "org.apache.hadoop.fs.impl.resilient.commit.by.rename";
+
+ /**
+ * Rename source file to dest path *Exactly*; no subdirectory games here.
+ * if the op does not raise an exception,then
+ * the data at dest is the data which was at source.
+ *
+ * Requirements
+ *
+ * <pre>
+ * exists(FS, source) else raise FileNotFoundException
+ * source != dest else raise PathIOException
+ * not exists(FS, dest)
+ * isDir(FS, dest.getParent)
+ * </pre>
+ * <ol>
+ * <li>supported in this instance else raise PathIOException</li>
+ * <li>source != dest else raise PathIOException</li>
+ * <li>source must exist else raise FileNotFoundException</li>
+ * <li>source must exist and be a file</li>
+ * <li>dest must not exist; </li>
+ * <li>dest.getParent() must be a dir</li>
+ * <li>if sourceEtag is non-empty, it MAY be used to qualify/validate the rename.</li>
+ * </ol>
+ *
+ * The outcome of the operation is undefined if source is not a file, dest exists,
+ * dest.getParent() doesn't exist/is a file.
+ * That is: implementations SHOULD assume that the code calling this method has
+ * set up the destination directory tree and is only invoking this call on a file.
+ * Accordingly: <i>implementations MAY skip validation checks</i>
+ *
+ * If sourceStatus is not null, its contents MAY be used to qualify the rename.
+ * <ol>
+ * <li>Values extracted from sourceStatus SHALL take priority over
+ * sourceEtag/sourceLastModified parameter.</li>
+ * <li>sourceStatus.getPath().getName() MUST equal source.getName()</li>
+ * <li>If store has a subclass of FileStatus and it is sourceStatus is of this type,
+ * custom information MAY be used to qualify/validate the request.
+ * This MAY include etag or S3 version ID extraction,</li>
+ * </ol>
+ *
+ * Filesystems MAY support this call on an instance-by-instance basis, depending on
+ * the nature of the remote store.
+ * If not available the implementation MUST {@code ResilientCommitByRenameUnsupported}.
+ * Callers SHOULD use a check of
+ * {@code hasPathCapability(source, RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY}
+ * before trying to use this call.
+ *
+ * PostConditions on a successful operation:
+ * <pre>
+ * FS' where:
+ * not exists(FS', source)
+ * and exists(FS', dest)
+ * and data(FS', dest) == data (FS, source)
+ * </pre>
+ * This is exactly the same outcome as `FileSystem.rename()` when the same preconditions
+ * are met. This API call simply restricts the operation to file rename with strict
+ * conditions, (no need to be 'clever' about dest path calculation) and the ability
+ * to pass in etags, modtimes and file status values.
+ *
+ * @param source path to source file
+ * @param dest destination of rename.
+ * @param sourceEtag etag of source file. may be null or empty
+ * @param sourceStatus nullable FileStatus of source.
+ * @param options rename flags
+ * @throws FileNotFoundException source file not found
+ * @throws ResilientCommitByRenameUnsupported not available on this store.
+ * @throws PathIOException failure, including source and dest being the same path
+ * @throws IOException any other exception
+ */
+ default CommitByRenameOutcome commitSingleFileByRename(
+ Path source,
+ Path dest,
+ @Nullable String sourceEtag,
+ @Nullable FileStatus sourceStatus,
+ ResilientCommitByRename.CommitFlqgs... options)
+ throws FileNotFoundException,
+ ResilientCommitByRenameUnsupported,
+ PathIOException,
+ IOException {
+ throw new ResilientCommitByRenameUnsupported(source.toString());
+ }
+
+ /**
+ * The outcome. This is always a success, but it
+ * may include some information about what happened.
+ */
+ class CommitByRenameOutcome {
+
+ /* Throttling encountered and recovered from. */
+ private boolean throttlingEncountered;
+
+ /* The new commit operation has been rejected; falling back. */
+ private boolean commitRejected;
+
+ /* Classic rename was used. */
+ private boolean classicRenameUsed;
+
+ public CommitByRenameOutcome() {
+ }
+
+ public CommitByRenameOutcome(
+ final boolean throttlingEncountered,
+ final boolean commitRejected,
+ final boolean classicRenameUsed) {
+ this.throttlingEncountered = throttlingEncountered;
+ this.commitRejected = commitRejected;
+ this.classicRenameUsed = classicRenameUsed;
+ }
+
+ public boolean isThrottlingEncountered() {
+ return throttlingEncountered;
+ }
+
+ public void setThrottlingEncountered(final boolean throttlingEncountered) {
+ this.throttlingEncountered = throttlingEncountered;
+ }
+
+ public boolean isCommitRejected() {
+ return commitRejected;
+ }
+
+ public void setCommitRejected(final boolean commitRejected) {
+ this.commitRejected = commitRejected;
+ }
+
+ public boolean isClassicRenameUsed() {
+ return classicRenameUsed;
+ }
+
+ public void setClassicRenameUsed(final boolean classicRenameUsed) {
+ this.classicRenameUsed = classicRenameUsed;
+ }
+ }
+
+ final class ResilientCommitByRenameUnsupported extends PathIOException {
+ public ResilientCommitByRenameUnsupported(final String path) {
+ super(path, "ResilientCommit operations not supported");
+ }
+ }
+
+ /**
+ * Enum of options.
+ */
+ enum CommitFlqgs {
Review comment:
nit : typo.. Flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958658416
> > This mechanism becomes very FileSystem specific. Implemented by Azure right now.
>
> I agree, which is why the API is restricted for its uses to mr-client-core only. as abfs is the only one which needs it for correctness under load, And I'm not worried about that specifity. Can I point to how much of the hadoop fs api are hdfs-only -and they are public.
>
> > Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
>
> Please don't try and use this particular interface in Hive.
>
Was referring to any potential usage - including Hive.
> > Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces.
>
> We are going straight from a listing (1 request/500 entries) to that rename. doing a HEAD first cuts the throughtput in half. so no.
>
In the scenario where this is encountered. Would not be the default behaviour, and limits the change to Abfs. Could also have the less consistent version which is not etag based, and responds only on failures. Again - limited to Abfs.
> > Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
>
> Possible but convoluted.
>
Agree. Quite convoluted. Tossing in potential options - to avoid a new public API.
> > Essentially - don't force usage of the new interface to get the benefits.
>
> I understand the interests of the hive team, but this fix is not the place to do a better API.
>
> Briefly cacheing the source FS entries is something to consider though. Not this week.
>
> What I could do with is some help getting #2735 in, then we can start on a public rename() builder API which will take a file status, as openFile does.
>
This particular change would be FSImpl agnostic, and potentially remove the need for the new interface here?
> > Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
>
> There is an option to say "i know it is not there"; this skips the check. the committer passes this option down because it issues a delete call first.
>
EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
> FWIW the manifest committer will make that pre-rename commit optional, saving that IO request. I am curious as to how well that will work I went executed on well formed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740782689
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/ResilientCommitByRename.java
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Options;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.statistics.IOStatisticsSource;
+
+/**
+ * This is something internal to make our rename-based job committers
+ * more resilient to failures.
+ * If you are in the hive team: do not use this as it lacks
+ * spec, tests, stability, etc. if we find you using it we will change
+ * the signature just to stop your code compiling.
+ * View this as a proof of concept of the functionality we'd want from a
+ * "modern" rename call, but not the API (which would be builder based,
+ * return a future, etc).
+ */
+@InterfaceAudience.LimitedPrivate({"Filesystems", "hadoop-mapreduce-client-core"})
+@InterfaceStability.Unstable
+public interface ResilientCommitByRename {
+
+ /**
+ * Path capability.
+ * FS Instances which support the operation MUST return
+ * true and implement the method; FileSystem instances which do not
+ * MUST return false.
+ * There's a risk wrapper filesystems may pass the probe
+ * through.
+ * Clients MUST check for both the interface and this
+ * cqpability.
+ */
+ String RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY =
+ "org.apache.hadoop.fs.impl.resilient.commit.by.rename";
+
+ /**
+ * Rename source file to dest path *Exactly*; no subdirectory games here.
+ * if the op does not raise an exception,then
+ * the data at dest is the data which was at source.
+ *
+ * Requirements
+ *
+ * <pre>
+ * exists(FS, source) else raise FileNotFoundException
+ * source != dest else raise PathIOException
+ * not exists(FS, dest)
+ * isDir(FS, dest.getParent)
+ * </pre>
+ * <ol>
+ * <li>supported in this instance else raise PathIOException</li>
+ * <li>source != dest else raise PathIOException</li>
+ * <li>source must exist else raise FileNotFoundException</li>
+ * <li>source must exist and be a file</li>
+ * <li>dest must not exist; </li>
+ * <li>dest.getParent() must be a dir</li>
+ * <li>if sourceEtag is non-empty, it MAY be used to qualify/validate the rename.</li>
+ * </ol>
+ *
+ * The outcome of the operation is undefined if source is not a file, dest exists,
+ * dest.getParent() doesn't exist/is a file.
+ * That is: implementations SHOULD assume that the code calling this method has
+ * set up the destination directory tree and is only invoking this call on a file.
+ * Accordingly: <i>implementations MAY skip validation checks</i>
+ *
+ * If sourceStatus is not null, its contents MAY be used to qualify the rename.
+ * <ol>
+ * <li>Values extracted from sourceStatus SHALL take priority over
+ * sourceEtag/sourceLastModified parameter.</li>
+ * <li>sourceStatus.getPath().getName() MUST equal source.getName()</li>
+ * <li>If store has a subclass of FileStatus and it is sourceStatus is of this type,
+ * custom information MAY be used to qualify/validate the request.
+ * This MAY include etag or S3 version ID extraction,</li>
+ * </ol>
+ *
+ * Filesystems MAY support this call on an instance-by-instance basis, depending on
+ * the nature of the remote store.
+ * If not available the implementation MUST {@code ResilientCommitByRenameUnsupported}.
+ * Callers SHOULD use a check of
+ * {@code hasPathCapability(source, RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY}
+ * before trying to use this call.
+ *
+ * PostConditions on a successful operation:
+ * <pre>
+ * FS' where:
+ * not exists(FS', source)
+ * and exists(FS', dest)
+ * and data(FS', dest) == data (FS, source)
+ * </pre>
+ * This is exactly the same outcome as `FileSystem.rename()` when the same preconditions
+ * are met. This API call simply restricts the operation to file rename with strict
+ * conditions, (no need to be 'clever' about dest path calculation) and the ability
+ * to pass in etags, modtimes and file status values.
+ *
+ * @param source path to source file
+ * @param dest destination of rename.
+ * @param sourceEtag etag of source file. may be null or empty
+ * @param sourceStatus nullable FileStatus of source.
+ * @param options rename flags
+ * @throws FileNotFoundException source file not found
+ * @throws ResilientCommitByRenameUnsupported not available on this store.
+ * @throws PathIOException failure, including source and dest being the same path
+ * @throws IOException any other exception
+ */
+ default CommitByRenameOutcome commitSingleFileByRename(
+ Path source,
+ Path dest,
+ @Nullable String sourceEtag,
+ @Nullable FileStatus sourceStatus,
+ ResilientCommitByRename.CommitFlqgs... options)
+ throws FileNotFoundException,
+ ResilientCommitByRenameUnsupported,
+ PathIOException,
+ IOException {
+ throw new ResilientCommitByRenameUnsupported(source.toString());
+ }
+
+ /**
+ * The outcome. This is always a success, but it
+ * may include some information about what happened.
+ */
+ class CommitByRenameOutcome {
+
+ /* Throttling encountered and recovered from. */
+ private boolean throttlingEncountered;
+
+ /* The new commit operation has been rejected; falling back. */
+ private boolean commitRejected;
+
+ /* Classic rename was used. */
+ private boolean classicRenameUsed;
+
+ public CommitByRenameOutcome() {
+ }
+
+ public CommitByRenameOutcome(
+ final boolean throttlingEncountered,
+ final boolean commitRejected,
+ final boolean classicRenameUsed) {
+ this.throttlingEncountered = throttlingEncountered;
+ this.commitRejected = commitRejected;
+ this.classicRenameUsed = classicRenameUsed;
+ }
+
+ public boolean isThrottlingEncountered() {
+ return throttlingEncountered;
+ }
+
+ public void setThrottlingEncountered(final boolean throttlingEncountered) {
+ this.throttlingEncountered = throttlingEncountered;
+ }
+
+ public boolean isCommitRejected() {
+ return commitRejected;
+ }
+
+ public void setCommitRejected(final boolean commitRejected) {
+ this.commitRejected = commitRejected;
+ }
+
+ public boolean isClassicRenameUsed() {
+ return classicRenameUsed;
+ }
+
+ public void setClassicRenameUsed(final boolean classicRenameUsed) {
+ this.classicRenameUsed = classicRenameUsed;
+ }
+ }
+
+ final class ResilientCommitByRenameUnsupported extends PathIOException {
+ public ResilientCommitByRenameUnsupported(final String path) {
+ super(path, "ResilientCommit operations not supported");
+ }
+ }
+
+ /**
+ * Enum of options.
+ */
+ enum CommitFlqgs {
Review comment:
nit : typo.. Flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-959193924
> EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
yes, and both hive and hbase are known to do that, often ending forcing hdfs only apis to get pulled up without concern for the other stores (see git history of FileSystem there). a builder rename would be a big job with
1. rename/3 public with tests and performant object store impls
2. default builder to delegate to that with same semantics
3. add options for etag etc on a store by store or cross store basis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958011751
see #3611 . i think this pr had got too complex and so have gone back to what I had originally done in the manifest committer. filesystem provides etags, application code recovers.
This is something which we can expose to the hive team, as all that will be added is a single stable extension to filestatus to get the etag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958658416
> > This mechanism becomes very FileSystem specific. Implemented by Azure right now.
>
> I agree, which is why the API is restricted for its uses to mr-client-core only. as abfs is the only one which needs it for correctness under load, And I'm not worried about that specifity. Can I point to how much of the hadoop fs api are hdfs-only -and they are public.
>
> > Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
>
> Please don't try and use this particular interface in Hive.
>
Was referring to any potential usage - including Hive.
> > Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces.
>
> We are going straight from a listing (1 request/500 entries) to that rename. doing a HEAD first cuts the throughtput in half. so no.
>
In the scenario where this is encountered. Would not be the default behaviour, and limits the change to Abfs. Could also have the less consistent version which is not etag based, and responds only on failures. Again - limited to Abfs.
> > Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
>
> Possible but convoluted.
>
Agree. Quite convoluted. Tossing in potential options - to avoid a new public API.
> > Essentially - don't force usage of the new interface to get the benefits.
>
> I understand the interests of the hive team, but this fix is not the place to do a better API.
>
> Briefly cacheing the source FS entries is something to consider though. Not this week.
>
> What I could do with is some help getting #2735 in, then we can start on a public rename() builder API which will take a file status, as openFile does.
>
This particular change would be FSImpl agnostic, and potentially remove the need for the new interface here?
> > Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
>
> There is an option to say "i know it is not there"; this skips the check. the committer passes this option down because it issues a delete call first.
>
EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
> FWIW the manifest committer will make that pre-rename commit optional, saving that IO request. I am curious as to how well that will work I went executed on well formed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 18s | | trunk passed |
| +1 :green_heart: | compile | 21m 38s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 20s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 52s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 21m 2s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 2s | | the patch passed |
| +1 :green_heart: | compile | 19m 7s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 37s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 17 new + 99 unchanged - 0 fixed = 116 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 49s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 21s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 5m 57s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 135m 38s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 370m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 80640d7e2e23 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 712c7cbc0dad45bf31378ec38935dc6141faff2e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/testReport/ |
| Max. process+thread count | 1573 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 18s | | trunk passed |
| +1 :green_heart: | compile | 21m 38s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 20s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 52s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 21m 2s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 2s | | the patch passed |
| +1 :green_heart: | compile | 19m 7s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 37s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 17 new + 99 unchanged - 0 fixed = 116 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 49s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 21s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 5m 57s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 135m 38s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 370m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 80640d7e2e23 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 712c7cbc0dad45bf31378ec38935dc6141faff2e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/testReport/ |
| Max. process+thread count | 1573 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958658416
> > This mechanism becomes very FileSystem specific. Implemented by Azure right now.
>
> I agree, which is why the API is restricted for its uses to mr-client-core only. as abfs is the only one which needs it for correctness under load, And I'm not worried about that specifity. Can I point to how much of the hadoop fs api are hdfs-only -and they are public.
>
> > Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
>
> Please don't try and use this particular interface in Hive.
>
Was referring to any potential usage - including Hive.
> > Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces.
>
> We are going straight from a listing (1 request/500 entries) to that rename. doing a HEAD first cuts the throughtput in half. so no.
>
In the scenario where this is encountered. Would not be the default behaviour, and limits the change to Abfs. Could also have the less consistent version which is not etag based, and responds only on failures. Again - limited to Abfs.
> > Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
>
> Possible but convoluted.
>
Agree. Quite convoluted. Tossing in potential options - to avoid a new public API.
> > Essentially - don't force usage of the new interface to get the benefits.
>
> I understand the interests of the hive team, but this fix is not the place to do a better API.
>
> Briefly cacheing the source FS entries is something to consider though. Not this week.
>
> What I could do with is some help getting #2735 in, then we can start on a public rename() builder API which will take a file status, as openFile does.
>
This particular change would be FSImpl agnostic, and potentially remove the need for the new interface here?
> > Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
>
> There is an option to say "i know it is not there"; this skips the check. the committer passes this option down because it issues a delete call first.
>
EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
> FWIW the manifest committer will make that pre-rename commit optional, saving that IO request. I am curious as to how well that will work I went executed on well formed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-957310383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740782689
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/ResilientCommitByRename.java
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Options;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.statistics.IOStatisticsSource;
+
+/**
+ * This is something internal to make our rename-based job committers
+ * more resilient to failures.
+ * If you are in the hive team: do not use this as it lacks
+ * spec, tests, stability, etc. if we find you using it we will change
+ * the signature just to stop your code compiling.
+ * View this as a proof of concept of the functionality we'd want from a
+ * "modern" rename call, but not the API (which would be builder based,
+ * return a future, etc).
+ */
+@InterfaceAudience.LimitedPrivate({"Filesystems", "hadoop-mapreduce-client-core"})
+@InterfaceStability.Unstable
+public interface ResilientCommitByRename {
+
+ /**
+ * Path capability.
+ * FS Instances which support the operation MUST return
+ * true and implement the method; FileSystem instances which do not
+ * MUST return false.
+ * There's a risk wrapper filesystems may pass the probe
+ * through.
+ * Clients MUST check for both the interface and this
+ * cqpability.
+ */
+ String RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY =
+ "org.apache.hadoop.fs.impl.resilient.commit.by.rename";
+
+ /**
+ * Rename source file to dest path *Exactly*; no subdirectory games here.
+ * if the op does not raise an exception,then
+ * the data at dest is the data which was at source.
+ *
+ * Requirements
+ *
+ * <pre>
+ * exists(FS, source) else raise FileNotFoundException
+ * source != dest else raise PathIOException
+ * not exists(FS, dest)
+ * isDir(FS, dest.getParent)
+ * </pre>
+ * <ol>
+ * <li>supported in this instance else raise PathIOException</li>
+ * <li>source != dest else raise PathIOException</li>
+ * <li>source must exist else raise FileNotFoundException</li>
+ * <li>source must exist and be a file</li>
+ * <li>dest must not exist; </li>
+ * <li>dest.getParent() must be a dir</li>
+ * <li>if sourceEtag is non-empty, it MAY be used to qualify/validate the rename.</li>
+ * </ol>
+ *
+ * The outcome of the operation is undefined if source is not a file, dest exists,
+ * dest.getParent() doesn't exist/is a file.
+ * That is: implementations SHOULD assume that the code calling this method has
+ * set up the destination directory tree and is only invoking this call on a file.
+ * Accordingly: <i>implementations MAY skip validation checks</i>
+ *
+ * If sourceStatus is not null, its contents MAY be used to qualify the rename.
+ * <ol>
+ * <li>Values extracted from sourceStatus SHALL take priority over
+ * sourceEtag/sourceLastModified parameter.</li>
+ * <li>sourceStatus.getPath().getName() MUST equal source.getName()</li>
+ * <li>If store has a subclass of FileStatus and it is sourceStatus is of this type,
+ * custom information MAY be used to qualify/validate the request.
+ * This MAY include etag or S3 version ID extraction,</li>
+ * </ol>
+ *
+ * Filesystems MAY support this call on an instance-by-instance basis, depending on
+ * the nature of the remote store.
+ * If not available the implementation MUST {@code ResilientCommitByRenameUnsupported}.
+ * Callers SHOULD use a check of
+ * {@code hasPathCapability(source, RESILIENT_COMMIT_BY_RENAME_PATH_CAPABILITY}
+ * before trying to use this call.
+ *
+ * PostConditions on a successful operation:
+ * <pre>
+ * FS' where:
+ * not exists(FS', source)
+ * and exists(FS', dest)
+ * and data(FS', dest) == data (FS, source)
+ * </pre>
+ * This is exactly the same outcome as `FileSystem.rename()` when the same preconditions
+ * are met. This API call simply restricts the operation to file rename with strict
+ * conditions, (no need to be 'clever' about dest path calculation) and the ability
+ * to pass in etags, modtimes and file status values.
+ *
+ * @param source path to source file
+ * @param dest destination of rename.
+ * @param sourceEtag etag of source file. may be null or empty
+ * @param sourceStatus nullable FileStatus of source.
+ * @param options rename flags
+ * @throws FileNotFoundException source file not found
+ * @throws ResilientCommitByRenameUnsupported not available on this store.
+ * @throws PathIOException failure, including source and dest being the same path
+ * @throws IOException any other exception
+ */
+ default CommitByRenameOutcome commitSingleFileByRename(
+ Path source,
+ Path dest,
+ @Nullable String sourceEtag,
+ @Nullable FileStatus sourceStatus,
+ ResilientCommitByRename.CommitFlqgs... options)
+ throws FileNotFoundException,
+ ResilientCommitByRenameUnsupported,
+ PathIOException,
+ IOException {
+ throw new ResilientCommitByRenameUnsupported(source.toString());
+ }
+
+ /**
+ * The outcome. This is always a success, but it
+ * may include some information about what happened.
+ */
+ class CommitByRenameOutcome {
+
+ /* Throttling encountered and recovered from. */
+ private boolean throttlingEncountered;
+
+ /* The new commit operation has been rejected; falling back. */
+ private boolean commitRejected;
+
+ /* Classic rename was used. */
+ private boolean classicRenameUsed;
+
+ public CommitByRenameOutcome() {
+ }
+
+ public CommitByRenameOutcome(
+ final boolean throttlingEncountered,
+ final boolean commitRejected,
+ final boolean classicRenameUsed) {
+ this.throttlingEncountered = throttlingEncountered;
+ this.commitRejected = commitRejected;
+ this.classicRenameUsed = classicRenameUsed;
+ }
+
+ public boolean isThrottlingEncountered() {
+ return throttlingEncountered;
+ }
+
+ public void setThrottlingEncountered(final boolean throttlingEncountered) {
+ this.throttlingEncountered = throttlingEncountered;
+ }
+
+ public boolean isCommitRejected() {
+ return commitRejected;
+ }
+
+ public void setCommitRejected(final boolean commitRejected) {
+ this.commitRejected = commitRejected;
+ }
+
+ public boolean isClassicRenameUsed() {
+ return classicRenameUsed;
+ }
+
+ public void setClassicRenameUsed(final boolean classicRenameUsed) {
+ this.classicRenameUsed = classicRenameUsed;
+ }
+ }
+
+ final class ResilientCommitByRenameUnsupported extends PathIOException {
+ public ResilientCommitByRenameUnsupported(final String path) {
+ super(path, "ResilientCommit operations not supported");
+ }
+ }
+
+ /**
+ * Enum of options.
+ */
+ enum CommitFlqgs {
Review comment:
nit : typo.. Flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-955112045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 36s | | trunk passed |
| +1 :green_heart: | compile | 33m 57s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 28m 42s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 26m 9s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 26m 9s | | the patch passed |
| +1 :green_heart: | compile | 22m 14s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 22m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 59s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 19 new + 99 unchanged - 0 fixed = 118 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 57s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 6m 19s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 143m 58s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 434m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6ba9a731eb8d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f4950b3d0416b764433d18d24e8b19b09b663cc4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-954717074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 10s | | trunk passed |
| +1 :green_heart: | compile | 22m 34s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 47s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 22s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 24s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvninstall | 0m 26s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | compile | 10m 14s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 10m 14s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 9m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 9m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 15s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 16 new + 40 unchanged - 0 fixed = 56 total (was 40) |
| -1 :x: | mvnsite | 0m 31s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvnsite | 0m 33s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 0m 28s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 31s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | shadedclient | 9m 1s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 58s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 29s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | unit | 0m 31s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 188m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 924b1293ecbe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ee6854c3b32b3e0cc5e96c9d10f4d91d3b9f790c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-959193924
> EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
yes, and both hive and hbase are known to do that, often ending forcing hdfs only apis to get pulled up without concern for the other stores (see git history of FileSystem there). a builder rename would be a big job with
1. rename/3 public with tests and performant object store impls
2. default builder to delegate to that with same semantics
3. add options for etag etc on a store by store or cross store basis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-954717074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 10s | | trunk passed |
| +1 :green_heart: | compile | 22m 34s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 47s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 22s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 24s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvninstall | 0m 26s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | compile | 10m 14s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 10m 14s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 9m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 9m 10s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 15s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 16 new + 40 unchanged - 0 fixed = 56 total (was 40) |
| -1 :x: | mvnsite | 0m 31s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvnsite | 0m 33s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 0m 28s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 31s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | shadedclient | 9m 1s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 58s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 29s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | unit | 0m 31s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 188m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 924b1293ecbe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ee6854c3b32b3e0cc5e96c9d10f4d91d3b9f790c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740107063
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/StoreOperationsThroughFileSystem.java
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.CommonPathCapabilities;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.Trash;
+import org.apache.hadoop.fs.impl.ResilientCommitByRename;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.AbstractManifestData;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.FileOrDirEntry;
+import org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.TaskManifest;
+import org.apache.hadoop.util.JsonSerialization;
+
+import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY;
+
+/**
+ * Implement task and job operations through the filesystem API.
+ */
+public class StoreOperationsThroughFileSystem extends StoreOperations {
Review comment:
I screwed up that commit by cherrypicking from the manifest committer branch (which I needed to) -but missed that git adds that as a file. Removed them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-958658416
> > This mechanism becomes very FileSystem specific. Implemented by Azure right now.
>
> I agree, which is why the API is restricted for its uses to mr-client-core only. as abfs is the only one which needs it for correctness under load, And I'm not worried about that specifity. Can I point to how much of the hadoop fs api are hdfs-only -and they are public.
>
> > Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
>
> Please don't try and use this particular interface in Hive.
>
Was referring to any potential usage - including Hive.
> > Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces.
>
> We are going straight from a listing (1 request/500 entries) to that rename. doing a HEAD first cuts the throughtput in half. so no.
>
In the scenario where this is encountered. Would not be the default behaviour, and limits the change to Abfs. Could also have the less consistent version which is not etag based, and responds only on failures. Again - limited to Abfs.
> > Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
>
> Possible but convoluted.
>
Agree. Quite convoluted. Tossing in potential options - to avoid a new public API.
> > Essentially - don't force usage of the new interface to get the benefits.
>
> I understand the interests of the hive team, but this fix is not the place to do a better API.
>
> Briefly cacheing the source FS entries is something to consider though. Not this week.
>
> What I could do with is some help getting #2735 in, then we can start on a public rename() builder API which will take a file status, as openFile does.
>
This particular change would be FSImpl agnostic, and potentially remove the need for the new interface here?
> > Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
>
> There is an option to say "i know it is not there"; this skips the check. the committer passes this option down because it issues a delete call first.
>
EOD - this ends up being a new API (almost on the FileSystem), which is used by the committer first; then someone discovers it and decides to make use of it.
> FWIW the manifest committer will make that pre-rename commit optional, saving that IO request. I am curious as to how well that will work I went executed on well formed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-959193924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-957554120
@sidseth I discussed the idea of caching status etags with @mukund-thakur
considering that you'd need to remove entries from the cache on: create, rename, & delete, it would actually be a copy of `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore`, as #3534 is removing all of s3guard, I'll consider that a WONTFIX. sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 18s | | trunk passed |
| +1 :green_heart: | compile | 21m 38s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 20s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 52s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 21m 2s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 2s | | the patch passed |
| +1 :green_heart: | compile | 19m 7s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 37s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 17 new + 99 unchanged - 0 fixed = 116 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 49s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 21s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 5m 57s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 135m 38s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 370m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 80640d7e2e23 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 712c7cbc0dad45bf31378ec38935dc6141faff2e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/testReport/ |
| Max. process+thread count | 1573 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-954195372
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 16s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 37s | | trunk passed |
| +1 :green_heart: | compile | 28m 39s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 22m 24s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 56s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 47s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 26s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvninstall | 0m 27s | [/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | mvninstall | 0m 20s | [/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | compile | 13m 9s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 13m 9s | [/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt) | root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 11m 17s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 11m 17s | [/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt) | root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 47s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 16 new + 40 unchanged - 0 fixed = 56 total (was 40) |
| -1 :x: | mvnsite | 0m 31s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | mvnsite | 0m 33s | [/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | mvnsite | 0m 27s | [/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 0m 28s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 32s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | spotbugs | 0m 25s | [/patch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | shadedclient | 11m 31s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 59s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 28s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | unit | 0m 29s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-client-jobclient in the patch failed. |
| -1 :x: | unit | 0m 25s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 213m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux b5dc798b0c12 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08e20d03565504d248f642dc38d42329e7a98508 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/testReport/ |
| Max. process+thread count | 2911 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-957310383
> This mechanism becomes very FileSystem specific. Implemented by Azure right now.
I agree, which is why the API is restricted for its uses to mr-client-core only.
as abfs is the only one which needs it for correctness under load,
And I'm not worried about that specifity.
Can I point to how much of the hadoop fs api are hdfs-only -and they are public.
> Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
Please don't try and use this particular interface in Hive.
> Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces.
We are going straight from a listing (1 request/500 entries) to that rename. doing a HEAD first cuts the throughtput in half. so no.
> Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
Possible but convoluted.
> Essentially - don't force usage of the new interface to get the benefits.
I understand the interests of the hive team, but this fix is not the place to do a better API.
Briefly cacheing the source FS entries is something to consider though. Not this week.
What I could do with is some help getting #2735 in, then we can start on a public rename() builder API which will take a file status, as openFile does.
> Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
There is an option to say "i know it is not there"; this skips the check. the committer passes this option down because it issues a delete call first.
FWIW the manifest committer will make that pre-rename commit optional, saving that IO request. I am curious as to how well that will work I went executed on well formed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740720279
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -497,6 +535,302 @@ private void mergePaths(FileSystem fs, final FileStatus from,
}
}
+ @VisibleForTesting
+ public int getMoveThreads() {
+ return moveThreads;
+ }
+
+ public boolean isParallelMoveEnabled() {
+ // Only available for algo v1
+ return (moveThreads > 1 && algorithmVersion == 1);
+ }
+
+ void validateParallelMove() throws IOException {
+ if (!isParallelMoveEnabled()) {
+ throw new IOException("Parallel file move is not enabled. "
+ + "moveThreads=" + moveThreads
+ + ", algo=" + algorithmVersion);
+ }
+ }
+
+ void validateThreadPool(ExecutorService pool, BlockingQueue<Future<Void>> futures)
+ throws IOException {
+ boolean threadPoolEnabled = isParallelMoveEnabled();
+ if (!threadPoolEnabled || pool == null || futures == null) {
+ String errorMsg = "Thread pool is not configured correctly. "
+ + "threadPoolEnabled: " + threadPoolEnabled
+ + ", pool: " + pool
+ + ", futures: " + futures;
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+
+ /**
+ * Get executor service for moving files for v1 algorithm.
+ * @return executor service
+ * @throws IOException on error
+ */
+ private ExecutorService createExecutorService() throws IOException {
+ // intentional validation
+ validateParallelMove();
+
+ ExecutorService pool = new ThreadPoolExecutor(moveThreads, moveThreads,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(),
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setNameFormat("FileCommitter-v1-move-thread-%d")
+ .build(),
+ new ThreadPoolExecutor.CallerRunsPolicy()
+ );
+ LOG.info("Size of move thread pool: {}, pool: {}", moveThreads, pool);
+ return pool;
+ }
+
+ private void parallelCommitJobInternal(JobContext context) throws IOException {
+ // validate to be on safer side.
+ validateParallelMove();
+
+ if (hasOutputPath()) {
+ Path finalOutput = getOutputPath();
+ FileSystem fs = finalOutput.getFileSystem(context.getConfiguration());
+ // created resilient commit helper bonded to the destination FS/path
+ resilientCommitHelper = new ResilientCommitByRenameHelper(fs);
Review comment:
This mechanism becomes very FileSystem specific. Implemented by Azure right now.
Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces. Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
Essentially - don't force usage of the new interface to get the benefits.
Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-955112045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 36s | | trunk passed |
| +1 :green_heart: | compile | 33m 57s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 28m 42s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 26m 9s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 26m 9s | | the patch passed |
| +1 :green_heart: | compile | 22m 14s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 22m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 59s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 19 new + 99 unchanged - 0 fixed = 118 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 57s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 6m 19s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 143m 58s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 434m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6ba9a731eb8d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f4950b3d0416b764433d18d24e8b19b09b663cc4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740720279
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -497,6 +535,302 @@ private void mergePaths(FileSystem fs, final FileStatus from,
}
}
+ @VisibleForTesting
+ public int getMoveThreads() {
+ return moveThreads;
+ }
+
+ public boolean isParallelMoveEnabled() {
+ // Only available for algo v1
+ return (moveThreads > 1 && algorithmVersion == 1);
+ }
+
+ void validateParallelMove() throws IOException {
+ if (!isParallelMoveEnabled()) {
+ throw new IOException("Parallel file move is not enabled. "
+ + "moveThreads=" + moveThreads
+ + ", algo=" + algorithmVersion);
+ }
+ }
+
+ void validateThreadPool(ExecutorService pool, BlockingQueue<Future<Void>> futures)
+ throws IOException {
+ boolean threadPoolEnabled = isParallelMoveEnabled();
+ if (!threadPoolEnabled || pool == null || futures == null) {
+ String errorMsg = "Thread pool is not configured correctly. "
+ + "threadPoolEnabled: " + threadPoolEnabled
+ + ", pool: " + pool
+ + ", futures: " + futures;
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+
+ /**
+ * Get executor service for moving files for v1 algorithm.
+ * @return executor service
+ * @throws IOException on error
+ */
+ private ExecutorService createExecutorService() throws IOException {
+ // intentional validation
+ validateParallelMove();
+
+ ExecutorService pool = new ThreadPoolExecutor(moveThreads, moveThreads,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(),
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setNameFormat("FileCommitter-v1-move-thread-%d")
+ .build(),
+ new ThreadPoolExecutor.CallerRunsPolicy()
+ );
+ LOG.info("Size of move thread pool: {}, pool: {}", moveThreads, pool);
+ return pool;
+ }
+
+ private void parallelCommitJobInternal(JobContext context) throws IOException {
+ // validate to be on safer side.
+ validateParallelMove();
+
+ if (hasOutputPath()) {
+ Path finalOutput = getOutputPath();
+ FileSystem fs = finalOutput.getFileSystem(context.getConfiguration());
+ // created resilient commit helper bonded to the destination FS/path
+ resilientCommitHelper = new ResilientCommitByRenameHelper(fs);
Review comment:
This mechanism becomes very FileSystem specific. Implemented by Azure right now.
Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces. Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
Essentially - don't force usage of the new interface to get the benefits.
Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on a change in pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
sidseth commented on a change in pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#discussion_r740720279
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -497,6 +535,302 @@ private void mergePaths(FileSystem fs, final FileStatus from,
}
}
+ @VisibleForTesting
+ public int getMoveThreads() {
+ return moveThreads;
+ }
+
+ public boolean isParallelMoveEnabled() {
+ // Only available for algo v1
+ return (moveThreads > 1 && algorithmVersion == 1);
+ }
+
+ void validateParallelMove() throws IOException {
+ if (!isParallelMoveEnabled()) {
+ throw new IOException("Parallel file move is not enabled. "
+ + "moveThreads=" + moveThreads
+ + ", algo=" + algorithmVersion);
+ }
+ }
+
+ void validateThreadPool(ExecutorService pool, BlockingQueue<Future<Void>> futures)
+ throws IOException {
+ boolean threadPoolEnabled = isParallelMoveEnabled();
+ if (!threadPoolEnabled || pool == null || futures == null) {
+ String errorMsg = "Thread pool is not configured correctly. "
+ + "threadPoolEnabled: " + threadPoolEnabled
+ + ", pool: " + pool
+ + ", futures: " + futures;
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+
+ /**
+ * Get executor service for moving files for v1 algorithm.
+ * @return executor service
+ * @throws IOException on error
+ */
+ private ExecutorService createExecutorService() throws IOException {
+ // intentional validation
+ validateParallelMove();
+
+ ExecutorService pool = new ThreadPoolExecutor(moveThreads, moveThreads,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<>(),
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setNameFormat("FileCommitter-v1-move-thread-%d")
+ .build(),
+ new ThreadPoolExecutor.CallerRunsPolicy()
+ );
+ LOG.info("Size of move thread pool: {}, pool: {}", moveThreads, pool);
+ return pool;
+ }
+
+ private void parallelCommitJobInternal(JobContext context) throws IOException {
+ // validate to be on safer side.
+ validateParallelMove();
+
+ if (hasOutputPath()) {
+ Path finalOutput = getOutputPath();
+ FileSystem fs = finalOutput.getFileSystem(context.getConfiguration());
+ // created resilient commit helper bonded to the destination FS/path
+ resilientCommitHelper = new ResilientCommitByRenameHelper(fs);
Review comment:
This mechanism becomes very FileSystem specific. Implemented by Azure right now.
Other users of rename will not see the benefits without changing interfaces, which in turn requires shimming etc.
Would it be better for AzureFileSystem rename itself to add a config parameter which can lookup the src etag (at the cost of a performance hit for consistency), so that downstream components / any users of the rename operation can benefit from this change without having to change interfaces. Also, if the performance penalty is a big problem - Abfs could create very short-lived caches for FileStatus objects, and handle errors on discrepancies with the cached copy.
Essentially - don't force usage of the new interface to get the benefits.
Side note: The fs.getStatus within ResilientCommitByRenameHelper for FileSystems where this new functionality is not supported will lead to a performance penalty for the other FileSystems (performing a getFileStatus on src).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-957310383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-955112045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 36s | | trunk passed |
| +1 :green_heart: | compile | 33m 57s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 28m 42s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 26m 9s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 26m 9s | | the patch passed |
| +1 :green_heart: | compile | 22m 14s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 22m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 59s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 19 new + 99 unchanged - 0 fixed = 118 total (was 99) |
| +1 :green_heart: | mvnsite | 4m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 57s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 6m 19s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 143m 58s | | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | unit | 2m 38s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 434m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6ba9a731eb8d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f4950b3d0416b764433d18d24e8b19b09b663cc4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3597/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-954195372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3597: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3597:
URL: https://github.com/apache/hadoop/pull/3597#issuecomment-956975430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org