You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/02/06 15:12:23 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1834:
HADOOP-16848. Experimental directory marker optimization.
steveloughran opened a new pull request #1834: HADOOP-16848. Experimental directory marker optimization.
URL: https://github.com/apache/hadoop/pull/1834
# Optimizes directory marker operations through:-
* Async create/clear
* An option "fs.s3a.experimental.optimized.directory.operations"
That performs whatever optimisations to directory marker IO are considered
effective to reduce that I while still avoiding problems with missing markers
and/or markers not deleted.
# Important
1. Work in progress
2. Experimental: use at own risk
3. Not for use against buckets whether other S3A releases are active
4. And not for use with data you value.
View it as a Proof of Concept to explore what we can do here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1834: HADOOP-16848.
Experimental directory marker optimization.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1834: HADOOP-16848. Experimental directory marker optimization.
URL: https://github.com/apache/hadoop/pull/1834#issuecomment-586451195
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 54s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 32s | trunk passed |
| +1 :green_heart: | compile | 0m 43s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 31s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 5s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| -0 :warning: | checkstyle | 0m 22s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 38s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed |
| +1 :green_heart: | findbugs | 1m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 47s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 76m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1834 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8ce8b8fb1c88 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c75756f |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/testReport/ |
| Max. process+thread count | 337 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1834: HADOOP-16848.
Experimental directory marker optimization.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1834: HADOOP-16848. Experimental directory marker optimization.
URL: https://github.com/apache/hadoop/pull/1834#issuecomment-582952466
Tested. Ireland
Tests failed; not looked into the details
* some were clearly as the #of IOPs were less than the metrics' assertions
* and rename() contract tests were failing with, as usual. nothing meaningful from rename() other than "false"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1834: HADOOP-16848.
Experimental directory marker optimization.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1834: HADOOP-16848. Experimental directory marker optimization.
URL: https://github.com/apache/hadoop/pull/1834#issuecomment-583018959
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 36s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 3s | trunk passed |
| +1 :green_heart: | compile | 19m 33s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 52s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 1s | trunk passed |
| +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 32s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 17m 59s | the patch passed |
| +1 :green_heart: | javac | 17m 59s | the patch passed |
| -0 :warning: | checkstyle | 2m 57s | root: The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25) |
| +1 :green_heart: | mvnsite | 2m 16s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 4s | the patch passed |
| +1 :green_heart: | findbugs | 3m 42s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 24s | hadoop-common in the patch failed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
| | | 138m 7s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.TestFixKerberosTicketOrder |
| | hadoop.security.TestRaceWhenRelogin |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1834 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux dc58b925e539 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 146ca0f |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/testReport/ |
| Max. process+thread count | 1334 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1834: HADOOP-16848.
Experimental directory marker optimization.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1834: HADOOP-16848. Experimental directory marker optimization.
URL: https://github.com/apache/hadoop/pull/1834#issuecomment-583023065
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 25m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 29s | trunk passed |
| +1 :green_heart: | compile | 16m 47s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 33s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed |
| +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 8s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed |
| +1 :green_heart: | compile | 15m 51s | the patch passed |
| +1 :green_heart: | javac | 15m 51s | the patch passed |
| -0 :warning: | checkstyle | 2m 36s | root: The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25) |
| +1 :green_heart: | mvnsite | 2m 14s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 12m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | the patch passed |
| +1 :green_heart: | findbugs | 3m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 6s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. |
| | | 143m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1834 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c84056736665 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 146ca0f |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/testReport/ |
| Max. process+thread count | 1444 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1834/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org