You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2024/05/02 17:58:33 UTC
[PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
steveloughran opened a new pull request, #6789:
URL: https://github.com/apache/hadoop/pull/6789
Initial design
* no tests or docs
* served up via StoreContext. Not sure about the merits of that I think it is needed so it gets down to all AbstractStoreOperation instances, but should that be where the decision is made?
* create performance is wired up.
* as is path capabilities
For testing we need to make sure ths is unset from all cost tests.
relates to #6543; the logic to set up that operation is here...that PR would
just be the implementation.
Same for a delete optimisation where we'd skip parent dir probe.
rename could do the same for its source dir too.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2115097967
```
Reason: Use hadoop-common provided Sets rather than Guava provided Sets
in file: org/apache/hadoop/util/ConfigurationUtil.java
org.apache.hadoop.thirdparty.com.google.common.collect.Sets (Line: 33, Matched by: org.apache.hadoop.thirdparty.com.google.common.collect.Sets)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2148196238
@mukund-thakur @saxenapranav @HarshitGupta11 @ahmarsuhail @virajjasani
FYI, this is ready to review. Once in, viraj's mkdirs option can go in, and then we can document this option in performance.md
ABFS can use it for their options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran merged PR #6789:
URL: https://github.com/apache/hadoop/pull/6789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2215403995
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 35s | | trunk passed |
| +1 :green_heart: | compile | 18m 42s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 17m 47s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 47s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 18m 50s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 18m 50s | | the patch passed |
| +1 :green_heart: | compile | 18m 10s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 18m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 5m 22s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 116 unchanged - 0 fixed = 118 total (was 116) |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 43m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 4s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 43s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 285m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 8c687f3b19c4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b5f98533dc341d594402862bf648da25398b2ff |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2100244716
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 93m 12s | | trunk passed |
| +1 :green_heart: | compile | 5m 19s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 148m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 00s | | the patch passed |
| +1 :green_heart: | compile | 2m 20s | | the patch passed |
| +1 :green_heart: | javac | 2m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 09s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 163m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 44s | | The patch does not generate ASF License warnings. |
| | | 429m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 82fc63c5dfa5 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / feb384f7a19b454c0517f8cadc4a9f6865afcab4 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/1/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/1/console |
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2091557594
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 13s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 11s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 18s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. |
| | | 91m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 65a66ec22773 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / feb384f7a19b454c0517f8cadc4a9f6865afcab4 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/2/testReport/ |
| Max. process+thread count | 730 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1666949043
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FlagSet.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.util.ConfigurationHelper;
+import org.apache.hadoop.util.Preconditions;
+
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+
+/**
+ * A set of flags, constructed from a configuration option or from a string,
+ * with the semantics of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}
+ * and implementing {@link StreamCapabilities}.
+ * <p>
+ * Thread safety: there is no synchronization on a mutable {@code FlagSet}.
+ * Once declared immutable, flags cannot be changed, so they
+ * becomes implicitly thread-safe.
+ */
+public final class FlagSet<E extends Enum<E>> implements StreamCapabilities {
+
+ /**
+ * Set of flags.
+ */
+ private final Set<E> flags;
+
+ /**
+ * Is the set immutable?
+ */
+ private final AtomicBoolean immutable = new AtomicBoolean(false);
+
+ /**
+ * Mapping of prefixed flag names to enum values.
+ */
+ private final Map<String, E> namesToValues;
+
+ /**
+ * Create a FlagSet.
+ * @param enumClass class of enum
+ * @param prefix prefix (with trailing ".") for path capabilities probe
+ * @param flags flags. A copy of these are made.
+ */
+ private FlagSet(final Class<E> enumClass,
+ final String prefix,
+ @Nullable final EnumSet<E> flags) {
+
+ this.flags = flags != null
+ ? EnumSet.copyOf(flags)
+ : EnumSet.noneOf(enumClass);
+ this.namesToValues = mapEnumNamesToValues(prefix, enumClass);
+ }
+
+ /**
+ * Get a copy of the flags.
+ * <p>
+ * This is immutable.
+ * @return the flags.
+ */
+ public EnumSet<E> flags() {
+ return EnumSet.copyOf(flags);
+ }
+
+ /**
+ * Probe for the FlagSet being empty.
+ * @return true if there are no flags set.
+ */
+ public boolean isEmpty() {
+ return flags.isEmpty();
+ }
+
+ /**
+ * Is a flag enabled?
+ * @param flag flag to check
+ * @return true if it is in the set of enabled flags.
+ */
+ public boolean enabled(final E flag) {
+ return flags.contains(flag);
+ }
+
+ /**
+ * Check for mutability before any mutating operation.
+ * @throws IllegalStateException if the set is still mutable
+ */
+ private void checkMutable() {
+ Preconditions.checkState(!immutable.get(),
+ "FlagSet is immutable");
+ }
+
+ /**
+ * Enable a flag.
+ * @param flag flag to enable.
+ */
+ public void enable(final E flag) {
+ checkMutable();
+ flags.add(flag);
+ }
+
+ /**
+ * Disable a flag.
+ * @param flag flag to disable
+ */
+ public void disable(final E flag) {
+ checkMutable();
+ flags.remove(flag);
+ }
+
+ /**
+ * Set a flag to the chosen value.
+ * @param flag flag
+ * @param state true to enable, false to disable.
+ */
+ public void set(final E flag, boolean state) {
+ if (state) {
+ enable(flag);
+ } else {
+ disable(flag);
+ }
+ }
+
+ /**
+ * Is a flag enabled?
+ * @param capability string to query the stream support for.
+ * @return true if the capability maps to an enum value and
+ * that value is set.
+ */
+ @Override
+ public boolean hasCapability(final String capability) {
+ final E e = namesToValues.get(capability);
+ return e != null && enabled(e);
+ }
+
+ /**
+ * Make immutable; no-op if already set.
+ */
+ public void makeImmutable() {
+ immutable.set(true);
+ }
+
+ @Override
+ public String toString() {
+ return "{" +
+ (flags.stream()
+ .map(e -> e.name())
+ .collect(Collectors.joining(", ")))
+ + '}';
Review Comment:
aren't curly braces the ones for set representations?
also: https://xkcd.com/2954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2211093699
@virajjasani will adopt your comments next week, will need to rebase against the changes done by one "stevel" who appears to have broken this merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1673756812
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FlagSet.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.util.ConfigurationHelper;
+import org.apache.hadoop.util.Preconditions;
+
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+
+/**
+ * A set of flags, constructed from a configuration option or from a string,
+ * with the semantics of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}
+ * and implementing {@link StreamCapabilities}.
+ * <p>
+ * Thread safety: there is no synchronization on a mutable {@code FlagSet}.
+ * Once declared immutable, flags cannot be changed, so they
+ * becomes implicitly thread-safe.
+ */
+public final class FlagSet<E extends Enum<E>> implements StreamCapabilities {
+
+ /**
+ * Set of flags.
+ */
+ private final Set<E> flags;
+
+ /**
+ * Is the set immutable?
+ */
+ private final AtomicBoolean immutable = new AtomicBoolean(false);
Review Comment:
you can dynamically update things as you go along (which is being done to push that fs.s3a.create.performance flag in), then flip to being immutable after which it is safe to pass across threads. I had thought about keeping it permanently mutable and doing synchronized() on all changes, but felt this way was more lightweight
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "mukund-thakur (via GitHub)" <gi...@apache.org>.
mukund-thakur commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2223904082
> @mukund-thakur thanks, altered the tests and tried to clarify the immutability.
>
> One final thought: should we have a default list of enum values to set if, when parsing the config string, the string isn't found?
>
> Currently if nothing is found we return the empty set.
I am good with an empty set. What is the default value you are thinking? create is already getting added based on the create perf flag explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1687142291
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -2209,7 +2237,8 @@ public FSDataOutputStreamBuilder createFile(final Path path) {
builder
.create()
.overwrite(true)
- .must(FS_S3A_CREATE_PERFORMANCE, performanceCreation);
+ .must(FS_S3A_CREATE_PERFORMANCE,
+ getPerformanceFlags().enabled(PerformanceFlagEnum.Create));
Review Comment:
no worries, it was just a minor thing i was thinking about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2255569984
ok, everything is addressed, going to merge to the branches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2159447616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 54s | | trunk passed |
| +1 :green_heart: | compile | 16m 41s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 15m 27s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 44s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 16m 5s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 5s | | the patch passed |
| +1 :green_heart: | compile | 15m 47s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 10s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/8/artifact/out/results-checkstyle-root.txt) | root: The patch generated 6 new + 114 unchanged - 0 fixed = 120 total (was 114) |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| -1 :x: | javadoc | 1m 7s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/8/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 32s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 4s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 247m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 61c3d4e4c016 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / af2dd8d7fa56a93053e44372a32e8a1a7635f0f3 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/8/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2172302849
Sounds good, i would take a detailed look at ConfigurationHelper and FlagSet soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2221425671
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 9s | | trunk passed |
| +1 :green_heart: | compile | 16m 50s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 15m 32s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | compile | 15m 23s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 59s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 22s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 49s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 236m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 7722d8fce459 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d55ffebcba7f6c64f1d51220b3e9a789069958cc |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/testReport/ |
| Max. process+thread count | 1309 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1588408014
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3APerformanceFlags.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.util.Locale;
+
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE_ENABLED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+
+/**
+ * Performance flags.
+ * These are stored as a map of options.
+ */
+public final class S3APerformanceFlags implements StreamCapabilities {
+
+ /**
+ * Flag for create performance: {@value}.
+ */
+ public static final String CREATE = "create";
+
+ /**
+ * Flag for delete performance: {@value}.
+ */
+ public static final String DELETE = "delete";
+
+ /**
+ * Flag for mkdir performance: {@value}.
+ */
+ public static final String MKDIR = "mkdir";
+
+ /**
+ * Enable all performance flags: {@value}.
+ */
+ public static final String ALL = "*";
+
+ /**
+ * Higher performance create operations.
+ */
+ private boolean create;
+
+ /**
+ * Delete operation to skip parent probe.
+ */
+ private boolean delete;
+
+ /**
+ * Mkdir to skip checking for type of parent paths.
+ */
+ private boolean mkdir;
+
+ public S3APerformanceFlags() {
+ }
+
+ public boolean isCreate() {
+ return create;
+ }
+
+ public boolean isDelete() {
+ return delete;
+ }
Review Comment:
This one also we want to tackle as separate task (after HADOOP-19072), correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2129799708
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 7s | | trunk passed |
| +1 :green_heart: | compile | 19m 10s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 17m 13s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 18m 26s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 18m 26s | | the patch passed |
| +1 :green_heart: | compile | 17m 49s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 36s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 8 new + 114 unchanged - 0 fixed = 122 total (was 114) |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| -1 :x: | javadoc | 1m 8s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 13s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| -1 :x: | unit | 3m 13s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 270m 14s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestFilterFileSystem |
| | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.s3a.commit.staging.TestStagingCommitter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 11a43faccc32 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a564aa4d63f6b8a382b5365637f932dd4c28350b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2214711644
@virajjasani done your comments; had to rebase for what turned out to be a spurious merge conflict (adjacent methods in s3afs), so you can't see exactly what changed in the final commit. suffice to say: did your requests except related to builder and mutability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1589313816
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3APerformanceFlags.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.util.Locale;
+
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE_ENABLED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+
+/**
+ * Performance flags.
+ * These are stored as a map of options.
+ */
+public final class S3APerformanceFlags implements StreamCapabilities {
+
+ /**
+ * Flag for create performance: {@value}.
+ */
+ public static final String CREATE = "create";
+
+ /**
+ * Flag for delete performance: {@value}.
+ */
+ public static final String DELETE = "delete";
+
+ /**
+ * Flag for mkdir performance: {@value}.
+ */
+ public static final String MKDIR = "mkdir";
+
+ /**
+ * Enable all performance flags: {@value}.
+ */
+ public static final String ALL = "*";
+
+ /**
+ * Higher performance create operations.
+ */
+ private boolean create;
+
+ /**
+ * Delete operation to skip parent probe.
+ */
+ private boolean delete;
+
+ /**
+ * Mkdir to skip checking for type of parent paths.
+ */
+ private boolean mkdir;
+
+ public S3APerformanceFlags() {
+ }
+
+ public boolean isCreate() {
+ return create;
+ }
+
+ public boolean isDelete() {
+ return delete;
+ }
Review Comment:
yes. harshit did an experiment where he turned off all attempts at creating parent dirs after delete. fairly brittle, i think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1668938331
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestConfigurationHelper.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util;
+
+import java.util.Set;
+
+import org.assertj.core.api.Assertions;
+import org.assertj.core.api.IterableAssert;
+import org.junit.Test;
+
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static org.apache.hadoop.util.ConfigurationHelper.ERROR_MULTIPLE_ELEMENTS_MATCHING_TO_LOWER_CASE_VALUE;
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+import static org.apache.hadoop.util.ConfigurationHelper.parseEnumSet;
+
+/**
+ * Test for {@link ConfigurationHelper}.
+ */
+public class TestConfigurationHelper extends AbstractHadoopTestBase {
+
+ /**
+ * Create assertion about the outcome of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}.
+ * @param valueString value from Configuration
+ * @param enumClass class of enum
+ * @param ignoreUnknown should unknown values be ignored?
+ * @param <E> enum type
+ * @return an assertion on the outcome.
+ * @throws IllegalArgumentException if one of the entries was unknown and ignoreUnknown is false,
+ * or there are two entries in the enum which differ only by case.
+ */
+ private static <E extends Enum<E>> IterableAssert<E> assertEnumParse(
+ final String valueString,
+ final Class<E> enumClass,
+ final boolean ignoreUnknown) {
+ final Set<E> enumSet = parseEnumSet("key", valueString, enumClass, ignoreUnknown);
+ final IterableAssert<E> assertion = Assertions.assertThat(enumSet);
+ return assertion.describedAs("parsed enum set '%s'", valueString);
+ }
+
+ /**
+ * Simple Enums.
+ * "i" is included for case tests, as it is special in turkey.
+ */
+ private enum SimpleEnum { a, b, c, i }
+
+ @Test
+ public void testEnumParseAll() throws Throwable {
+ assertEnumParse("*", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEnumParse() throws Throwable {
+ assertEnumParse("a, b,c", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c);
+ }
+
+ @Test
+ public void testEnumCaseIndependence() throws Throwable {
+ assertEnumParse("A, B, C, I", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEmptyArguments() throws Throwable {
+ assertEnumParse(" ", SimpleEnum.class, false)
+ .isEmpty();
+ }
+
+ @Test
+ public void testUnknownEnumNotIgnored() throws Throwable {
+ intercept(IllegalArgumentException.class, "unrecognized", () ->
+ parseEnumSet("key", "c, unrecognized", SimpleEnum.class, false));
Review Comment:
let me do that as a separate test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2218728045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 24s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 16m 10s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 16m 55s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 55s | | the patch passed |
| +1 :green_heart: | compile | 16m 9s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 16m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 29s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 116 unchanged - 0 fixed = 117 total (was 116) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 49s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 55s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 242m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 7a1904c6c487 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c2168908fd65ec5be399289e425c537f6ba3a63c |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/testReport/ |
| Max. process+thread count | 1379 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2113444831
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 57s | | trunk passed |
| +1 :green_heart: | compile | 17m 3s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 15m 44s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 38s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 22s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | compile | 0m 36s | [/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 36s | [/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt) | root in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 35s | [/patch-compile-root-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 35s | [/patch-compile-root-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 45s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 7 new + 5 unchanged - 0 fixed = 12 total (was 5) |
| -1 :x: | mvnsite | 0m 26s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt) | hadoop-common in the patch failed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 22s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | shadedclient | 2m 14s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 22s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 149m 15s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux da1d873b768f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 28cd50ac8773061969049071eab0c8adca5faa83 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/testReport/ |
| Max. process+thread count | 549 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2168578759
OK, I'm happy with this.
@saxenapranav can you look at this? this flagset is what I hope to use for a similar set of options in abfs -which is why the parser lives outside of Configuration.
@mukund-thakur @virajjasani thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2218160335
```
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:997: .setPerformanceFlags(FlagSet.createFlagSet(PerformanceFlagEnum.class,FS_S3A_PERFORMANCE_FLAGS)): Line is longer than 100 characters (found 103). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:997: .setPerformanceFlags(FlagSet.createFlagSet(PerformanceFlagEnum.class,FS_S3A_PERFORMANCE_FLAGS)):77: ',' is not followed by whitespace. [WhitespaceAfter]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2223176470
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 9s | | trunk passed |
| +1 :green_heart: | compile | 16m 59s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 15m 48s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 16m 19s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 19s | | the patch passed |
| +1 :green_heart: | compile | 15m 25s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 19s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 48s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 236m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 4c294b2bd8b6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f330597d6801db879011a95793e8555705e6e355 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/testReport/ |
| Max. process+thread count | 2014 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2108338924
I have a better design for this. changign this to draft.
Proposed: we have a `Configuration.getEnumOptions(Enum x, boolean failIfUnknown)` which returns an EnumSet of all values of the enum class whose valueOf() matches an entry in the CSV list (with some mapping such as case conversion, and map - and . to "_".
this makes it trivial to reuse/process. the implementation would be outside the actual Configuration class to make it easy for AbfsConfiguration to use too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1689944706
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md:
##########
@@ -276,7 +283,45 @@ Fix: Use one of the dedicated [S3A Committers](committers.md).
## <a name="tuning"></a> Options to Tune
-### <a name="pooling"></a> Thread and connection pool settings.
+### <a name="flags"></a> Performance Flags: `fs.s3a.performance.flag`
+
+This option takes a comma separated list of performance flags.
+View it as the equivalent of the `-O` compiler optimization list C/C++ compilers offer.
+That is a complicated list of options which deliver speed if the person setting them
+understands the risks.
Review Comment:
no, its an individual who takes the blame
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md:
##########
@@ -276,7 +283,45 @@ Fix: Use one of the dedicated [S3A Committers](committers.md).
## <a name="tuning"></a> Options to Tune
-### <a name="pooling"></a> Thread and connection pool settings.
+### <a name="flags"></a> Performance Flags: `fs.s3a.performance.flag`
+
+This option takes a comma separated list of performance flags.
+View it as the equivalent of the `-O` compiler optimization list C/C++ compilers offer.
+That is a complicated list of options which deliver speed if the person setting them
+understands the risks.
+
+* The list of flags MAY change across releases
+* The semantics of specific flags SHOULD NOT change across releases.
+* If an option is to be tuned which may relax semantics, a new option MUST be defined.
+* Unknown flags are ignored; this is to avoid compatibility.
+* The option `*` means "turn everything on". This is implicitly unstable across releases.
+
+| *Option* | *Meaning* | Since |
+|----------|--------------------|:------|
+| `create` | Create Performance | 3.4.1 |
Review Comment:
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2248536907
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 10s | | trunk passed |
| +1 :green_heart: | compile | 9m 0s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 8m 5s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | compile | 8m 7s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 8m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 12s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 9s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 37s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 10s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 150m 45s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 0378336d9b15 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce8dac0c544508a002790d4a8c5e09251b6500ff |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/testReport/ |
| Max. process+thread count | 1282 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2215296682
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 21s | | trunk passed |
| +1 :green_heart: | compile | 17m 21s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 16m 7s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 16m 42s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 42s | | the patch passed |
| +1 :green_heart: | compile | 15m 54s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 25s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 116 unchanged - 0 fixed = 118 total (was 116) |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 59s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 48s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 53s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not generate ASF License warnings. |
| | | 242m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 937d04fdd13c 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b5f98533dc341d594402862bf648da25398b2ff |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2150381808
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 26s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 5s | | trunk passed |
| +1 :green_heart: | compile | 16m 57s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 15m 39s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 16m 5s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 5s | | the patch passed |
| +1 :green_heart: | compile | 15m 26s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 15m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 35s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 57s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 252m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5002c77b301e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 206bfaba910bd7bceb60e9be8329db822e67e346 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/7/testReport/ |
| Max. process+thread count | 3151 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2166096602
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 10s | | trunk passed |
| +1 :green_heart: | compile | 17m 3s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 15m 25s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 38s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 37m 2s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 16m 11s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 11s | | the patch passed |
| +1 :green_heart: | compile | 15m 21s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 20s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 239m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux e0075c95c38c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 475671470f555d38ea74eea0ae65efc3a84e170c |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/10/testReport/ |
| Max. process+thread count | 1309 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2137891048
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 53s | | trunk passed |
| +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 24s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 5m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 53s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 17m 27s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 17m 27s | | the patch passed |
| +1 :green_heart: | compile | 16m 23s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 16m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 1 new + 114 unchanged - 0 fixed = 115 total (was 114) |
| +1 :green_heart: | mvnsite | 2m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 35s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/6/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 55s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 255m 9s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.TestFilterFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a3d5f3ebdd16 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 82974c4851436db780129c4d952442fb40586029 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/6/testReport/ |
| Max. process+thread count | 3151 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2127773836
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 9s | | trunk passed |
| +1 :green_heart: | compile | 16m 51s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 15m 53s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 16m 20s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 20s | | the patch passed |
| +1 :green_heart: | compile | 15m 34s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 15m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 12s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 8 new + 114 unchanged - 0 fixed = 122 total (was 114) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| -1 :x: | javadoc | 1m 8s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 20m 33s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch passed. |
| -1 :x: | unit | 3m 6s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 241m 25s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| | hadoop.fs.TestFilterFileSystem |
| | hadoop.fs.s3a.commit.staging.TestStagingCommitter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ae49e1bace7b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f3571ac99b7c0821d4a446bf3bf7ab7cc49752da |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/testReport/ |
| Max. process+thread count | 3151 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2091484477
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 138m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 35b13dae427f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5b2974fa412a7f6d67a7d5809cfdc8f3020d96c3 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/1/testReport/ |
| Max. process+thread count | 558 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1588404029
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3APerformanceFlags.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.util.Locale;
+
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE_ENABLED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+
+/**
+ * Performance flags.
+ * These are stored as a map of options.
+ */
+public final class S3APerformanceFlags implements StreamCapabilities {
+
+ /**
+ * Flag for create performance: {@value}.
+ */
+ public static final String CREATE = "create";
+
+ /**
+ * Flag for delete performance: {@value}.
+ */
+ public static final String DELETE = "delete";
+
+ /**
+ * Flag for mkdir performance: {@value}.
+ */
+ public static final String MKDIR = "mkdir";
+
+ /**
+ * Enable all performance flags: {@value}.
+ */
+ public static final String ALL = "*";
+
+ /**
+ * Higher performance create operations.
+ */
+ private boolean create;
+
+ /**
+ * Delete operation to skip parent probe.
+ */
+ private boolean delete;
+
+ /**
+ * Mkdir to skip checking for type of parent paths.
+ */
+ private boolean mkdir;
+
+ public S3APerformanceFlags() {
+ }
+
+ public boolean isCreate() {
+ return create;
+ }
+
+ public boolean isDelete() {
+ return delete;
+ }
+
+ public boolean isMkdir() {
+ return mkdir;
+ }
+
+ public S3APerformanceFlags setCreate(final boolean create) {
+ this.create = create;
+ return this;
+ }
+
+ public S3APerformanceFlags setDelete(final boolean delete) {
+ this.delete = delete;
+ return this;
+ }
+
+ public S3APerformanceFlags setMkdir(final boolean mkdir) {
+ this.mkdir = mkdir;
+ return this;
+ }
+
+
+ @Override
+ public boolean hasCapability(final String capability) {
+ switch (capability.toLowerCase(Locale.ROOT)) {
+ case FS_S3A_PERFORMANCE_FLAGS + CREATE:
+ case FS_S3A_CREATE_PERFORMANCE_ENABLED:
+ return isCreate();
+
+ case FS_S3A_PERFORMANCE_FLAGS + MKDIR:
+ return isMkdir();
+
+ case FS_S3A_PERFORMANCE_FLAGS + DELETE:
+ return isDelete();
+
+ default:
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return "S3APerformanceFlags{" +
+ "create=" + create +
+ ", delete=" + delete +
+ ", mkdir=" + mkdir +
+ '}';
+ }
+
+ /**
+ * Create a performance flags instance from a list of options.
+ * @param options options from a configuration string.
+ * @return a set of options
+ */
+ public static S3APerformanceFlags build(String... options) {
+ S3APerformanceFlags flags = new S3APerformanceFlags();
+ for (String option : options) {
+ switch (option.trim().toLowerCase(Locale.ROOT)) {
+ case CREATE:
+ flags.create = true;
+ break;
+ case DELETE:
+ flags.delete = true;
+ break;
+ case MKDIR:
+ flags.mkdir = true;
+ break;
+ case ALL:
+ flags.create = true;
+ flags.mkdir = true;
+ flags.delete = true;
+ break;
+
+/* case "hive":
+ case "impala":
+ case "spark":
+ case "distcp":
Review Comment:
Should we not let downstreamers decide what flags they want (after extensive testing)? And across different releases, they might need different flags to be turned on (in case of any regression)?
We can just recommend the flags (as already commented out here) but not set the flags for them. Thoughts?
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3APerformanceFlags.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.util.Locale;
+
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE_ENABLED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+
+/**
+ * Performance flags.
+ * These are stored as a map of options.
+ */
+public final class S3APerformanceFlags implements StreamCapabilities {
+
+ /**
+ * Flag for create performance: {@value}.
+ */
+ public static final String CREATE = "create";
+
+ /**
+ * Flag for delete performance: {@value}.
+ */
+ public static final String DELETE = "delete";
+
+ /**
+ * Flag for mkdir performance: {@value}.
+ */
+ public static final String MKDIR = "mkdir";
+
+ /**
+ * Enable all performance flags: {@value}.
+ */
+ public static final String ALL = "*";
+
+ /**
+ * Higher performance create operations.
+ */
+ private boolean create;
+
+ /**
+ * Delete operation to skip parent probe.
+ */
+ private boolean delete;
+
+ /**
+ * Mkdir to skip checking for type of parent paths.
+ */
+ private boolean mkdir;
+
+ public S3APerformanceFlags() {
+ }
+
+ public boolean isCreate() {
+ return create;
+ }
+
+ public boolean isDelete() {
+ return delete;
+ }
Review Comment:
This one also we want to tackle as separate task (similar to HADOOP-19072), correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1589312846
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3APerformanceFlags.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.util.Locale;
+
+import org.apache.hadoop.fs.StreamCapabilities;
+
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE_ENABLED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+
+/**
+ * Performance flags.
+ * These are stored as a map of options.
+ */
+public final class S3APerformanceFlags implements StreamCapabilities {
+
+ /**
+ * Flag for create performance: {@value}.
+ */
+ public static final String CREATE = "create";
+
+ /**
+ * Flag for delete performance: {@value}.
+ */
+ public static final String DELETE = "delete";
+
+ /**
+ * Flag for mkdir performance: {@value}.
+ */
+ public static final String MKDIR = "mkdir";
+
+ /**
+ * Enable all performance flags: {@value}.
+ */
+ public static final String ALL = "*";
+
+ /**
+ * Higher performance create operations.
+ */
+ private boolean create;
+
+ /**
+ * Delete operation to skip parent probe.
+ */
+ private boolean delete;
+
+ /**
+ * Mkdir to skip checking for type of parent paths.
+ */
+ private boolean mkdir;
+
+ public S3APerformanceFlags() {
+ }
+
+ public boolean isCreate() {
+ return create;
+ }
+
+ public boolean isDelete() {
+ return delete;
+ }
+
+ public boolean isMkdir() {
+ return mkdir;
+ }
+
+ public S3APerformanceFlags setCreate(final boolean create) {
+ this.create = create;
+ return this;
+ }
+
+ public S3APerformanceFlags setDelete(final boolean delete) {
+ this.delete = delete;
+ return this;
+ }
+
+ public S3APerformanceFlags setMkdir(final boolean mkdir) {
+ this.mkdir = mkdir;
+ return this;
+ }
+
+
+ @Override
+ public boolean hasCapability(final String capability) {
+ switch (capability.toLowerCase(Locale.ROOT)) {
+ case FS_S3A_PERFORMANCE_FLAGS + CREATE:
+ case FS_S3A_CREATE_PERFORMANCE_ENABLED:
+ return isCreate();
+
+ case FS_S3A_PERFORMANCE_FLAGS + MKDIR:
+ return isMkdir();
+
+ case FS_S3A_PERFORMANCE_FLAGS + DELETE:
+ return isDelete();
+
+ default:
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return "S3APerformanceFlags{" +
+ "create=" + create +
+ ", delete=" + delete +
+ ", mkdir=" + mkdir +
+ '}';
+ }
+
+ /**
+ * Create a performance flags instance from a list of options.
+ * @param options options from a configuration string.
+ * @return a set of options
+ */
+ public static S3APerformanceFlags build(String... options) {
+ S3APerformanceFlags flags = new S3APerformanceFlags();
+ for (String option : options) {
+ switch (option.trim().toLowerCase(Locale.ROOT)) {
+ case CREATE:
+ flags.create = true;
+ break;
+ case DELETE:
+ flags.delete = true;
+ break;
+ case MKDIR:
+ flags.mkdir = true;
+ break;
+ case ALL:
+ flags.create = true;
+ flags.mkdir = true;
+ flags.delete = true;
+ break;
+
+/* case "hive":
+ case "impala":
+ case "spark":
+ case "distcp":
Review Comment:
harshit and I were discussing this. i think it's best to have that option list, as app settings could be too brittle to changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2091180025
note the commented out bit where we considered adding options like "hive" or "spark".
@HarshitGupta11 and I discussed this; for now lets go with a list of options and "*"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1683479251
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md:
##########
@@ -276,7 +283,45 @@ Fix: Use one of the dedicated [S3A Committers](committers.md).
## <a name="tuning"></a> Options to Tune
-### <a name="pooling"></a> Thread and connection pool settings.
+### <a name="flags"></a> Performance Flags: `fs.s3a.performance.flag`
+
+This option takes a comma separated list of performance flags.
+View it as the equivalent of the `-O` compiler optimization list C/C++ compilers offer.
+That is a complicated list of options which deliver speed if the person setting them
+understands the risks.
+
+* The list of flags MAY change across releases
+* The semantics of specific flags SHOULD NOT change across releases.
+* If an option is to be tuned which may relax semantics, a new option MUST be defined.
+* Unknown flags are ignored; this is to avoid compatibility.
+* The option `*` means "turn everything on". This is implicitly unstable across releases.
+
+| *Option* | *Meaning* | Since |
+|----------|--------------------|:------|
+| `create` | Create Performance | 3.4.1 |
Review Comment:
nit: shall we mention `3.4.1 / 3.5.0`?
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -2209,7 +2237,8 @@ public FSDataOutputStreamBuilder createFile(final Path path) {
builder
.create()
.overwrite(true)
- .must(FS_S3A_CREATE_PERFORMANCE, performanceCreation);
+ .must(FS_S3A_CREATE_PERFORMANCE,
+ getPerformanceFlags().enabled(PerformanceFlagEnum.Create));
Review Comment:
just a thought not related to this patch, but shall we make `FSDataOutputStreamBuilder` more specific for s3afs
e.g.
`public FSDataOutputStreamBuilder<FSDataOutputStream, CreateFileBuilder> createFile(final Path path)`
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestFlagSet.java:
##########
@@ -0,0 +1,431 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.EnumSet;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+
+import static java.util.EnumSet.allOf;
+import static java.util.EnumSet.noneOf;
+import static org.apache.hadoop.fs.impl.FlagSet.buildFlagSet;
+import static org.apache.hadoop.fs.impl.FlagSet.createFlagSet;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Unit tests for {@link FlagSet} class.
+ */
+public final class TestFlagSet extends AbstractHadoopTestBase {
+
+ private static final String KEY = "key";
+
+ public static final String CAPABILITY_B = KEY + ".b";
+
+ public static final String CAPABILITY_C = KEY + ".c";
+
+ public static final String CAPABILITY_A = KEY + ".a";
+
+ private static final String KEYDOT = KEY + ".";
+
+ /**
+ * Flagset used in tests and assertions.
+ */
+ private FlagSet<SimpleEnum> flagSet =
+ createFlagSet(SimpleEnum.class, KEYDOT, noneOf(SimpleEnum.class));
+
+ /**
+ * Simple Enums for the tests.
+ */
+ private enum SimpleEnum { a, b, c }
+
+ /**
+ * Enum with a single value.
+ */
+ private enum OtherEnum { a }
+
+ /**
+ * Test that an entry can be enabled and disabled.
+ */
+ @Test
+ public void testEntryEnableDisable() {
+ Assertions.assertThat(flagSet.flags()).isEmpty();
+ assertDisabled(SimpleEnum.a);
+ flagSet.enable(SimpleEnum.a);
+ assertEnabled(SimpleEnum.a);
+ flagSet.disable(SimpleEnum.a);
+ assertDisabled(SimpleEnum.a);
+ }
+
+ /**
+ * Test the setter.
+ */
+ @Test
+ public void testSetMethod() {
+ Assertions.assertThat(flagSet.flags()).isEmpty();
+ flagSet.set(SimpleEnum.a, true);
+ assertEnabled(SimpleEnum.a);
+ flagSet.set(SimpleEnum.a, false);
+ assertDisabled(SimpleEnum.a);
+ }
+
+ /**
+ * Test mutability by making immutable and
+ * expecting setters to fail.
+ */
+ @Test
+ public void testMutability() throws Throwable {
+ flagSet.set(SimpleEnum.a, true);
+ flagSet.makeImmutable();
+ intercept(IllegalStateException.class, () ->
+ flagSet.disable(SimpleEnum.a));
+ assertEnabled(SimpleEnum.a);
+ intercept(IllegalStateException.class, () ->
+ flagSet.set(SimpleEnum.a, false));
+ assertEnabled(SimpleEnum.a);
+ // now look at the setters
+ intercept(IllegalStateException.class, () ->
+ flagSet.enable(SimpleEnum.b));
+ assertDisabled(SimpleEnum.b);
+ intercept(IllegalStateException.class, () ->
+ flagSet.set(SimpleEnum.b, true));
+ assertDisabled(SimpleEnum.b);
+ }
+
+ /**
+ * Test stringification.
+ */
+ @Test
+ public void testToString() throws Throwable {
+ // empty
+ assertStringValue("{}");
+ assertConfigurationStringMatches("");
+
+ // single value
+ flagSet.enable(SimpleEnum.a);
+ assertStringValue("{a}");
+ assertConfigurationStringMatches("a");
+
+ // add a second value.
+ flagSet.enable(SimpleEnum.b);
+ assertStringValue("{a, b}");
+ }
+
+ /**
+ * Assert that {@link FlagSet#toString()} matches the expected
+ * value.
+ * @param expected expected value
+ */
+ private void assertStringValue(final String expected) {
+ Assertions.assertThat(flagSet.toString())
+ .isEqualTo(expected);
+ }
+
+ /**
+ * Assert the configuration string form matches that expected.
+ */
+ public void assertConfigurationStringMatches(final String expected) {
+ Assertions.assertThat(flagSet.toConfigurationString())
+ .describedAs("Configuration string of %s", flagSet)
+ .isEqualTo(expected);
+ }
+
+ /**
+ * Test parsing from a configuration file.
+ * Multiple entries must be parsed, whitespace trimmed.
+ */
+ @Test
+ public void testConfEntry() {
+ flagSet = flagSetFromConfig("a\t,\nc ", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a, SimpleEnum.c);
+ assertHasCapability(CAPABILITY_A);
+ assertHasCapability(CAPABILITY_C);
+ assertLacksCapability(CAPABILITY_B);
+ assertPathCapabilitiesMatch(flagSet, CAPABILITY_A, CAPABILITY_C);
+ }
+
+ /**
+ * Create a flagset from a configuration string.
+ * @param string configuration string.
+ * @param ignoreUnknown should unknown values be ignored?
+ * @return a flagset
+ */
+ private static FlagSet<SimpleEnum> flagSetFromConfig(final String string,
+ final boolean ignoreUnknown) {
+ final Configuration conf = mkConf(string);
+ return buildFlagSet(SimpleEnum.class, conf, KEY, ignoreUnknown);
+ }
+
+ /**
+ * Test parsing from a configuration file,
+ * where an entry is unknown; the builder is set to ignoreUnknown.
+ */
+ @Test
+ public void testConfEntryWithUnknownIgnored() {
+ flagSet = flagSetFromConfig("a, unknown", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a);
+ assertHasCapability(CAPABILITY_A);
+ assertLacksCapability(CAPABILITY_B);
+ assertLacksCapability(CAPABILITY_C);
+ }
+
+ /**
+ * Test parsing from a configuration file where
+ * the same entry is duplicated.
+ */
+ @Test
+ public void testDuplicateConfEntry() {
+ flagSet = flagSetFromConfig("a,\ta,\na\"", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a);
+ assertHasCapability(CAPABILITY_A);
+ }
+
+ /**
+ * Handle an unknown configuration value.
+ */
+ @Test
+ public void testConfUnknownFailure() throws Throwable {
+ intercept(IllegalArgumentException.class, () ->
+ flagSetFromConfig("a, unknown", false));
+ }
+
+ /**
+ * Create a configuration with {@link #KEY} set to the given value.
+ * @param value value to set
+ * @return the configuration.
+ */
+ private static Configuration mkConf(final String value) {
+ final Configuration conf = new Configuration(false);
+ conf.set(KEY, value);
+ return conf;
+ }
+
+ /**
+ * Assert that the flagset has a capability.
+ * @param capability capability to probe for
+ */
+ private void assertHasCapability(final String capability) {
+ Assertions.assertThat(flagSet.hasCapability(capability))
+ .describedAs("Capabiilty of %s on %s", capability, flagSet)
Review Comment:
nit: `Capability`
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md:
##########
@@ -276,7 +283,45 @@ Fix: Use one of the dedicated [S3A Committers](committers.md).
## <a name="tuning"></a> Options to Tune
-### <a name="pooling"></a> Thread and connection pool settings.
+### <a name="flags"></a> Performance Flags: `fs.s3a.performance.flag`
+
+This option takes a comma separated list of performance flags.
+View it as the equivalent of the `-O` compiler optimization list C/C++ compilers offer.
+That is a complicated list of options which deliver speed if the person setting them
+understands the risks.
Review Comment:
nit: just in case if it looks better? `if the person or client application setting them`
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestFlagSet.java:
##########
@@ -0,0 +1,431 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.EnumSet;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+
+import static java.util.EnumSet.allOf;
+import static java.util.EnumSet.noneOf;
+import static org.apache.hadoop.fs.impl.FlagSet.buildFlagSet;
+import static org.apache.hadoop.fs.impl.FlagSet.createFlagSet;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Unit tests for {@link FlagSet} class.
+ */
+public final class TestFlagSet extends AbstractHadoopTestBase {
+
+ private static final String KEY = "key";
+
+ public static final String CAPABILITY_B = KEY + ".b";
+
+ public static final String CAPABILITY_C = KEY + ".c";
+
+ public static final String CAPABILITY_A = KEY + ".a";
+
+ private static final String KEYDOT = KEY + ".";
+
+ /**
+ * Flagset used in tests and assertions.
+ */
+ private FlagSet<SimpleEnum> flagSet =
+ createFlagSet(SimpleEnum.class, KEYDOT, noneOf(SimpleEnum.class));
+
+ /**
+ * Simple Enums for the tests.
+ */
+ private enum SimpleEnum { a, b, c }
+
+ /**
+ * Enum with a single value.
+ */
+ private enum OtherEnum { a }
+
+ /**
+ * Test that an entry can be enabled and disabled.
+ */
+ @Test
+ public void testEntryEnableDisable() {
+ Assertions.assertThat(flagSet.flags()).isEmpty();
+ assertDisabled(SimpleEnum.a);
+ flagSet.enable(SimpleEnum.a);
+ assertEnabled(SimpleEnum.a);
+ flagSet.disable(SimpleEnum.a);
+ assertDisabled(SimpleEnum.a);
+ }
+
+ /**
+ * Test the setter.
+ */
+ @Test
+ public void testSetMethod() {
+ Assertions.assertThat(flagSet.flags()).isEmpty();
+ flagSet.set(SimpleEnum.a, true);
+ assertEnabled(SimpleEnum.a);
+ flagSet.set(SimpleEnum.a, false);
+ assertDisabled(SimpleEnum.a);
+ }
+
+ /**
+ * Test mutability by making immutable and
+ * expecting setters to fail.
+ */
+ @Test
+ public void testMutability() throws Throwable {
+ flagSet.set(SimpleEnum.a, true);
+ flagSet.makeImmutable();
+ intercept(IllegalStateException.class, () ->
+ flagSet.disable(SimpleEnum.a));
+ assertEnabled(SimpleEnum.a);
+ intercept(IllegalStateException.class, () ->
+ flagSet.set(SimpleEnum.a, false));
+ assertEnabled(SimpleEnum.a);
+ // now look at the setters
+ intercept(IllegalStateException.class, () ->
+ flagSet.enable(SimpleEnum.b));
+ assertDisabled(SimpleEnum.b);
+ intercept(IllegalStateException.class, () ->
+ flagSet.set(SimpleEnum.b, true));
+ assertDisabled(SimpleEnum.b);
+ }
+
+ /**
+ * Test stringification.
+ */
+ @Test
+ public void testToString() throws Throwable {
+ // empty
+ assertStringValue("{}");
+ assertConfigurationStringMatches("");
+
+ // single value
+ flagSet.enable(SimpleEnum.a);
+ assertStringValue("{a}");
+ assertConfigurationStringMatches("a");
+
+ // add a second value.
+ flagSet.enable(SimpleEnum.b);
+ assertStringValue("{a, b}");
+ }
+
+ /**
+ * Assert that {@link FlagSet#toString()} matches the expected
+ * value.
+ * @param expected expected value
+ */
+ private void assertStringValue(final String expected) {
+ Assertions.assertThat(flagSet.toString())
+ .isEqualTo(expected);
+ }
+
+ /**
+ * Assert the configuration string form matches that expected.
+ */
+ public void assertConfigurationStringMatches(final String expected) {
+ Assertions.assertThat(flagSet.toConfigurationString())
+ .describedAs("Configuration string of %s", flagSet)
+ .isEqualTo(expected);
+ }
+
+ /**
+ * Test parsing from a configuration file.
+ * Multiple entries must be parsed, whitespace trimmed.
+ */
+ @Test
+ public void testConfEntry() {
+ flagSet = flagSetFromConfig("a\t,\nc ", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a, SimpleEnum.c);
+ assertHasCapability(CAPABILITY_A);
+ assertHasCapability(CAPABILITY_C);
+ assertLacksCapability(CAPABILITY_B);
+ assertPathCapabilitiesMatch(flagSet, CAPABILITY_A, CAPABILITY_C);
+ }
+
+ /**
+ * Create a flagset from a configuration string.
+ * @param string configuration string.
+ * @param ignoreUnknown should unknown values be ignored?
+ * @return a flagset
+ */
+ private static FlagSet<SimpleEnum> flagSetFromConfig(final String string,
+ final boolean ignoreUnknown) {
+ final Configuration conf = mkConf(string);
+ return buildFlagSet(SimpleEnum.class, conf, KEY, ignoreUnknown);
+ }
+
+ /**
+ * Test parsing from a configuration file,
+ * where an entry is unknown; the builder is set to ignoreUnknown.
+ */
+ @Test
+ public void testConfEntryWithUnknownIgnored() {
+ flagSet = flagSetFromConfig("a, unknown", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a);
+ assertHasCapability(CAPABILITY_A);
+ assertLacksCapability(CAPABILITY_B);
+ assertLacksCapability(CAPABILITY_C);
+ }
+
+ /**
+ * Test parsing from a configuration file where
+ * the same entry is duplicated.
+ */
+ @Test
+ public void testDuplicateConfEntry() {
+ flagSet = flagSetFromConfig("a,\ta,\na\"", true);
+ assertFlagSetMatches(flagSet, SimpleEnum.a);
+ assertHasCapability(CAPABILITY_A);
+ }
+
+ /**
+ * Handle an unknown configuration value.
+ */
+ @Test
+ public void testConfUnknownFailure() throws Throwable {
+ intercept(IllegalArgumentException.class, () ->
+ flagSetFromConfig("a, unknown", false));
+ }
+
+ /**
+ * Create a configuration with {@link #KEY} set to the given value.
+ * @param value value to set
+ * @return the configuration.
+ */
+ private static Configuration mkConf(final String value) {
+ final Configuration conf = new Configuration(false);
+ conf.set(KEY, value);
+ return conf;
+ }
+
+ /**
+ * Assert that the flagset has a capability.
+ * @param capability capability to probe for
+ */
+ private void assertHasCapability(final String capability) {
+ Assertions.assertThat(flagSet.hasCapability(capability))
+ .describedAs("Capabiilty of %s on %s", capability, flagSet)
+ .isTrue();
+ }
+
+ /**
+ * Assert that the flagset lacks a capability.
+ * @param capability capability to probe for
+ */
+ private void assertLacksCapability(final String capability) {
+ Assertions.assertThat(flagSet.hasCapability(capability))
+ .describedAs("Capabiilty of %s on %s", capability, flagSet)
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "mukund-thakur (via GitHub)" <gi...@apache.org>.
mukund-thakur commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1672860768
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ConfigurationHelper.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util;
+
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+
+import static java.util.EnumSet.noneOf;
+import static org.apache.hadoop.util.Preconditions.checkArgument;
+import static org.apache.hadoop.util.StringUtils.getTrimmedStringCollection;
+
+/**
+ * Configuration Helper class to provide advanced configuration parsing.
+ * Private; external code MUST use {@link Configuration} instead
+ */
+@InterfaceAudience.Private
+public final class ConfigurationHelper {
+
+ /**
+ * Error string if there are multiple enum elements which only differ
+ * by case: {@value}.
+ */
+ @VisibleForTesting
+ static final String ERROR_MULTIPLE_ELEMENTS_MATCHING_TO_LOWER_CASE_VALUE =
+ "has multiple elements matching to lower case value";
+
+ private ConfigurationHelper() {
+ }
+
+ /**
+ * Given a comma separated list of enum values,
+ * trim the list, map to enum values in the message (case insensitive)
+ * and return the set.
+ * Special handling of "*" meaning: all values.
+ * @param key key for error messages.
Review Comment:
don't see much use of key param.
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestConfigurationHelper.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util;
+
+import java.util.Set;
+
+import org.assertj.core.api.Assertions;
+import org.assertj.core.api.IterableAssert;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static org.apache.hadoop.util.ConfigurationHelper.ERROR_MULTIPLE_ELEMENTS_MATCHING_TO_LOWER_CASE_VALUE;
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+import static org.apache.hadoop.util.ConfigurationHelper.parseEnumSet;
+
+/**
+ * Test for {@link ConfigurationHelper}.
+ */
+public class TestConfigurationHelper extends AbstractHadoopTestBase {
+
+ /**
+ * Simple Enums.
+ * "i" is included for case tests, as it is special in turkey.
+ */
+ private enum SimpleEnum { a, b, c, i }
+
+
+ /**
+ * Special case: an enum with no values.
+ */
+ private enum EmptyEnum { }
+
+ /**
+ * Create assertion about the outcome of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}.
+ * @param valueString value from Configuration
+ * @param enumClass class of enum
+ * @param ignoreUnknown should unknown values be ignored?
+ * @param <E> enum type
+ * @return an assertion on the outcome.
+ * @throws IllegalArgumentException if one of the entries was unknown and ignoreUnknown is false,
+ * or there are two entries in the enum which differ only by case.
+ */
+ private static <E extends Enum<E>> IterableAssert<E> assertEnumParse(
+ final String valueString,
+ final Class<E> enumClass,
+ final boolean ignoreUnknown) {
+ final Set<E> enumSet = parseEnumSet("key", valueString, enumClass, ignoreUnknown);
+ final IterableAssert<E> assertion = Assertions.assertThat(enumSet);
+ return assertion.describedAs("parsed enum set '%s'", valueString);
+ }
+
+
+ /**
+ * Create a configuration with the key {@code key} set to a {@code value}.
+ * @param value value for the key
+ * @return a configuration with only key set.
+ */
+ private Configuration confWithKey(String value) {
+ final Configuration conf = new Configuration(false);
+ conf.set("key", value);
+ return conf;
+ }
+
+ @Test
+ public void testEnumParseAll() throws Throwable {
+ assertEnumParse("*", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEnumParse() throws Throwable {
+ assertEnumParse("a, b,c", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c);
+ }
+
+ @Test
+ public void testEnumCaseIndependence() throws Throwable {
+ assertEnumParse("A, B, C, I", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEmptyArguments() throws Throwable {
+ assertEnumParse(" ", SimpleEnum.class, false)
+ .isEmpty();
+ }
+
+ @Test
+ public void testUnknownEnumNotIgnored() throws Throwable {
+ intercept(IllegalArgumentException.class, "unrecognized", () ->
+ parseEnumSet("key", "c, unrecognized", SimpleEnum.class, false));
+ }
+
+ @Test
+ public void testUnknownEnumNotIgnoredThroughConf() throws Throwable {
+ intercept(IllegalArgumentException.class, "unrecognized", () ->
+ confWithKey("c, unrecognized")
+ .getEnumSet("key", SimpleEnum.class, false));
+ }
+
+ @Test
+ public void testUnknownEnumIgnored() throws Throwable {
+ assertEnumParse("c, d", SimpleEnum.class, true)
+ .containsExactly(SimpleEnum.c);
+ }
+
+ @Test
+ public void testStarEnum() throws Throwable {
Review Comment:
this test is already present above testEnumParseAll
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestConfigurationHelper.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util;
+
+import java.util.Set;
+
+import org.assertj.core.api.Assertions;
+import org.assertj.core.api.IterableAssert;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static org.apache.hadoop.util.ConfigurationHelper.ERROR_MULTIPLE_ELEMENTS_MATCHING_TO_LOWER_CASE_VALUE;
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+import static org.apache.hadoop.util.ConfigurationHelper.parseEnumSet;
+
+/**
+ * Test for {@link ConfigurationHelper}.
+ */
+public class TestConfigurationHelper extends AbstractHadoopTestBase {
+
+ /**
+ * Simple Enums.
+ * "i" is included for case tests, as it is special in turkey.
+ */
+ private enum SimpleEnum { a, b, c, i }
+
+
+ /**
+ * Special case: an enum with no values.
+ */
+ private enum EmptyEnum { }
+
+ /**
+ * Create assertion about the outcome of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}.
+ * @param valueString value from Configuration
+ * @param enumClass class of enum
+ * @param ignoreUnknown should unknown values be ignored?
+ * @param <E> enum type
+ * @return an assertion on the outcome.
+ * @throws IllegalArgumentException if one of the entries was unknown and ignoreUnknown is false,
+ * or there are two entries in the enum which differ only by case.
+ */
+ private static <E extends Enum<E>> IterableAssert<E> assertEnumParse(
+ final String valueString,
+ final Class<E> enumClass,
+ final boolean ignoreUnknown) {
+ final Set<E> enumSet = parseEnumSet("key", valueString, enumClass, ignoreUnknown);
+ final IterableAssert<E> assertion = Assertions.assertThat(enumSet);
+ return assertion.describedAs("parsed enum set '%s'", valueString);
+ }
+
+
+ /**
+ * Create a configuration with the key {@code key} set to a {@code value}.
+ * @param value value for the key
+ * @return a configuration with only key set.
+ */
+ private Configuration confWithKey(String value) {
+ final Configuration conf = new Configuration(false);
+ conf.set("key", value);
+ return conf;
+ }
+
+ @Test
+ public void testEnumParseAll() throws Throwable {
+ assertEnumParse("*", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEnumParse() throws Throwable {
+ assertEnumParse("a, b,c", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c);
+ }
+
+ @Test
+ public void testEnumCaseIndependence() throws Throwable {
+ assertEnumParse("A, B, C, I", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testEmptyArguments() throws Throwable {
+ assertEnumParse(" ", SimpleEnum.class, false)
+ .isEmpty();
+ }
+
+ @Test
+ public void testUnknownEnumNotIgnored() throws Throwable {
+ intercept(IllegalArgumentException.class, "unrecognized", () ->
+ parseEnumSet("key", "c, unrecognized", SimpleEnum.class, false));
+ }
+
+ @Test
+ public void testUnknownEnumNotIgnoredThroughConf() throws Throwable {
+ intercept(IllegalArgumentException.class, "unrecognized", () ->
+ confWithKey("c, unrecognized")
+ .getEnumSet("key", SimpleEnum.class, false));
+ }
+
+ @Test
+ public void testUnknownEnumIgnored() throws Throwable {
+ assertEnumParse("c, d", SimpleEnum.class, true)
+ .containsExactly(SimpleEnum.c);
+ }
+
+ @Test
+ public void testStarEnum() throws Throwable {
+ assertEnumParse("*", SimpleEnum.class, false)
+ .containsExactly(SimpleEnum.a, SimpleEnum.b, SimpleEnum.c, SimpleEnum.i);
+ }
+
+ @Test
+ public void testUnknownStarEnum() throws Throwable {
Review Comment:
add a test with repeated values "a, b, a". should pass
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ConfigurationHelper.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util;
+
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+
+import static java.util.EnumSet.noneOf;
+import static org.apache.hadoop.util.Preconditions.checkArgument;
+import static org.apache.hadoop.util.StringUtils.getTrimmedStringCollection;
+
+/**
+ * Configuration Helper class to provide advanced configuration parsing.
+ * Private; external code MUST use {@link Configuration} instead
+ */
+@InterfaceAudience.Private
+public final class ConfigurationHelper {
+
+ /**
+ * Error string if there are multiple enum elements which only differ
+ * by case: {@value}.
+ */
+ @VisibleForTesting
+ static final String ERROR_MULTIPLE_ELEMENTS_MATCHING_TO_LOWER_CASE_VALUE =
+ "has multiple elements matching to lower case value";
+
+ private ConfigurationHelper() {
+ }
+
+ /**
+ * Given a comma separated list of enum values,
+ * trim the list, map to enum values in the message (case insensitive)
+ * and return the set.
+ * Special handling of "*" meaning: all values.
+ * @param key key for error messages.
Review Comment:
okay I get it now. maybe mention - configuration key which was used to configure the flags.
I got confused initially because of the UT,
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FlagSet.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.util.ConfigurationHelper;
+import org.apache.hadoop.util.Preconditions;
+
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+
+/**
+ * A set of flags, constructed from a configuration option or from a string,
+ * with the semantics of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}
+ * and implementing {@link StreamCapabilities}.
+ * <p>
+ * Thread safety: there is no synchronization on a mutable {@code FlagSet}.
+ * Once declared immutable, flags cannot be changed, so they
+ * becomes implicitly thread-safe.
+ */
+public final class FlagSet<E extends Enum<E>> implements StreamCapabilities {
+
+ /**
+ * Set of flags.
+ */
+ private final Set<E> flags;
+
+ /**
+ * Is the set immutable?
+ */
+ private final AtomicBoolean immutable = new AtomicBoolean(false);
Review Comment:
Isn't this class supposed to be singleton initialized only once during the file system creation while the configuration is being loaded? And read only flags such that callers can know if a falg is enabled or not? wondering what is the use case of updating the flags?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1668934899
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FlagSet.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.util.ConfigurationHelper;
+import org.apache.hadoop.util.Preconditions;
+
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+
+/**
+ * A set of flags, constructed from a configuration option or from a string,
+ * with the semantics of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}
+ * and implementing {@link StreamCapabilities}.
+ * <p>
+ * Thread safety: there is no synchronization on a mutable {@code FlagSet}.
+ * Once declared immutable, flags cannot be changed, so they
+ * becomes implicitly thread-safe.
+ */
+public final class FlagSet<E extends Enum<E>> implements StreamCapabilities {
+
+ /**
+ * Set of flags.
+ */
+ private final Set<E> flags;
+
+ /**
+ * Is the set immutable?
+ */
+ private final AtomicBoolean immutable = new AtomicBoolean(false);
+
+ /**
+ * Mapping of prefixed flag names to enum values.
+ */
+ private final Map<String, E> namesToValues;
+
+ /**
+ * Create a FlagSet.
+ * @param enumClass class of enum
+ * @param prefix prefix (with trailing ".") for path capabilities probe
+ * @param flags flags. A copy of these are made.
+ */
+ private FlagSet(final Class<E> enumClass,
+ final String prefix,
+ @Nullable final EnumSet<E> flags) {
+
+ this.flags = flags != null
+ ? EnumSet.copyOf(flags)
+ : EnumSet.noneOf(enumClass);
+ this.namesToValues = mapEnumNamesToValues(prefix, enumClass);
+ }
+
+ /**
+ * Get a copy of the flags.
+ * <p>
+ * This is immutable.
+ * @return the flags.
+ */
+ public EnumSet<E> flags() {
+ return EnumSet.copyOf(flags);
+ }
+
+ /**
+ * Probe for the FlagSet being empty.
+ * @return true if there are no flags set.
+ */
+ public boolean isEmpty() {
+ return flags.isEmpty();
+ }
+
+ /**
+ * Is a flag enabled?
+ * @param flag flag to check
+ * @return true if it is in the set of enabled flags.
+ */
+ public boolean enabled(final E flag) {
+ return flags.contains(flag);
+ }
Review Comment:
i started off fully mutable, added immutabilty as an option to avoid all sync issues. you can manipulated it as much as you want and only once made immutable do you need to worry about thread safety etc. lets you buld by copying and updating, from a default value with changes etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2222553969
@mukund-thakur thanks, altered the tests and tried to clarify the immutability.
One final thought: should we have a default list of enum values to set if, when parsing the config string, the string isn't found?
Currently if nothing is found we return the empty set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1663043431
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FlagSet.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.util.ConfigurationHelper;
+import org.apache.hadoop.util.Preconditions;
+
+import static org.apache.hadoop.util.ConfigurationHelper.mapEnumNamesToValues;
+
+/**
+ * A set of flags, constructed from a configuration option or from a string,
+ * with the semantics of
+ * {@link ConfigurationHelper#parseEnumSet(String, String, Class, boolean)}
+ * and implementing {@link StreamCapabilities}.
+ * <p>
+ * Thread safety: there is no synchronization on a mutable {@code FlagSet}.
+ * Once declared immutable, flags cannot be changed, so they
+ * becomes implicitly thread-safe.
+ */
+public final class FlagSet<E extends Enum<E>> implements StreamCapabilities {
+
+ /**
+ * Set of flags.
+ */
+ private final Set<E> flags;
+
+ /**
+ * Is the set immutable?
+ */
+ private final AtomicBoolean immutable = new AtomicBoolean(false);
+
+ /**
+ * Mapping of prefixed flag names to enum values.
+ */
+ private final Map<String, E> namesToValues;
+
+ /**
+ * Create a FlagSet.
+ * @param enumClass class of enum
+ * @param prefix prefix (with trailing ".") for path capabilities probe
+ * @param flags flags. A copy of these are made.
+ */
+ private FlagSet(final Class<E> enumClass,
+ final String prefix,
+ @Nullable final EnumSet<E> flags) {
+
+ this.flags = flags != null
+ ? EnumSet.copyOf(flags)
+ : EnumSet.noneOf(enumClass);
+ this.namesToValues = mapEnumNamesToValues(prefix, enumClass);
+ }
+
+ /**
+ * Get a copy of the flags.
+ * <p>
+ * This is immutable.
+ * @return the flags.
+ */
+ public EnumSet<E> flags() {
+ return EnumSet.copyOf(flags);
+ }
+
+ /**
+ * Probe for the FlagSet being empty.
+ * @return true if there are no flags set.
+ */
+ public boolean isEmpty() {
+ return flags.isEmpty();
+ }
+
+ /**
+ * Is a flag enabled?
+ * @param flag flag to check
+ * @return true if it is in the set of enabled flags.
+ */
+ public boolean enabled(final E flag) {
+ return flags.contains(flag);
+ }
Review Comment:
If we need it immutable, probably it's good case for builder or constructor so maybe keeping the object mutable is also fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#discussion_r1686918836
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -2209,7 +2237,8 @@ public FSDataOutputStreamBuilder createFile(final Path path) {
builder
.create()
.overwrite(true)
- .must(FS_S3A_CREATE_PERFORMANCE, performanceCreation);
+ .must(FS_S3A_CREATE_PERFORMANCE,
+ getPerformanceFlags().enabled(PerformanceFlagEnum.Create));
Review Comment:
I worry about doing things like this which can have unexpected consequences with stuff like mocking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2113717207
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 91m 05s | | trunk passed |
| +1 :green_heart: | compile | 40m 21s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 18s | | trunk passed |
| -1 :x: | mvnsite | 4m 30s | [/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 9m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 163m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 46s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 2m 32s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | compile | 2m 31s | [/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/patch-compile-root.txt) | root in the patch failed. |
| -1 :x: | javac | 2m 31s | [/patch-compile-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/patch-compile-root.txt) | root in the patch failed. |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 5m 21s | | the patch passed |
| -1 :x: | mvnsite | 2m 13s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | javadoc | 2m 21s | [/patch-javadoc-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | shadedclient | 34m 05s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 2m 59s | | The patch does not generate ASF License warnings. |
| | | 355m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 2e28357cc08c 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 28cd50ac8773061969049071eab0c8adca5faa83 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6789/2/console |
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #6789:
URL: https://github.com/apache/hadoop/pull/6789#issuecomment-2161006506
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 39s | | trunk passed |
| +1 :green_heart: | compile | 16m 47s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 15m 21s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 4m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 48s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 13s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 16m 24s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 16m 24s | | the patch passed |
| +1 :green_heart: | compile | 15m 39s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 15m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 11s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/9/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 114 unchanged - 0 fixed = 118 total (was 114) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 4m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 25s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 4s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 237m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6789 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 7aed502e4ad7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 56e980e9263d7ceff0e9b572a1b799a7600df115 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/9/testReport/ |
| Max. process+thread count | 1264 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6789/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org