You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/19 03:42:29 UTC
[GitHub] [hadoop] bilaharith opened a new pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
bilaharith opened a new pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179
Making the AbfsOutputStream maxConcurrentRequests and the maximum size to which the threadpool queue can grow up to.
**Driver test results using accounts in Central India**
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
**Client credentials
Account with HNS Support**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 75
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
**Account without HNS support**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 248
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
**Accesskey
Account with HNS Support**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[ERROR] ITestGetNameSpaceEnabled.testFailedRequestWhenCredentialsNotCorrect:160->AbstractAbfsIntegrationTest.getFileSystem:254 » KeyProvider
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 2, Skipped: 42
WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
**Account without HNS support**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith edited a comment on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith edited a comment on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675833767
> Test results posted have failures. Whats the plan to handle them ?
PFB the JIRAs to track the same.
https://issues.apache.org/jira/browse/HADOOP-17160
https://issues.apache.org/jira/browse/HADOOP-17149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#discussion_r482685347
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java
##########
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+
+/**
+ * Test create operation.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ private static final Path TEST_FILE_PATH = new Path("testfile");
+
+ public ITestAbfsOutputStream() throws Exception {
+ super();
+ }
+
+ @Test
+ public void testMaxRequestsAndQueueCapacityDefaults() throws Exception {
+ Configuration conf = getRawConfiguration();
+ final AzureBlobFileSystem fs = getFileSystem(conf);
+ FSDataOutputStream out = fs.create(TEST_FILE_PATH);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-683107211
> Here is my current view of this patch
>
> * as a quick workaround for scale-up issues in the existing code -it works
> * as the long term design for something which delivers great performance when scaled up as well as down -it's not suitable
> * the code you need is already there
>
> If you look at the S3A code, we have a single shared thread pool, which is based on another ASF project: https://github.com/apache/incubator-retired-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
>
> ```java
> private ListeningExecutorService boundedThreadPool;
>
> // and in initialize()
> int totalTasks = intOption(conf,
> MAX_TOTAL_TASKS, DEFAULT_MAX_TOTAL_TASKS, 1);
> long keepAliveTime = longOption(conf, KEEPALIVE_TIME,
> DEFAULT_KEEPALIVE_TIME, 0);
> boundedThreadPool = BlockingThreadPoolExecutorService.newInstance(
> maxThreads,
> maxThreads + totalTasks,
> keepAliveTime, TimeUnit.SECONDS,
> "s3a-transfer-shared");
>
> // default value is 4
> blockOutputActiveBlocks = intOption(conf,
> FAST_UPLOAD_ACTIVE_BLOCKS, DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS, 1);
> ```
>
> When we create the output stream, we create a new output stream which, although it uses the thread pool, limits the #of blocks each worker can queue for upload.
>
> ```java
> ...
> new S3ABlockOutputStream(this,
> destKey,
> new SemaphoredDelegatingExecutor(boundedThreadPool,
> blockOutputActiveBlocks, true),
> progress,
> partSize,
> blockFactory,
> statisticsContext.newOutputStreamStatistics(),
> getWriteOperationHelper(),
> putTracker),
> ```
>
> This gives us
>
> * low latency for output stream launch (shared pool)
> * observability into total pool size/load (could make it a gauge...)
> * semaphore to stop a single worker from overloading the system
> * A thread pool to use for other operations.
>
> Weaknesses
>
> * I think we overuse the blocking thread pool in other places (openFile(),..)& should do a review to make sure we are using the unbounded one more. I managed to create a deadlock once.
> * If you set the active block option "fs.s3a.fast.upload.active.blocks" too big then you can still OOM if you buffer in RAM. Default S3A block buffer is on disk to avoid this problem. Now people only complain about us not cleaning up temp disk space when spark workers are killed
> * You still need to decide that blocking thread pool size (default = 32).
>
> I think the S3A code
>
> 1. should lift that * cores option and say "if the pool size < 0 then we use it as a multiplier for cores"", so -4 would mean "4 * cores". Maybe too late to do this in a backwards compatible way though now, unless I add a new option, deprecate the old one, etc.
> 2. Add pool size as a gauge. If we could also (somehow) include #of pending blocks across all semaphored executors then you'd get a view of how many threads were being held up.
> 3. And we could add "time to wait for a thread" as a metric too. Actually, I like that...if we can tie to a stream we can tie to a task, and hence to a job. Tricky though. Let me see what I can do there with the IOStatistics PR in front of me.
>
> for point #3: just done it for IOStatistics: https://github.com/apache/hadoop/blob/f5efa4b27536a9e266d9dc06cd3a1e11ded3bfd3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java
>
> If you pass in a duration factory then the executor will measure the time to acquire a thread before the actual execution. This is nice to know when trying to answer the "why so slow?" question -as it will either show a problem or show where not to look. of course -doesn't just mean 'not enough executors' as 'why are all the submitted operations taking so long?' could be a sign of network problems.
>
> To summarise then
>
> * just go for the same thread pool in Store and semaphored wrapper around this passed in to each output stream
> * choose some pool size, with the idea of a negative value == per core making it easy to do a scalable config
> * plan for wiring up statistics collection in the near future.
Thanks for the inputs. There is already work in progress for a long term fix. But I will try these suggestions before raising a PR for the same.
Currently I would like this change as a quick workaround/interim fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-680270262
> We are working on a similar approach. The same needs extensive tests to ensure the perf side is fine. Till the fix is available this can be used to control the memory consumption. This has been verified with a customer as well
The s3a queue design is pretty well tested
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675237006
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 1s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 14m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 72m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 63a10e73583a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b367942fe49 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/5/artifact/out/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/5/testReport/ |
| Max. process+thread count | 402 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-683139224
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 31s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 27s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 41s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 12s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 31s | The patch generated 3 ASF License warnings. |
| | | 78m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 01aa91d20d02 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 44542863f41 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/8/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/8/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/8/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#discussion_r479751801
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -52,6 +52,8 @@
public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF = "fs.azure.oauth.token.fetch.retry.delta.backoff";
// Read and write buffer sizes defined by the user
+ public static final String AZURE_WRITE_MAX_CONCURRENT_REQUESTS = "fs.azure.write.max.concurrent.requests";
Review comment:
use `experimental` in name to show they are exactly that
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
##########
@@ -796,6 +796,18 @@ will be -1. To disable readaheads, set this value to 0. If your workload is
doing only random reads (non-sequential) or you are seeing throttling, you
may try setting this value to 0.
+To run under limited memory situations configure the following.
Review comment:
make clear: or when doing many writes in same process (bulk uploads, hive LLAP/spark with many workers)
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java
##########
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys;
+
+/**
+ * Test create operation.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ private static final Path TEST_FILE_PATH = new Path("testfile");
+
+ public ITestAbfsOutputStream() throws Exception {
+ super();
+ }
+
+ @Test
+ public void testMaxRequestsAndQueueCapacityDefaults() throws Exception {
+ Configuration conf = getRawConfiguration();
+ final AzureBlobFileSystem fs = getFileSystem(conf);
+ FSDataOutputStream out = fs.create(TEST_FILE_PATH);
Review comment:
try with resources, always
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675395752
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 72m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux fffe87c8269a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fefacf2578e |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/testReport/ |
| Max. process+thread count | 401 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-676290053
**Driver test results using accounts in Canary region**
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
**Account with HNS Support**
**SharedKey**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 32
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
**OAuth**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 453, Failures: 0, Errors: 1, Skipped: 74
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
**Account without HNS support**
**SharedKey**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 453, Failures: 0, Errors: 1, Skipped: 245
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
**OAuth**
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 453, Failures: 0, Errors: 1, Skipped: 249
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
Reported the error on the following JIRA.
https://issues.apache.org/jira/browse/HADOOP-17158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-666927562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-686411091
LGTM, +1 *pending you make clear in the option names and docs that this is experimental*
> These configs are tested on prod environments.
> The same can remain as a means to control the resource usage. With the internal discussions we had we would like to keep the same this way.
We all have this problem, we all want to get a fix in.
My point of view is that a shared thread pool with queue managed to stop one single output stream using up all the capacity is that the correct solution. I base this on S3ABlockOutputStream, whose pool class is in hadoop-common.
I also understand, why a simple "let's do this right now" fix can address a situation will rapidly before the ABFS streams switch to `BlockingThreadPoolExecutorService`.
However, it is precisely because we know it is an interim fix that I want all the options to have 'experimental' in their name. That way, when they get removed, people won't get upset that the options they were using have gone away.
I recognise that you are shipping with this fix, and that you have cluster configurations which use them. However, it is long-standing policy in the project which is "the ASF project must not have its decisions determined by the fact that someone has already shipped a feature in their own branch". That's important: we have all shipped fixes early, and then had to deal with catching up with production releases. I believe the HDFS IPC wire format change between Hadoop 2.0.205 (used in CDH) and Hadoop 2.2.0 was the most controversial here as it was actual protocol incompatibility. The situation here is minor in comparision.
_Anyone is free to ship a hadoop build with their own changes, but that cannot be used as a veto on changes in the open source codebase itself_
This makes sense, when you think of it.
The good news, `Configuration.addDeprecations()` lets you provide an automated way to map from deprecated values (your current set of options) to ones with "experimental" in the name, such as here (S3ATestUtils)
```
private static void addDeprecatedKeys() {
Configuration.DeprecationDelta[] deltas = {
// STS endpoint configuration option
new Configuration.DeprecationDelta(
S3ATestConstants.TEST_STS_ENDPOINT,
ASSUMED_ROLE_STS_ENDPOINT)
};
Configuration.addDeprecations(deltas);
Configuration.reloadExistingConfigurations();
}
static {
addDeprecatedKeys();
}
```
If the old option is used, .the user will see a message logged @ info (unless they have turned off the deprecation log), and the value is remapped to the new one.
```
log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN
```
So, please use experimental in the name, it gives us freedom to come up with a good design for how to do block upload buffering/pooling without having any future design decisions constrained by this intermediate patch.
thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-679307318
> Prefer you use the strategy outlined in HADOOP-17195. Rather than pool settings for each output stream, the store has a single shared pool with semaphores to limit #of active entries per output stream. org.apache.hadoop.util.SemaphoredDelegatingExecutor does this for you. This ensures that when there is a low #of output streams, they good upload performance, but in heavy use then it throttles back.
We are working on a similar approach. The same needs extensive tests to ensure the perf side is fine. Till the fix is available this can be used to control the memory consumption. This has been verified with a customer as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675831394
Test results posted have failures. Whats the plan to handle them ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-680982644
Here is my current view of this patch
* as a quick workaround for scale-up issues in the existing code -it works
* as the long term design for something which delivers great performance when scaled up as well as down -it's not suitable
* the code you need is already there
If you look at the S3A code, we have a single shared thread pool, which is based on another ASF project: https://github.com/apache/incubator-retired-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
```java
private ListeningExecutorService boundedThreadPool;
// and in initialize()
int totalTasks = intOption(conf,
MAX_TOTAL_TASKS, DEFAULT_MAX_TOTAL_TASKS, 1);
long keepAliveTime = longOption(conf, KEEPALIVE_TIME,
DEFAULT_KEEPALIVE_TIME, 0);
boundedThreadPool = BlockingThreadPoolExecutorService.newInstance(
maxThreads,
maxThreads + totalTasks,
keepAliveTime, TimeUnit.SECONDS,
"s3a-transfer-shared");
// default value is 4
blockOutputActiveBlocks = intOption(conf,
FAST_UPLOAD_ACTIVE_BLOCKS, DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS, 1);
```
When we create the output stream, we create a new output stream which, although it uses the thread pool, limits the #of blocks each worker can queue for upload.
```java
...
new S3ABlockOutputStream(this,
destKey,
new SemaphoredDelegatingExecutor(boundedThreadPool,
blockOutputActiveBlocks, true),
progress,
partSize,
blockFactory,
statisticsContext.newOutputStreamStatistics(),
getWriteOperationHelper(),
putTracker),
```
This gives us
* low latency for output stream launch (shared pool)
* observability into total pool size/load (could make it a gauge...)
* semaphore to stop a single worker from overloading the system
* A thread pool to use for other operations.
Weaknesses
* I think we overuse the blocking thread pool in other places (openFile(),..)& should do a review to make sure we are using the unbounded one more. I managed to create a deadlock once.
* If you set the active block option "fs.s3a.fast.upload.active.blocks" too big then you can still OOM if you buffer in RAM. Default S3A block buffer is on disk to avoid this problem. Now people only complain about us not cleaning up temp disk space when spark workers are killed
* You still need to decide that blocking thread pool size (default = 32).
I think the S3A code
1. should lift that * cores option and say "if the pool size < 0 then we use it as a multiplier for cores"", so -4 would mean "4 * cores". Maybe too late to do this in a backwards compatible way though now, unless I add a new option, deprecate the old one, etc.
1. Add pool size as a gauge. If we could also (somehow) include #of pending blocks across all semaphored executors then you'd get a view of how many threads were being held up.
1. And we could add "time to wait for a thread" as a metric too. Actually, I like that...if we can tie to a stream we can tie to a task, and hence to a job. Tricky though. Let me see what I can do there with the IOStatistics PR in front of me.
for point #3: just done it for IOStatistics: https://github.com/apache/hadoop/blob/f5efa4b27536a9e266d9dc06cd3a1e11ded3bfd3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java
If you pass in a duration factory then the executor will measure the time to acquire a thread before the actual execution. This is nice to know when trying to answer the "why so slow?" question -as it will either show a problem or show where not to look. of course -doesn't just mean 'not enough executors' as 'why are all the submitted operations taking so long?' could be a sign of network problems.
To summarise then
- just go for the same thread pool in Store and semaphored wrapper around this passed in to each output stream
- choose some pool size, with the idea of a negative value == per core making it easy to do a scalable config
- plan for wiring up statistics collection in the near future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-678634936
Prefer you use the strategy outlined in HADOOP-17195. Rather than pool settings for each output stream, the store has a single shared pool with semaphores to limit #of active entries per output stream. org.apache.hadoop.util.SemaphoredDelegatingExecutor does this for you. This ensures that when there is a low #of output streams, they good upload performance, but in heavy use then it throttles back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#discussion_r482685168
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -52,6 +52,8 @@
public static final String AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF = "fs.azure.oauth.token.fetch.retry.delta.backoff";
// Read and write buffer sizes defined by the user
+ public static final String AZURE_WRITE_MAX_CONCURRENT_REQUESTS = "fs.azure.write.max.concurrent.requests";
Review comment:
These configs are tested on prod environments. The same can remain as a means to controle the resource usage. With the internal discussions we had we would like to keep the same this way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-676079455
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 43s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 44s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 79m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 6302a6b7a17e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 82ec28f4421 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/testReport/ |
| Max. process+thread count | 421 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-686248502
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 4s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 29s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 53s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 15m 30s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 14s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 74m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 240462ef6ff8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0207f5cf461 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/9/artifact/out/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/9/testReport/ |
| Max. process+thread count | 357 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/9/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675015282
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 70m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 2a5aca16f1f9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 092bfe7c8e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/4/artifact/out/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/4/testReport/ |
| Max. process+thread count | 423 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-666927562
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 36s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 31s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 46s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 27s | hadoop-azure in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 13m 47s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s | hadoop-azure in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 61m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 983e7ffbf1ba 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 05b3337a460 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/artifact/out/whitespace-eol.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675833767
> Test results posted have failures. Whats the plan to handle them ?
PFB the JIRAs to track the same.
https://issues.apache.org/jira/browse/HADOOP-17160
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-675395752
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 72m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux fffe87c8269a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fefacf2578e |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/testReport/ |
| Max. process+thread count | 401 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/6/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179#issuecomment-676079455
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 43s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 44s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 79m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 6302a6b7a17e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 82ec28f4421 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/testReport/ |
| Max. process+thread count | 421 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2179/7/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith closed pull request #2179: HADOOP-17166. ABFS: making max concurrent requests and max requests that can be que…
Posted by GitBox <gi...@apache.org>.
bilaharith closed pull request #2179:
URL: https://github.com/apache/hadoop/pull/2179
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org