You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/10 09:09:36 UTC
[GitHub] [hadoop] mukund-thakur opened a new pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
mukund-thakur opened a new pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207
Tested using ap-south-1 bucket. All good apart from known failures.
https://issues.apache.org/jira/browse/HADOOP-17192
https://issues.apache.org/jira/browse/HADOOP-17190
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#discussion_r467906968
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2244,8 +2263,9 @@ public UploadInfo putObject(PutObjectRequest putObjectRequest) {
* not be saved to the metadata store and
* fs.s3a.metadatastore.fail.on.write.error=true
*/
+ @VisibleForTesting
@Retries.OnceRaw("For PUT; post-PUT actions are RetryTranslated")
- PutObjectResult putObjectDirect(PutObjectRequest putObjectRequest)
+ public PutObjectResult putObjectDirect(PutObjectRequest putObjectRequest)
Review comment:
get a WriteOperationHelper instance and invoke via that
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
##########
@@ -137,6 +159,78 @@ public void testListOperations() throws Throwable {
}
}
+ @Test
+ public void testMultiPagesListingPerformanceAndCorrectness() throws Throwable {
+ describe("Check performance and correctness for multi page listing " +
+ "using different listing api");
+ Path dir = path(this.getMethodName());
+ Configuration conf = getConfigurationWithConfiguredBatchSize(10);
+ fs = (S3AFileSystem) FileSystem.get(dir.toUri(), conf);
+ assume("Test is only for raw fs", !fs.hasMetadataStore());
+ fs.create(dir);
+ final InputStream im = new InputStream() {
+ @Override
+ public int read() throws IOException {
+ return -1;
+ }
+ };
+ final int numOfPutRequests = 500;
+ final List<String> originalListOfFiles = new ArrayList<>();
+ List<Callable<PutObjectResult>> putObjectRequests = new ArrayList<>();
+ ExecutorService executorService = Executors.newFixedThreadPool(50);
+ try {
+ for (int i=0; i<numOfPutRequests; i++) {
+ Path file = new Path(dir, String.format("file-%03d", i));
+ originalListOfFiles.add(file.toString());
+ ObjectMetadata om = fs.newObjectMetadata(0L);
+ PutObjectRequest put = new PutObjectRequest(fs.getBucket(),
+ fs.pathToKey(file),
+ im,
+ om);
+ putObjectRequests.add(() -> fs.putObjectDirect(put));
+ }
+ executorService.invokeAll(putObjectRequests);
Review comment:
Add some tracking/logging of how long it took to create all these files.
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##########
@@ -761,29 +777,44 @@ public boolean hasNext() throws IOException {
@Retries.RetryTranslated
public S3ListResult next() throws IOException {
if (firstListing) {
- // on the first listing, don't request more data.
- // Instead just clear the firstListing flag so that it future calls
- // will request new data.
+ // clear the firstListing flag for future calls.
firstListing = false;
+ // Calculating the result of last async list call.
+ objects = awaitFuture(s3ListResultFuture);
+ fetchNextBatchAsyncIfPresent();
} else {
try {
- if (!objects.isTruncated()) {
+ if (objectsPrev!= null && !objectsPrev.isTruncated()) {
// nothing more to request: fail.
throw new NoSuchElementException("No more results in listing of "
- + listPath);
+ + listPath);
}
- // need to request a new set of objects.
+ // Calculating the result of last async list call.
+ objects = awaitFuture(s3ListResultFuture);
+ // Requesting next batch of results.
LOG.debug("[{}], Requesting next {} objects under {}",
Review comment:
move the log into fetchNextBatchAsyncIfPresent
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
##########
@@ -137,6 +159,78 @@ public void testListOperations() throws Throwable {
}
}
+ @Test
+ public void testMultiPagesListingPerformanceAndCorrectness() throws Throwable {
+ describe("Check performance and correctness for multi page listing " +
+ "using different listing api");
+ Path dir = path(this.getMethodName());
Review comment:
methodPath()
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
##########
@@ -137,6 +159,78 @@ public void testListOperations() throws Throwable {
}
}
+ @Test
+ public void testMultiPagesListingPerformanceAndCorrectness() throws Throwable {
+ describe("Check performance and correctness for multi page listing " +
+ "using different listing api");
+ Path dir = path(this.getMethodName());
+ Configuration conf = getConfigurationWithConfiguredBatchSize(10);
+ fs = (S3AFileSystem) FileSystem.get(dir.toUri(), conf);
+ assume("Test is only for raw fs", !fs.hasMetadataStore());
+ fs.create(dir);
+ final InputStream im = new InputStream() {
+ @Override
+ public int read() throws IOException {
+ return -1;
+ }
+ };
+ final int numOfPutRequests = 500;
+ final List<String> originalListOfFiles = new ArrayList<>();
+ List<Callable<PutObjectResult>> putObjectRequests = new ArrayList<>();
+ ExecutorService executorService = Executors.newFixedThreadPool(50);
+ try {
+ for (int i=0; i<numOfPutRequests; i++) {
+ Path file = new Path(dir, String.format("file-%03d", i));
+ originalListOfFiles.add(file.toString());
+ ObjectMetadata om = fs.newObjectMetadata(0L);
+ PutObjectRequest put = new PutObjectRequest(fs.getBucket(),
+ fs.pathToKey(file),
+ im,
+ om);
+ putObjectRequests.add(() -> fs.putObjectDirect(put));
+ }
+ executorService.invokeAll(putObjectRequests);
+ } finally {
+ executorService.shutdown();
+ }
+ RemoteIterator<LocatedFileStatus> locatedFileStatusRemoteIterator =
Review comment:
use a shorter name so the lines after are less wide
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -1956,6 +1956,14 @@ protected S3ListResult listObjects(S3ListRequest request) throws IOException {
}
}
+ protected CompletableFuture<S3ListResult> listObjectsAsync(S3ListRequest request) {
Review comment:
make private unless someone needs to get at these in mockito tests
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
##########
@@ -137,6 +159,78 @@ public void testListOperations() throws Throwable {
}
}
+ @Test
+ public void testMultiPagesListingPerformanceAndCorrectness() throws Throwable {
+ describe("Check performance and correctness for multi page listing " +
+ "using different listing api");
+ Path dir = path(this.getMethodName());
+ Configuration conf = getConfigurationWithConfiguredBatchSize(10);
+ fs = (S3AFileSystem) FileSystem.get(dir.toUri(), conf);
Review comment:
any reason to not use a local variable here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-673382861
Fixed all review comments and re ran the new test. All good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-671252550
Performance result using new test:
`2020-08-10 14:55:31,985 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 500 files using listFiles() api with batch size of 10 including 10ms of processing time for each file: 7,394,348,570 nS
2020-08-10 14:55:42,288 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 500 files using listStatus() api with batch size of 10 including 10ms of processing time for each file: 10,204,574,314 nS`
CC @steveloughran @mehakmeet @bgaborg
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-679102653
...LGTM, let's see what Yetus says
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-679127524
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 28s | trunk passed |
| +1 :green_heart: | compile | 0m 53s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 27s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 15s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | the patch passed |
| +1 :green_heart: | compile | 0m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 21s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 44s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 44s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 84m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2207 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fb536f212292 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 17cd8a1b162 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/3/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur edited a comment on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
mukund-thakur edited a comment on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-671252550
Performance result using new test:
`2020-08-10 15:04:35,966 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 1000 files using listFiles() api with batch size of 10 including 10ms of processing time for each file: 12,039,952,465 nS
2020-08-10 15:04:52,170 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 1000 files using listStatus() api with batch size of 10 including 10ms of processing time for each file: 16,088,964,963 nS`
We can see an improvement of 4s with these configs.
Result when the same test is run in trunk having sync listing.
`2020-08-10 15:10:03,815 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 1000 files using listFiles() api with batch size of 10 including 10ms of processing time for each file: 16,722,638,860 nS
2020-08-10 15:10:20,293 [JUnit-testMultiPagesListingPerformanceAndCorrectness] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1847)) - Duration of listing 1000 files using listStatus() api with batch size of 10 including 10ms of processing time for each file: 16,364,577,964 nS`
It is evident from the logs that without the improvements, listStatus and listFiles took same time.
CC @steveloughran @mehakmeet @bgaborg
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#discussion_r469837444
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADirectoryPerformance.java
##########
@@ -137,6 +159,78 @@ public void testListOperations() throws Throwable {
}
}
+ @Test
+ public void testMultiPagesListingPerformanceAndCorrectness() throws Throwable {
+ describe("Check performance and correctness for multi page listing " +
+ "using different listing api");
+ Path dir = path(this.getMethodName());
+ Configuration conf = getConfigurationWithConfiguredBatchSize(10);
+ fs = (S3AFileSystem) FileSystem.get(dir.toUri(), conf);
Review comment:
Initially there were two tests so I thought of reusing the class level variable and closing the fs in teardown.
Since there is just one test, I have moved to a local variable and closing using try-with-resources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-673417118
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 37s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 4s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 4s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | the patch passed |
| +1 :green_heart: | compile | 0m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 35s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 6 new + 12 unchanged - 0 fixed = 18 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 75m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2207 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b39672fe98f7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e592ec5f8bf |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2207/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2207: HADOOP-17074 Optimise s3a Listing to be fully asynchronous.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2207:
URL: https://github.com/apache/hadoop/pull/2207#issuecomment-673484752
LGTM. +1 pending the changes needed to get checkstyle to be (mostly) quiet
nice bit of work here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org