You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/07/27 14:14:31 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
steveloughran opened a new pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240
Changes
* subclass can declare whether or not -direct should be default
* all tests then switch to that
* major cut back on default depth/width of directories
* if you set the file size of "scale.test.distcp.file.size.kb" to 0
the large file test case is skipped.
* aggressive cutback on all needless mkdir() calls.
S3A suite
* declares -direct always
ABFS suites
* deletes superfluous ITestAbfsFileSystemContractSecureDistCp
* uses abfs scale test timeout
* only runs with -Dscale
All these changes bring execution time down from 4 min to 2 min against each store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887549690
tested
abfs: azure cardiff
s3: aws london
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888452283
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 24m 52s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 53s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 17s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 56s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | | the patch passed |
| +1 :green_heart: | compile | 23m 34s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 7s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 21m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 5s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 43s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 20m 1s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 15s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 254m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 572cbaae0ad3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / de1cfdb43770696929466725e839ffbe7c14883d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/4/testReport/ |
| Max. process+thread count | 2341 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r678038991
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -659,6 +683,41 @@ private int getTotalFiles() {
return totalFiles;
}
+ /**
+ * Override point: should direct write always be used?
+ * false by default; enable for stores where rename is slow.
+ * @return true if direct write should be used in all tests.
+ */
+ protected boolean directWriteAlways() {
Review comment:
nit: method name contains Always but default is false, a bit ambiguous no?
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -758,7 +828,7 @@ public void testDistCpWithUpdateExistFile() throws Exception {
verifyPathExists(remoteFS, "", source);
verifyPathExists(localFS, "", dest);
DistCpTestUtils.assertRunDistCp(DistCpConstants.SUCCESS, source.toString(),
- dest.toString(), "-delete -update", conf);
+ dest.toString(), "-delete -update" + getDefaultCLIOptions(), conf);
Review comment:
nit : options can be refactored like it is done above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888216640
Will add to release notes the fact you can turn off the large file uploads through
```xml
<property>
<name>scale.test.distcp.file.size.kb</name>
<value>0</value>
</property>
```
This is useful for anyone doing testing from home on a network with slower upload speeds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r678208587
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -532,13 +549,15 @@ private Path distCpDeepDirectoryStructure(FileSystem srcFS,
*/
private void largeFiles(FileSystem srcFS, Path srcDir, FileSystem dstFS,
Path dstDir) throws Exception {
+ int fileSizeKb = conf.getInt(SCALE_TEST_DISTCP_FILE_SIZE_KB,
+ DEFAULT_DISTCP_SIZE_KB);
+ if (fileSizeKb < 1) {
+ skip("File size in " + SCALE_TEST_DISTCP_FILE_SIZE_KB + " too small");
Review comment:
Nit: maybe, we should say to "make fileSize in `SCALE_TEST_DISTCP_FILE_SIZE_KB` be greater than or equal to 1" or "File Size in `SCALE_TEST_DISTCP_FILE_SIZE_KB` smaller than 1"
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -612,6 +634,9 @@ public void testDirectWrite() throws Exception {
@Test
public void testNonDirectWrite() throws Exception {
+ if (directWriteAlways()) {
+ skip("not needed");
Review comment:
Nit: maybe move below `describe()`, or mention in skip message what is being skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-890243250
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 40s | | trunk passed |
| +1 :green_heart: | compile | 27m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 10s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 22s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 4s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 37s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 4s | | the patch passed |
| +1 :green_heart: | compile | 28m 41s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 28m 41s | | the patch passed |
| +1 :green_heart: | compile | 19m 29s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 9s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 20m 16s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 2m 23s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 17s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 268m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e83c31a8977e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6e0e4d90e9bc3864c73df95376f3af5056eb044b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/5/testReport/ |
| Max. process+thread count | 1559 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888247513
Latest release
* Address review comments
* log IOStats after each test case.
Important: as the cached FS retains statistics, the numbers
get bigger over time.
* HDFS test is now reinstated, as we've identified that most
of its long execution time is from the large file upload/download
suites. Disable them and its execution time drops from 4m to 30s,
which means it can then be used to make sure the contract suite
is consistent between HDFS and the object stores.
IOStats of full suite against S3 london (1:43s)
```
2021-07-28 12:40:48,632 [setup] INFO statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_executor_acquired=47)
(action_http_get_request=38)
(action_http_head_request=111)
(audit_request_execution=420)
(audit_span_creation=483)
(directories_created=38)
(directories_deleted=1)
(fake_directories_deleted=485)
(files_copied=2)
(files_copied_bytes=264)
(files_created=47)
(files_deleted=48)
(ignored_errors=14)
(object_bulk_delete_request=88)
(object_copy_requests=2)
(object_delete_objects=534)
(object_delete_request=5)
(object_list_request=89)
(object_metadata_request=111)
(object_put_bytes=18880752)
(object_put_request=85)
(object_put_request_completed=85)
(op_create=47)
(op_delete=14)
(op_exists=13)
(op_exists.failures=3)
(op_get_file_status=194)
(op_get_file_status.failures=44)
(op_glob_status=25)
(op_is_file=1)
(op_list_files=9)
(op_list_status=60)
(op_mkdirs=64)
(op_open=39)
(op_rename=2)
(s3guard_metadatastore_initialization=1)
(s3guard_metadatastore_put_path_request=103)
(s3guard_metadatastore_record_deletes=2)
(s3guard_metadatastore_record_reads=1473)
(s3guard_metadatastore_record_writes=350)
(store_io_request=422)
(stream_read_bytes=18878052)
(stream_read_close_operations=39)
(stream_read_closed=38)
(stream_read_opened=38)
(stream_read_operations=2742)
(stream_read_operations_incomplete=1639)
(stream_read_seek_policy_changed=39)
(stream_read_total_bytes=18878052)
(stream_write_block_uploads=47)
(stream_write_bytes=18880752)
(stream_write_total_data=37761504));
gauges=((stream_write_block_uploads_pending=47));
minimums=((action_executor_acquired.min=0)
(action_http_get_request.min=31)
(action_http_head_request.min=22)
(object_bulk_delete_request.min=45)
(object_delete_request.min=34)
(object_list_request.min=28)
(object_put_request.min=42)
(op_create.min=16)
(op_delete.min=53)
(op_exists.failures.min=16)
(op_exists.min=15)
(op_get_file_status.failures.min=16)
(op_get_file_status.min=15)
(op_glob_status.min=15)
(op_is_file.min=43)
(op_list_files.min=176)
(op_list_status.min=64)
(op_mkdirs.min=16)
(op_rename.min=967));
maximums=((action_executor_acquired.max=0)
(action_http_get_request.max=123)
(action_http_head_request.max=317)
(object_bulk_delete_request.max=384)
(object_delete_request.max=91)
(object_list_request.max=202)
(object_put_request.max=2083)
(op_create.max=129)
(op_delete.max=2196)
(op_exists.failures.max=45)
(op_exists.max=43)
(op_get_file_status.failures.max=29)
(op_get_file_status.max=341)
(op_glob_status.max=192)
(op_is_file.max=43)
(op_list_files.max=589)
(op_list_status.max=260)
(op_mkdirs.max=729)
(op_rename.max=1199));
means=((action_executor_acquired.mean=(samples=47, sum=0, mean=0.0000))
(action_http_get_request.mean=(samples=38, sum=1490, mean=39.2105))
(action_http_head_request.mean=(samples=111, sum=4311, mean=38.8378))
(object_bulk_delete_request.mean=(samples=88, sum=12810, mean=145.5682))
(object_delete_request.mean=(samples=5, sum=260, mean=52.0000))
(object_list_request.mean=(samples=89, sum=4988, mean=56.0449))
(object_put_request.mean=(samples=85, sum=17463, mean=205.4471))
(op_create.mean=(samples=47, sum=1160, mean=24.6809))
(op_delete.mean=(samples=14, sum=11257, mean=804.0714))
(op_exists.failures.mean=(samples=3, sum=80, mean=26.6667))
(op_exists.mean=(samples=10, sum=250, mean=25.0000))
(op_get_file_status.failures.mean=(samples=44, sum=876, mean=19.9091))
(op_get_file_status.mean=(samples=150, sum=6404, mean=42.6933))
(op_glob_status.mean=(samples=25, sum=1826, mean=73.0400))
(op_is_file.mean=(samples=1, sum=43, mean=43.0000))
(op_list_files.mean=(samples=9, sum=3218, mean=357.5556))
(op_list_status.mean=(samples=60, sum=7084, mean=118.0667))
(op_mkdirs.mean=(samples=64, sum=15375, mean=240.2344))
(op_rename.mean=(samples=2, sum=2166, mean=1083.0000)));
```
IOStats of full suite against AWS cardiff (1:28). That region is about 30 miles away from here, though I don't know how cables are routed across the Bristol Channel; it'll probably be a bit longer. In contrast, london will be 100-120 miles away, so latency always going to be a bit higher there.
```
2021-07-28 12:43:57,686 INFO [setup]: statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_http_delete_request=48)
(action_http_delete_request.failures=34)
(action_http_get_request=161)
(action_http_head_request=333)
(action_http_head_request.failures=79)
(action_http_put_request=237)
(bytes_received=18878316)
(bytes_sent=18881016)
(connections_made=779)
(directories_created=71)
(files_created=49)
(get_responses=779)
(op_create=49)
(op_delete=48)
(op_exists=53)
(op_get_file_status=291)
(op_list_status=107)
(op_mkdirs=71)
(op_open=41)
(op_rename=22)
(send_requests=237));
gauges=();
minimums=((action_http_delete_request.failures.min=21)
(action_http_delete_request.min=31)
(action_http_get_request.min=21)
(action_http_head_request.failures.min=20)
(action_http_head_request.min=19)
(action_http_put_request.min=23));
maximums=((action_http_delete_request.failures.max=332)
(action_http_delete_request.max=146)
(action_http_get_request.max=2193)
(action_http_head_request.failures.max=262)
(action_http_head_request.max=822)
(action_http_put_request.max=3370));
means=((action_http_delete_request.failures.mean=(samples=34, sum=1901, mean=55.9118))
(action_http_delete_request.mean=(samples=14, sum=744, mean=53.1429))
(action_http_get_request.mean=(samples=161, sum=15025, mean=93.3230))
(action_http_head_request.failures.mean=(samples=79, sum=3668, mean=46.4304))
(action_http_head_request.mean=(samples=254, sum=9391, mean=36.9724))
(action_http_put_request.mean=(samples=237, sum=27099, mean=114.3418)));
```
ABFS is collecting many fewer stats, we really need
* duration of all FS API calls
* LIST performance numbers should be split from GET calls, which they currently aren't.
Really interesting there that HEAD -> 404 has a mean time of 46ms; HEAD to 200 of 36 millis.
There's always going to be some probes before creating files, dirs, so that negative cost
is going to be visible for those operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r677993164
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java
##########
@@ -59,42 +55,35 @@ protected Configuration createConfiguration() {
return newConf;
}
+ @Override
+ protected boolean directWriteAlways() {
+ return true;
+ }
+
@Override
protected S3AContract createContract(Configuration conf) {
return new S3AContract(conf);
}
- /**
- * Always inject the delay path in, so if the destination is inconsistent,
- * and uses this key, inconsistency triggered.
- * @param filepath path string in
- * @return path on the remote FS for distcp
- * @throws IOException IO failure
- */
@Override
- protected Path path(final String filepath) throws IOException {
- Path path = super.path(filepath);
- return new Path(path, FailureInjectionPolicy.DEFAULT_DELAY_KEY_SUBSTRING);
+ public void testDistCpWithIterator() throws Exception {
+ final long renames = getRenameOperationCount();
+ super.testDistCpWithIterator();
+ assertEquals("Expected no renames for a direct write distcp",
+ getRenameOperationCount(),
+ renames);
}
@Override
public void testDirectWrite() throws Exception {
- final long renames = getRenameOperationCount();
- super.testDirectWrite();
- assertEquals("Expected no renames for a direct write distcp", 0L,
- getRenameOperationCount() - renames);
+ skip("Not needed as all tests are direct by default");
}
@Override
public void testNonDirectWrite() throws Exception {
+// ContractTestUtils.skip("disabled for peformance reasons");
Review comment:
can delete this line itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888217775
optional OptionalTestHDFSContractDistCp runs fine from IDE'; 4 minutes. Probably mini DFSCluster overheads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887719506
Reviews invited from all, including @mukund-thakur @mehakmeet @ayushtkn @sumangala-patki @bilaharith
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887737121
full azure test run with parallel dir patch against cardiff, ` -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale`
This test is now a -Dscale option as even though its now a lot faster, it's still slow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887799435
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 5s | | trunk passed |
| +1 :green_heart: | compile | 27m 24s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 50s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 56s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 27m 14s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 27m 14s | | the patch passed |
| +1 :green_heart: | compile | 26m 2s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 26m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 52s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 29s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 8m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 49m 38s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 27s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not generate ASF License warnings. |
| | | 314m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 319ac07c320c 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5b4be629db0e7db624ebc169cdbd41ec37ead9c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887887092
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 40s | | trunk passed |
| +1 :green_heart: | compile | 23m 17s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 52s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 23s | | the patch passed |
| +1 :green_heart: | compile | 19m 42s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 1s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 12s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 26m 27s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 250m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0840ba8d81b5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c9d528468f03424dfa16650c0b67b71651c4bf6 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/testReport/ |
| Max. process+thread count | 2645 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran edited a comment on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-888217775
optional OptionalTestHDFSContractDistCp runs fine from IDE'; 4 minutes.
All the performance issues of the HDFS contract are related to the large file tests at 2 minutes for one, 1:30 for the other. If those tests were turned off then they could always be run, which would give us better regression checks on the object store behaviours matching HDFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r680145818
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -659,6 +683,41 @@ private int getTotalFiles() {
return totalFiles;
}
+ /**
+ * Override point: should direct write always be used?
+ * false by default; enable for stores where rename is slow.
+ * @return true if direct write should be used in all tests.
+ */
+ protected boolean directWriteAlways() {
Review comment:
changed to shouldUseDirectWrite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887799435
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 5s | | trunk passed |
| +1 :green_heart: | compile | 27m 24s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 23m 50s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 56s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 27m 14s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 27m 14s | | the patch passed |
| +1 :green_heart: | compile | 26m 2s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 26m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 52s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 29s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 8m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 49m 38s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 27s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not generate ASF License warnings. |
| | | 314m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 319ac07c320c 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5b4be629db0e7db624ebc169cdbd41ec37ead9c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r678227498
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -532,13 +549,15 @@ private Path distCpDeepDirectoryStructure(FileSystem srcFS,
*/
private void largeFiles(FileSystem srcFS, Path srcDir, FileSystem dstFS,
Path dstDir) throws Exception {
+ int fileSizeKb = conf.getInt(SCALE_TEST_DISTCP_FILE_SIZE_KB,
+ DEFAULT_DISTCP_SIZE_KB);
+ if (fileSizeKb < 1) {
+ skip("File size in " + SCALE_TEST_DISTCP_FILE_SIZE_KB + " too small");
Review comment:
now
"File size in " + SCALE_TEST_DISTCP_FILE_SIZE_KB + " is zero
It's not bug, just a fact...for HDFS suite it'll be zero by default now
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -612,6 +634,9 @@ public void testDirectWrite() throws Exception {
@Test
public void testNonDirectWrite() throws Exception {
+ if (directWriteAlways()) {
+ skip("not needed");
Review comment:
actually, it should be in the previous test. So moved up. thanks for drawing my attention to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#discussion_r678208290
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -758,7 +828,7 @@ public void testDistCpWithUpdateExistFile() throws Exception {
verifyPathExists(remoteFS, "", source);
verifyPathExists(localFS, "", dest);
DistCpTestUtils.assertRunDistCp(DistCpConstants.SUCCESS, source.toString(),
- dest.toString(), "-delete -update", conf);
+ dest.toString(), "-delete -update" + getDefaultCLIOptions(), conf);
Review comment:
did think about it, but I also felt that it might be prudent to have tests with the CLI to parse, so make sure that -direct is handled there properly. We've been hit in the past by some failures with s3guard and the FS shell API because we were always working at the API level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3240: HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240#issuecomment-887887092
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 40s | | trunk passed |
| +1 :green_heart: | compile | 23m 17s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 52s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 23s | | the patch passed |
| +1 :green_heart: | compile | 19m 42s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 1s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 54s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 12s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 26m 27s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | unit | 2m 26s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 250m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0840ba8d81b5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c9d528468f03424dfa16650c0b67b71651c4bf6 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/testReport/ |
| Max. process+thread count | 2645 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3240/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org