You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/07/26 21:22:40 UTC
[GitHub] [hadoop] steveloughran opened a new pull request, #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
steveloughran opened a new pull request, #4637:
URL: https://github.com/apache/hadoop/pull/4637
Change-Id: If0c8ea825f63a8925cdccb742e9ff5b378529a52
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
object store test in progress. first run had two failures I'd not see before
* landsat test timeout
* oom in a test writing data. I'd set my connector to use bytebuffer for block buffering and it ran out. But why now? SDK update *or* just upload delays triggering it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1197127350
Okay looks good. running the tests once complete will give a +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1198052208
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 2s | | trunk passed |
| +1 :green_heart: | compile | 24m 50s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 13s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 2m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 8s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 117m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 25m 15s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 25m 15s | | the patch passed |
| +1 :green_heart: | compile | 22m 42s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 2m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 30m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 3m 20s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 35s | | The patch does not generate ASF License warnings. |
| | | 206m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4637 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint |
| uname | Linux 58a8f285c8fd 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e67b4e8885b5ae8a0cb9092a6850c09ade99adbb |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/testReport/ |
| Max. process+thread count | 698 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a diff in pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on code in PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#discussion_r931496072
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:
##########
@@ -1274,6 +1282,21 @@ bin/hadoop s3guard markers -clean -verbose $BUCKET
# expect success and exit code of 0
bin/hadoop s3guard markers -audit -verbose $BUCKET
+# ---------------------------------------------------
+# Copy to from local
+# ---------------------------------------------------
+
+time bin/hadoop fs -copyFromLocal -t 10 share/hadoop/tools/lib/*aws*jar $BUCKET/
+
+# expect the iostatistics object_list_request value to be O(directories)
+bin/hadoop fs -ls -R $BUCKET/
+
+# expect the iostatistics object_list_request and op_get_content_summary values to be 1
Review Comment:
Yes there is a extra space here at the end as pointed by Yetus,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196566368
Test failures
```
[ERROR] Errors:
[ERROR] ITestS3AFailureHandling.testMultiObjectDeleteLargeNumKeys » OutOfMemory Direct...
[ERROR] ITestS3ADeleteFilesOneByOne.testBulkRenameAndDelete » OutOfMemory Direct buffe...
[ERROR] ITestS3SelectLandsat.testSelectSeekFullLandsat:427->AbstractS3SelectTest.seek:701 » TestTimedOut
[INFO]
```
* OOM in HADOOP-17937; ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete. This shows we would benefit from lazy creation of buffers.
* ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier
```
[ERROR] Failures:
[ERROR] ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier:129->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0001/test/testCreateAndCopyObjectWithStorageClassGlacier/file1]
Expecting:
<null>
to be equal to:
<"glacier">
ignoring case considerations
[ERROR] ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassReducedRedundancy:104->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0001/test/testCreateAndCopyObjectWithStorageClassReducedRedundancy/file1]
Expecting:
<null>
to be equal to:
<"reduced_redundancy">
ignoring case considerations
```
I will need to see if this is a regression. I believe i have disabled setting storage class.
```
* [ERROR] testSelectSeekFullLandsat(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat) Time elapsed: 600.013 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196060941
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 61m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 22m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | | hadoop-project in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 89m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4637 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux becb6109712b 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c191f50d8d902d4aac050217d3eb5046aac98080 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/testReport/ |
| Max. process+thread count | 750 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1197057816
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 10s | | trunk passed |
| +1 :green_heart: | compile | 24m 36s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 2m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 117m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 24m 6s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 6s | | the patch passed |
| +1 :green_heart: | compile | 22m 46s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 46s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | mvnsite | 2m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 32m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 3s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 3m 21s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not generate ASF License warnings. |
| | | 207m 35s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4637 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint |
| uname | Linux 4d5009939df0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9800dd2f1d5ee5d6fd728aae748f6dd7d7897faa |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/testReport/ |
| Max. process+thread count | 719 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#discussion_r931937781
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:
##########
@@ -1274,6 +1282,21 @@ bin/hadoop s3guard markers -clean -verbose $BUCKET
# expect success and exit code of 0
bin/hadoop s3guard markers -audit -verbose $BUCKET
+# ---------------------------------------------------
+# Copy to from local
+# ---------------------------------------------------
+
+time bin/hadoop fs -copyFromLocal -t 10 share/hadoop/tools/lib/*aws*jar $BUCKET/
+
+# expect the iostatistics object_list_request value to be O(directories)
+bin/hadoop fs -ls -R $BUCKET/
+
+# expect the iostatistics object_list_request and op_get_content_summary values to be 1
Review Comment:
fixed that last night; will push and then merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196742661
the storage class tests all work when i switch from bytebuffer as the upload buffering, as does the OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196597995
CLI tests good. but they are all logging a warning about unset storage class. Filed HADOOP-18371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1197974480
aah, forgot that license-binary change. pushed up a followup, and for the backports merged the two changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196682102
test failures occur on trunk even without sdk upgrade
```
[ERROR] Failures:
[ERROR] ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier:129->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0003/test/testCreateAndCopyObjectWithStorageClassGlacier/file1]
Expecting:
<null>
to be equal to:
<"glacier">
ignoring case considerations
[ERROR] ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassReducedRedundancy:104->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0003/test/testCreateAndCopyObjectWithStorageClassReducedRedundancy/file1]
Expecting:
<null>
to be equal to:
<"reduced_redundancy">
ignoring case considerations
[ERROR] Errors:
[ERROR] ITestS3ADeleteFilesOneByOne.testBulkRenameAndDelete » OutOfMemory Direct buffe...
[ERROR] ITestS3SelectLandsat.testSelectSeekFullLandsat:427->AbstractS3SelectTest.seek:701 » TestTimedOut
[INFO]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran merged PR #4637:
URL: https://github.com/apache/hadoop/pull/4637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1197231733
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 2s | | trunk passed |
| +1 :green_heart: | compile | 26m 24s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 2m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 52s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 119m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 22m 27s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 55s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 55s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | mvnsite | 2m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 21s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 28s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 33m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 3m 22s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 37s | | The patch does not generate ASF License warnings. |
| | | 206m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4637 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint |
| uname | Linux 0a1dfd256d92 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7e1b620d2fea3a4003a80cdddc5aa14f0812e388 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/testReport/ |
| Max. process+thread count | 648 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
ahmarsuhail commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196684405
ILoadTestS3ABulkDeleteThrottling (also failing without the upgrade) is fixed in https://github.com/apache/hadoop/pull/4642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196813978
CSE-KMS mode failing, but as i've not tried it before, can't blame the SDK
```
time bin/hadoop fs -copyFromLocal -t 10 share/hadoop/tools/lib/hadoop-azure-3.4.0-SNAPSHOT.jar $BUCKET/
2022-07-27 15:10:11,422 [main] WARN s3a.S3AFileSystem (S3AFileSystem.java:createRequestFactory(1004)) - Unknown storage class property fs.s3a.create.storage.class: ; falling back to default storage class
2022-07-27 15:10:11,999 [main] WARN s3.AmazonS3EncryptionClientV2 (AmazonS3EncryptionClientV2.java:warnOnLegacyCryptoMode(409)) - The S3 Encryption Client is configured to read encrypted data with legacy encryption modes through the CryptoMode setting. If you don't have objects encrypted with these legacy modes, you should disable support for them to enhance security. See https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html
2022-07-27 15:10:11,999 [main] WARN s3.AmazonS3EncryptionClientV2 (AmazonS3EncryptionClientV2.java:warnOnRangeGetsEnabled(401)) - The S3 Encryption Client is configured to support range get requests. Range gets do not provide authenticated encryption properties even when used with an authenticated mode (AES-GCM). See https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html
2022-07-27 15:10:12,000 [main] INFO s3a.DefaultS3ClientFactory (LogExactlyOnce.java:info(44)) - S3 client-side encryption enabled: Ignore S3-CSE Warnings.
2022-07-27 15:10:12,010 [main] INFO impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be kept
2022-07-27 15:10:12,477 [main] DEBUG shell.Command (Command.java:displayError(476)) - copyFromLocal failure
org.apache.hadoop.fs.PathExistsException: `s3a://stevel-london/hadoop-azure-3.4.0-SNAPSHOT.jar': File exists
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:421)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:362)
at org.apache.hadoop.fs.shell.CopyCommandWithMultiThread.copyFileToTarget(CopyCommandWithMultiThread.java:144)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:293)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:278)
at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:382)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:345)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:318)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:273)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:300)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:284)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:244)
at org.apache.hadoop.fs.shell.CopyCommandWithMultiThread.processArguments(CopyCommandWithMultiThread.java:89)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:313)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
at org.apache.hadoop.fs.shell.Command.run(Command.java:191)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
copyFromLocal: `s3a://stevel-london/hadoop-azure-3.4.0-SNAPSHOT.jar': File exists
2022-07-27 15:10:12,482 [shutdown-hook-0] INFO statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_http_head_request=1)
(audit_request_execution=1)
(audit_span_creation=4)
(object_metadata_request=1)
(op_get_file_status=2)
(op_glob_status=1)
(store_io_request=2));
gauges=((client_side_encryption_enabled=1));
minimums=((action_http_head_request.min=433)
(op_get_file_status.min=1)
(op_glob_status.min=4));
maximums=((action_http_head_request.max=433)
(op_get_file_status.max=436)
(op_glob_status.max=4));
means=((action_http_head_request.mean=(samples=1, sum=433, mean=433.0000))
(op_get_file_status.mean=(samples=2, sum=437, mean=218.5000))
(op_glob_status.mean=(samples=1, sum=4, mean=4.0000)));
________________________________________________________
Executed in 1.72 secs fish external
usr time 3.35 secs 0.08 millis 3.35 secs
sys time 0.16 secs 1.49 millis 0.16 secs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4637: HADOOP-18344. Upgrade AWS SDK to 1.12.262
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4637:
URL: https://github.com/apache/hadoop/pull/4637#issuecomment-1196606108
setup failure in ILoadTestS3ABulkDeleteThrottling; filed HADOOP-18372
```
[INFO] Running org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[ERROR] Tests run: 12, Failures: 12, Errors: 0, Skipped: 0, Time elapsed: 10.114 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[ERROR] test_010_Reset[bulk-delete-aws-retry=false-requests=100-size=250](org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling) Time elapsed: 2.291 s <<< FAILURE!
java.lang.AssertionError:
[page size]
Expecting actual not to be null
at org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling.setup(ILoadTestS3ABulkDeleteThrottling.java:175)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org