You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "cnauroth (via GitHub)" <gi...@apache.org> on 2023/03/24 16:27:15 UTC
[GitHub] [hadoop] cnauroth opened a new pull request, #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
cnauroth opened a new pull request, #5511:
URL: https://github.com/apache/hadoop/pull/5511
### Description of PR
During verification of releases on the 3.3 line, I often run out of heap during full test runs inside the Docker container. Let's increase the default in `MAVEN_OPTS` to match trunk.
### How was this patch tested?
After running `./start-build-env.sh` on branch-3.3:
```
mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
```
Before the patch, this would often fail from running out of heap. After the patch, it works consistently. The new `-Xmx` value was chosen to match the value on trunk.
### For code changes:
- [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153849194
##########
dev-support/docker/Dockerfile_aarch64:
##########
@@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8
###
# Avoid out of memory errors in builds
###
-ENV MAVEN_OPTS -Xms256m -Xmx1536m
+ENV MAVEN_OPTS -Xms256m -Xmx3072m
Review Comment:
Thanks for the review. Good idea. I requested your review on #5522 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#issuecomment-1483083732
@vinayakumarb , may I get your code review, since you had worked on the aarch64 Dockerfile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153646744
##########
dev-support/docker/Dockerfile_aarch64:
##########
@@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8
###
# Avoid out of memory errors in builds
###
-ENV MAVEN_OPTS -Xms256m -Xmx1536m
+ENV MAVEN_OPTS -Xms256m -Xmx3072m
Review Comment:
For aarch64 trunk doesn't have this change, worth pushing there as well.
https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile_aarch64#L77
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#issuecomment-1483086346
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5511/1/console in case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#issuecomment-1495100554
@ayushtkn and @slfan1989 , thank you for the code reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] cnauroth closed pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth closed pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.
URL: https://github.com/apache/hadoop/pull/5511
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#issuecomment-1492918106
LGTM +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#issuecomment-1483181089
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 10s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 15m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 23m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | hadolint | 0m 4s | | No new issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 22m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 72m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5511/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5511 |
| Optional Tests | dupname asflicense codespell detsecrets hadolint shellcheck shelldocs |
| uname | Linux 6112617bd83c 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 44c4fbf8e1bd97073ab9b40907b3e84db2e23f58 |
| Max. process+thread count | 690 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5511/1/console |
| versions | git=2.17.1 maven=3.6.0 hadolint=1.11.1-0-g0e692dd shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #5511:
URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153646744
##########
dev-support/docker/Dockerfile_aarch64:
##########
@@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8
###
# Avoid out of memory errors in builds
###
-ENV MAVEN_OPTS -Xms256m -Xmx1536m
+ENV MAVEN_OPTS -Xms256m -Xmx3072m
Review Comment:
For aarch64 trunk doesn't have this change, worth pushing there as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org