You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2023/10/26 09:54:31 UTC
[PR] [SPARK-45651][Build][Follow-up] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
EnricoMi opened a new pull request, #43538:
URL: https://github.com/apache/spark/pull/43538
### What changes were proposed in this pull request?
Limit max memory for `mvn clean deploy` to `2g` when run in `publish_snapshot` Github workflow.
### Why are the changes needed?
The host that runs the workflow has only 7G of memory, while the `release-build.sh` script sets the limit to 12g, causing the process to be killed (for branch `master`).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not tested
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782580570
The mem statistics with `2g` are
MiB Mem : 6922.0 total, 579.9 free, 5634.8 used, 707.2 buff/cache
MiB Swap: 4096.0 total, 2069.7 free, 2026.2 used. 976.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2494 runner 20 0 16.4g 5.1g 23040 S 118.8 75.5 103:16.37 java
So even with `2g`, memory usage is still quite high but significantly better than with `12g`, there is headroom of 2g free swap and 500m free mem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782197337
For now, we use a different Docker image between the actual test and snapshot build. We cache the image (see `image_url`at `.github/workflows/build_and_test.yml`, and https://github.com/apache/spark/blob/master/dev/infra/Dockerfile). Can we match the same image?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372923883
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
> So perhaps the -Xmx here could be larger.
Why so? Do you refer to the `12g` in this line or the `2g` in the workflow? Since the build workflow uses `2g` already, we should stick to those `2g` for consistency or bump the `2g` in `build.yml` to `3g` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372918698
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
So perhaps the -Xmx here could be larger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782568332
In fact, you manually triggered the publish, and it worked perfectly fine for `master`: https://github.com/apache/spark/actions/runs/6661389655/job/18104176828
Branch `branch-3.5` failed due to an unrelated `408 Request Timeout`. So no need to revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782565642
That workflow run did not pick up the changes of this PR:
https://github.com/apache/spark/actions/runs/6661131541/workflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782777583
Let's retry few times and see if it actually works first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782571084
Yes, please!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372993319
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
Understood, either of `2g`, `3g`, and `4g` is fine with me, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1780805138
CC @LuciferYang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782542097
> Seems not working (https://github.com/apache/spark/actions/runs/6661131541/job/18103495351). I reverted this for now.
The problem was unrelated:
408 Request Timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782569044
Shall we revive this pr and give it another try ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782607346
Same HTTP timeout:
```
2023-10-27T00:46:23.1689370Z mem: top - 00:46:23 up 7 min, 0 users, load average: 0.86, 0.96, 0.50
2023-10-27T00:46:23.1690374Z mem: Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie
2023-10-27T00:46:23.1691430Z mem: %Cpu(s): 0.0 us, 3.2 sy, 0.0 ni, 96.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
2023-10-27T00:46:23.1692524Z mem: MiB Mem : 6922.0 total, 596.6 free, 2637.2 used, 3688.3 buff/cache
2023-10-27T00:46:31.1391267Z mem: MiB Swap: 4096.0 total, 4095.2 free, 0.8 used. 39
2023-10-27T00:46:31.1396047Z org.apache.maven.wagon.TransferFailedException: transfer failed for https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-network-shuffle_2.12/3.5.1-SNAPSHOT/spark-network-shuffle_2.12-3.5.1-20231027.004212-88-sources.jar.md5, status: 408 Request Timeout
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372988443
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
What I mean is perhaps `MAVEN_MXM_OPT` could be 4g, because I've tested before and `java-other-versions` can also run successfully on GA with 4g.
But 2g is also fine to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782613227
Looks like a known unrelated issue, workaround here: https://github.com/kiegroup/kie-wb-common/pull/3416/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782599355
If that's the case we can get this in again and see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372920692
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
I suspect so. The memory logging that is currently in place will tell us for sure if any attempt has any or just too little effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372920692
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
> so the previous fix probably didn't take effect...
I suspect so. The memory logging that is currently in place will tell us for sure if any attempt has any or just too little effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372920692
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
I suspect so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43538: [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow
URL: https://github.com/apache/spark/pull/43538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782722490
I think a better alternative is to retry: https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html#retryfaileddeploymentcount
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782598957
I manually triggered once more at https://github.com/apache/spark/actions/runs/6661389655 and it failed too. Is that also unrelated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782600145
(sorry it's my phone now so can't properly check the logs on my own)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782599348
Revived in #43555.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372916497
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
Ok, I previously changed the -Xmx option to 3g in `build/mvn` script, not here, so the previous fix probably didn't take effect...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782098826
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43538:
URL: https://github.com/apache/spark/pull/43538#issuecomment-1782125338
Seems not working (https://github.com/apache/spark/actions/runs/6661131541/job/18103495351). I reverted this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #43538:
URL: https://github.com/apache/spark/pull/43538#discussion_r1372923883
##########
dev/create-release/release-build.sh:
##########
@@ -227,7 +227,7 @@ git clean -d -f -x
rm -f .gitignore
cd ..
-export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
+export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_MXM_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
Review Comment:
> So perhaps the -Xmx here could be larger.
Why so? Do you refer to the `12g` in this line or the `2g` in the workflow? Since the build workflow uses `2g` already, we should stick to those `2g` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org