You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2023/10/24 16:36:40 UTC
[PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
EnricoMi opened a new pull request, #43513:
URL: https://github.com/apache/spark/pull/43513
### What changes were proposed in this pull request?
This logs memory consumption while publishing snapshots. This is to investigate whether the suspected high memory usage is the root cause of `publish_snapshots` failures for master.
Merging this after #43512 allows to run this manually.
### Why are the changes needed?
The working assumption is that high memory usage is the root cause. This logging should provide proof or disproof for this assumption. This can be reverted once more is known or SPARK-45651 is fixed.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Locally
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780790906
See #43538.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1777613933
CC @LuciferYang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1782125124
i reverted this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780797508
> `java-other-versions` task use `... -Xmx2g ...`
This explains why `-Xmx3g` in #43364 did not suffice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780763639
The `release-build.sh` script defines
export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"
So it could simply allow to override the `-Xmx` value as
export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_XMX_OPT:-12g} -XX:ReservedCodeCacheSize=1g"
then, `publish_snapshot.yml` can define
```yaml
- name: Publish snapshot
env:
...
MAVEN_XMX_OPT: "2g"
run: |
./dev/create-release/release-build.sh publish-snapshot
```
The `release-build.sh` script should care for the context it is running in (e.g. Github actions), so it shouldn't define env variables that are context-specific (e.g. `GA_DEPLOY_MAVEN_OPTS`), but purpose specific (e.g. `MAVEN_XMX_OPT`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1778227490
If this is just a debug purpose, I would recommend doing this within your fork first ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780592435
> Please trigger the `publish_snapshot` workflow for the `master` branch manually.
https://github.com/apache/spark/actions/runs/6651054633/job/18072315690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780282721
Okey, let's get this in and see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780282855
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780803921
> > `java-other-versions` task use `... -Xmx2g ...`
>
> This explains why `-Xmx3g` in #43364 did not suffice.
I still underestimated the memory usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780689368
The last log entry is
```
2023-10-26T08:23:55.5712846Z mem: Thu Oct 26 08:23:07 UTC 2023
2023-10-26T08:23:55.5713956Z mem: top - 08:23:35 up 34 min, 0 users, load average: 10.56, 7.82, 6.44
2023-10-26T08:23:55.5714956Z mem: Tasks: 129 total, 4 running, 125 sleeping, 0 stopped, 0 zombie
2023-10-26T08:23:55.5715903Z mem: %Cpu(s): 0.5 us, 23.3 sy, 0.0 ni, 9.7 id, 65.6 wa, 0.0 hi, 1.0 si, 0.0 st
2023-10-26T08:23:55.5716952Z mem: MiB Mem : 6922.0 total, 103.6 free, 6692.7 used, 125.7 buff/cache
2023-10-26T08:23:55.5718023Z mem: MiB Swap: 4096.0 total, 0.0 free, 4096.0 used. 20.2 avail Mem
2023-10-26T08:23:55.5718710Z mem:
2023-10-26T08:23:55.5719344Z mem: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2023-10-26T08:23:55.5720469Z mem: 49 root 20 0 0 0 0 R 14.3 0.0 1:30.13 kswapd0
2023-10-26T08:23:55.5721397Z mem: 1 root 20 0 167724 4352 3072 R 2.0 0.1 0:05.46 systemd
2023-10-26T08:23:55.5722332Z mem: 572 root 20 0 261.6g 37392 8960 S 1.3 0.5 0:10.11 provisi+
2023-10-26T08:23:55.5723285Z mem: 585 root 20 0 1421916 13860 1408 S 1.2 0.2 0:05.43 contain+
2023-10-26T08:23:55.5724228Z mem: 863 root 20 0 365712 13580 3200 D 1.2 0.2 0:08.26 python3
2023-10-26T08:23:55.5725115Z mem: 1643 runner 20 0 3839788 39500 3036 S 0.8 0.6 0:24.63 Runner.+
2023-10-26T08:23:55.5726094Z mem: 821 root 20 0 1454428 20208 3712 S 0.7 0.3 0:03.47 dockerd
2023-10-26T08:23:55.5737404Z mem: 2462 runner 20 0 21.0g 6.0g 3132 S 0.7 89.3 33:02.14 java
2023-10-26T08:23:55.5738711Z mem: 1620 root 20 0 1297076 16768 1664 S 0.6 0.2 1:12.63 provjobd
```
The `java` build process consumes 89% of the available memory (6g of 7g). Even 4g of swap is being used by that point. Reducing the memory footprint of the build process should be the right strategy. Additionally, the available swap space could also be increased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780560281
Please trigger the `publish_snapshot` workflow for the `master` branch manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1779066166
I ran this logging in my fork, but cannot run the publish build there, of course: https://github.com/EnricoMi/spark/actions/runs/6638012356/job/18033491439#step:6:434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #43513: [SPARK-45651][Build] Log memory usage of publish snapshot workflow
URL: https://github.com/apache/spark/pull/43513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780704895
https://github.com/apache/spark/blob/7d7afb06f682c10f3900eb8adeab9fad6d49cb24/.github/workflows/build_and_test.yml#L825
`java-other-versions` task use `-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g`.
Maybe we could try adding an env to configure Java options, suppose it's called `GA_DEPLOY_MAVEN_OPTS`.
When `publish_snapshot.yml` sets this environment variable, run `export MAVEN_OPTS = "$GA_DEPLOY_MAVEN_OPTS"` before `mvn deploy` in the script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45651][Build] Log memory usage of publish snapshot workflow [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #43513:
URL: https://github.com/apache/spark/pull/43513#issuecomment-1780807019
And, since `release-build.sh` sets `MAVEN_OPTS`, that setting of 3g in `build/mvn` was ignored / overridden in `publish_snapshot.yml` anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org