You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/09 06:20:40 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
LuciferYang opened a new pull request, #38577:
URL: https://github.com/apache/spark/pull/38577
### What changes were proposed in this pull request?
Run
```
dev/make-distribution.sh --tgz -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
```
on the master branch fails as follows:
```
[ERROR] ## Exception when compiling 19 sources to /spark-source/connector/avro/target/scala-2.12/classes
java.lang.OutOfMemoryError: Metaspace
java.lang.ClassLoader.defineClass1(Native Method)
java.lang.ClassLoader.defineClass(ClassLoader.java:757)
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
java.net.URLClassLoader.access$100(URLClassLoader.java:74)
java.net.URLClassLoader$1.run(URLClassLoader.java:369)
java.net.URLClassLoader$1.run(URLClassLoader.java:363)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:362)
java.lang.ClassLoader.loadClass(ClassLoader.java:419)
scala_maven.ScalaCompilerLoader.loadClass(ScalaCompilerLoader.java:44)
java.lang.ClassLoader.loadClass(ClassLoader.java:352)
scala.collection.immutable.Set$Set2.$plus(Set.scala:170)
scala.collection.immutable.Set$Set2.$plus(Set.scala:164)
scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:28)
scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:24)
scala.collection.TraversableLike$grouper$1$.apply(TraversableLike.scala:467)
scala.collection.TraversableLike$grouper$1$.apply(TraversableLike.scala:455)
scala.collection.mutable.HashMap$$anon$1.$anonfun$foreach$2(HashMap.scala:153)
scala.collection.mutable.HashMap$$anon$1$$Lambda$68804/294594059.apply(Unknown Source)
scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:153)
scala.collection.TraversableLike.groupBy(TraversableLike.scala:524)
scala.collection.TraversableLike.groupBy$(TraversableLike.scala:454)
scala.collection.AbstractTraversable.groupBy(Traversable.scala:108)
scala.tools.nsc.settings.Warnings.$init$(Warnings.scala:91)
scala.tools.nsc.settings.MutableSettings.<init>(MutableSettings.scala:28)
scala.tools.nsc.Settings.<init>(Settings.scala:19)
scala.tools.nsc.Settings.<init>(Settings.scala:20)
xsbt.CachedCompiler0.<init>(CompilerBridge.scala:79)
```
So this pr remove `MaxMetaspaceSize` option from `make-distribution.sh` to make `MaxMetaspaceSize` can use more memory resources and make `make-distribution.sh` run successfully.
### Why are the changes needed?
Make `make-distribution.sh` run successfully.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test:
```
dev/make-distribution.sh --tgz -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
```
- Before: java.lang.OutOfMemoryError: Metaspace
- After: successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308297255
I reproduced the issue w/ openjdk 1.8.0_332 macos-aarch64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
URL: https://github.com/apache/spark/pull/38577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017499034
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
It was added by https://github.com/apache/spark/pull/37510. Seems it was added also because `make-distributions.sh` failed to build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308364776
`-XX: MaxMetaspaceSize=2g ` is also specified elsewhere in `pom.xml` and `SparkBuild.scala`. Perhaps as the amount of code increases, they all have the risk of `Metaspace OOM`, but that hasn't happened yet, so I haven't changed them in this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308298746
> As you mentioned in the PR description, did you hit this only at master branch issue?
Let me double check this
> Which java version are you using now?
test on linux:
```
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Zulu 8.66.0.15-CA-linux64) (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Zulu 8.66.0.15-CA-linux64) (build 25.352-b08, mixed mode)
```
test on MacOS：
```
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Zulu 8.66.0.15-CA-linux64) (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Zulu 8.66.0.15-CA-linux64) (build 25.352-b08, mixed mode)
```
Wait me do more test with Java 11/17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308270119
cc @HyukjinKwon @wangyum @dongjoon-hyun @srowen @pan3793 @panbingkun
Can you reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
wangyum commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017537591
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
+export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
Thanks for the fix. I can reproduce this issue through github action: https://github.com/wangyum/make-distribution.sh/actions/runs/3425964379/jobs/5707275344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017563872
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
+export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
Thank @wangyum for helping check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017685914
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
+export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
<img width="1171" alt="image" src="https://user-images.githubusercontent.com/1475305/200793242-a9fabc22-d83c-4a10-9a04-3942ed2762d5.png">
https://github.com/LuciferYang/make-distribution.sh/actions/runs/3426521418
I fork your repo and I modified the script to executed `gh pr checkout 38577` to switch to this branch, then run successfully
Thanks @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308698688
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308293543
To @LuciferYang ,
- As you mentioned in the PR description, did you hit this only at `master` branch issue?
- Which java version are you using now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38577:
URL: https://github.com/apache/spark/pull/38577#issuecomment-1308290786
cc @viirya too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017502280
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
I see, but there still seems to be build fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017677470
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
Yea, I can reproduce it locally too. I think there are some changes since #37510.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38577: [SPARK-41071][BUILD] Remove `MaxMetaspaceSize` option from `make-distribution.sh` to make it run successfully
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38577:
URL: https://github.com/apache/spark/pull/38577#discussion_r1017678485
##########
dev/make-distribution.sh:
##########
@@ -161,7 +161,7 @@ fi
# Build uber fat JAR
cd "$SPARK_HOME"
-export MAVEN_OPTS="${MAVEN_OPTS:--Xss128m -Xmx4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m}"
Review Comment:
Thank @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org