You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Bulldog20630405 <bu...@gmail.com> on 2022/03/14 13:58:34 UTC
spark distribution build fails
using tag v3.2.1 with java 8 getting a stackoverflow when building the
distribution:
> alias mvn
alias mvn='mvn --errors --fail-at-end -DskipTests '
> dev/make-distribution.sh --name 'hadoop-3.2' --pip --tgz -Phive
-Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.2.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [
2.978 s]
[INFO] Spark Project Tags ................................. SUCCESS [
6.585 s]
[INFO] Spark Project Sketch ............................... SUCCESS [
6.684 s]
[INFO] Spark Project Local DB ............................. SUCCESS [
2.497 s]
[INFO] Spark Project Networking ........................... SUCCESS [
6.312 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
3.925 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
7.879 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
2.238 s]
[INFO] Spark Project Core ................................. SUCCESS [02:33
min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [
24.566 s]
[INFO] Spark Project GraphX ............................... SUCCESS [
28.293 s]
[INFO] Spark Project Streaming ............................ SUCCESS [
51.070 s]
[INFO] Spark Project Catalyst ............................. FAILURE [
36.920 s]
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
....
[INFO] Spark Avro ......................................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 05:33 min
[INFO] Finished at: 2022-03-14T13:45:15Z
[INFO]
------------------------------------------------------------------------
---------------------------------------------------
constituent[0]: file:/home/bulldog/software/maven/maven-3.8.4/conf/logging/
constituent[1]:
file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-embedder-3.8.4.jar
constituent[2]:
file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-3.8.4.jar
constituent[3]:
file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-builder-3.8.4.jar
constituent[4]:
file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-plugin-api-3.8.4.jar
....
---------------------------------------------------
Exception in thread "main" java.lang.StackOverflowError
at
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:49)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
at
scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
...
Re: spark distribution build fails
Posted by Martin Grigorov <mg...@apache.org>.
Hi,
For the mail archives: this error happens when the user has MAVEN_OPTS env
var pre-exported. In this case ./build/mvn|sbt does not export its own
MAVEN_OPTS with the -XssXYZ value, and the default one is too low and leads
to the StackOverflowError
On Mon, Mar 14, 2022 at 11:13 PM Bulldog20630405 <bu...@gmail.com>
wrote:
>
> thanx; that worked great!
>
> On Mon, Mar 14, 2022 at 11:17 AM Sean Owen <sr...@gmail.com> wrote:
>
>> Try increasing the stack size in the build. It's the Xss argument you
>> find in various parts of the pom or sbt build. I have seen this and not
>> sure why it happens on certain envs, but that's the workaround
>>
>> On Mon, Mar 14, 2022, 8:59 AM Bulldog20630405 <bu...@gmail.com>
>> wrote:
>>
>>>
>>> using tag v3.2.1 with java 8 getting a stackoverflow when building the
>>> distribution:
>>>
>>> > alias mvn
>>> alias mvn='mvn --errors --fail-at-end -DskipTests '
>>> > dev/make-distribution.sh --name 'hadoop-3.2' --pip --tgz -Phive
>>> -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary for Spark Project Parent POM 3.2.1:
>>> [INFO]
>>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>>> 2.978 s]
>>> [INFO] Spark Project Tags ................................. SUCCESS [
>>> 6.585 s]
>>> [INFO] Spark Project Sketch ............................... SUCCESS [
>>> 6.684 s]
>>> [INFO] Spark Project Local DB ............................. SUCCESS [
>>> 2.497 s]
>>> [INFO] Spark Project Networking ........................... SUCCESS [
>>> 6.312 s]
>>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>>> 3.925 s]
>>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>>> 7.879 s]
>>> [INFO] Spark Project Launcher ............................. SUCCESS [
>>> 2.238 s]
>>> [INFO] Spark Project Core ................................. SUCCESS
>>> [02:33 min]
>>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>>> 24.566 s]
>>> [INFO] Spark Project GraphX ............................... SUCCESS [
>>> 28.293 s]
>>> [INFO] Spark Project Streaming ............................ SUCCESS [
>>> 51.070 s]
>>> [INFO] Spark Project Catalyst ............................. FAILURE [
>>> 36.920 s]
>>> [INFO] Spark Project SQL .................................. SKIPPED
>>> [INFO] Spark Project ML Library ........................... SKIPPED
>>> [INFO] Spark Project Tools ................................ SKIPPED
>>> ....
>>>
>>> [INFO] Spark Avro ......................................... SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 05:33 min
>>> [INFO] Finished at: 2022-03-14T13:45:15Z
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> ---------------------------------------------------
>>> constituent[0]:
>>> file:/home/bulldog/software/maven/maven-3.8.4/conf/logging/
>>> constituent[1]:
>>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-embedder-3.8.4.jar
>>> constituent[2]:
>>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-3.8.4.jar
>>> constituent[3]:
>>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-builder-3.8.4.jar
>>> constituent[4]:
>>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-plugin-api-3.8.4.jar
>>> ....
>>> ---------------------------------------------------
>>> Exception in thread "main" java.lang.StackOverflowError
>>> at
>>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:49)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>>> at
>>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>>> ...
>>>
>>>
>>>
>>>
>>>
>>>
Re: spark distribution build fails
Posted by Bulldog20630405 <bu...@gmail.com>.
thanx; that worked great!
On Mon, Mar 14, 2022 at 11:17 AM Sean Owen <sr...@gmail.com> wrote:
> Try increasing the stack size in the build. It's the Xss argument you find
> in various parts of the pom or sbt build. I have seen this and not sure why
> it happens on certain envs, but that's the workaround
>
> On Mon, Mar 14, 2022, 8:59 AM Bulldog20630405 <bu...@gmail.com>
> wrote:
>
>>
>> using tag v3.2.1 with java 8 getting a stackoverflow when building the
>> distribution:
>>
>> > alias mvn
>> alias mvn='mvn --errors --fail-at-end -DskipTests '
>> > dev/make-distribution.sh --name 'hadoop-3.2' --pip --tgz -Phive
>> -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary for Spark Project Parent POM 3.2.1:
>> [INFO]
>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>> 2.978 s]
>> [INFO] Spark Project Tags ................................. SUCCESS [
>> 6.585 s]
>> [INFO] Spark Project Sketch ............................... SUCCESS [
>> 6.684 s]
>> [INFO] Spark Project Local DB ............................. SUCCESS [
>> 2.497 s]
>> [INFO] Spark Project Networking ........................... SUCCESS [
>> 6.312 s]
>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>> 3.925 s]
>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>> 7.879 s]
>> [INFO] Spark Project Launcher ............................. SUCCESS [
>> 2.238 s]
>> [INFO] Spark Project Core ................................. SUCCESS
>> [02:33 min]
>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>> 24.566 s]
>> [INFO] Spark Project GraphX ............................... SUCCESS [
>> 28.293 s]
>> [INFO] Spark Project Streaming ............................ SUCCESS [
>> 51.070 s]
>> [INFO] Spark Project Catalyst ............................. FAILURE [
>> 36.920 s]
>> [INFO] Spark Project SQL .................................. SKIPPED
>> [INFO] Spark Project ML Library ........................... SKIPPED
>> [INFO] Spark Project Tools ................................ SKIPPED
>> ....
>>
>> [INFO] Spark Avro ......................................... SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 05:33 min
>> [INFO] Finished at: 2022-03-14T13:45:15Z
>> [INFO]
>> ------------------------------------------------------------------------
>> ---------------------------------------------------
>> constituent[0]:
>> file:/home/bulldog/software/maven/maven-3.8.4/conf/logging/
>> constituent[1]:
>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-embedder-3.8.4.jar
>> constituent[2]:
>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-3.8.4.jar
>> constituent[3]:
>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-builder-3.8.4.jar
>> constituent[4]:
>> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-plugin-api-3.8.4.jar
>> ....
>> ---------------------------------------------------
>> Exception in thread "main" java.lang.StackOverflowError
>> at
>> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:49)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
>> at
>> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
>> ...
>>
>>
>>
>>
>>
>>
Re: spark distribution build fails
Posted by Sean Owen <sr...@gmail.com>.
Try increasing the stack size in the build. It's the Xss argument you find
in various parts of the pom or sbt build. I have seen this and not sure why
it happens on certain envs, but that's the workaround
On Mon, Mar 14, 2022, 8:59 AM Bulldog20630405 <bu...@gmail.com>
wrote:
>
> using tag v3.2.1 with java 8 getting a stackoverflow when building the
> distribution:
>
> > alias mvn
> alias mvn='mvn --errors --fail-at-end -DskipTests '
> > dev/make-distribution.sh --name 'hadoop-3.2' --pip --tgz -Phive
> -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Spark Project Parent POM 3.2.1:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [
> 2.978 s]
> [INFO] Spark Project Tags ................................. SUCCESS [
> 6.585 s]
> [INFO] Spark Project Sketch ............................... SUCCESS [
> 6.684 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [
> 2.497 s]
> [INFO] Spark Project Networking ........................... SUCCESS [
> 6.312 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
> 3.925 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [
> 7.879 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [
> 2.238 s]
> [INFO] Spark Project Core ................................. SUCCESS [02:33
> min]
> [INFO] Spark Project ML Local Library ..................... SUCCESS [
> 24.566 s]
> [INFO] Spark Project GraphX ............................... SUCCESS [
> 28.293 s]
> [INFO] Spark Project Streaming ............................ SUCCESS [
> 51.070 s]
> [INFO] Spark Project Catalyst ............................. FAILURE [
> 36.920 s]
> [INFO] Spark Project SQL .................................. SKIPPED
> [INFO] Spark Project ML Library ........................... SKIPPED
> [INFO] Spark Project Tools ................................ SKIPPED
> ....
>
> [INFO] Spark Avro ......................................... SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 05:33 min
> [INFO] Finished at: 2022-03-14T13:45:15Z
> [INFO]
> ------------------------------------------------------------------------
> ---------------------------------------------------
> constituent[0]: file:/home/bulldog/software/maven/maven-3.8.4/conf/logging/
> constituent[1]:
> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-embedder-3.8.4.jar
> constituent[2]:
> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-3.8.4.jar
> constituent[3]:
> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-settings-builder-3.8.4.jar
> constituent[4]:
> file:/home/bulldog/software/maven/maven-3.8.4/lib/maven-plugin-api-3.8.4.jar
> ....
> ---------------------------------------------------
> Exception in thread "main" java.lang.StackOverflowError
> at
> scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:49)
> at
> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275)
> at
> scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:133)
> ...
>
>
>
>
>
>