You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/11/22 04:14:05 UTC
[GitHub] [bigtop] iwasakims opened a new pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
iwasakims opened a new pull request #833:
URL: https://github.com/apache/bigtop/pull/833
https://issues.apache.org/jira/browse/BIGTOP-3602
* we can remove patch2-snappy-java-1.1.8.diff since Spark 3.1.2 uses snappy-java 1.1.8.2 including fixes we need.
* we need SPARK-35420 (patch0-SPARK-35420.diff) for compiling Spark 3.1.2 against Guava 27.0-jre.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #833:
URL: https://github.com/apache/bigtop/pull/833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #833:
URL: https://github.com/apache/bigtop/pull/833#issuecomment-1000128898
I was able to build the package on debian 11, but the SparkR smoke test failed. I tried to re-run in manually on the docker container, and I got:
```
# /usr/lib/spark/bin/spark-submit --master yarn /tmp/dataframe.R
[..]
Error: package or namespace load failed for 'SparkR':
package 'SparkR' was installed before R 4.0.0: please re-install it
Execution halted
```
Debian Bullseye ships with R 4.X, meanwhile Buster is on 3.5.x afaics..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #833:
URL: https://github.com/apache/bigtop/pull/833#issuecomment-1000197027
My bad, I missed to mention that I was trying to run the smoke tests with the current version of Spark on Debian 11, and the above error popped up. I tried to install 3.1.2 on the docker provisioner (built on debian-11, with R 4.x) and the error went away.
Building + smoke tests succeeded on Debian 11, +1 to go!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #833:
URL: https://github.com/apache/bigtop/pull/833#issuecomment-975083946
I got no issue on running smoke-tests of Spark on CentOS 8 container (x86_64).
```
$ ./docker-hadoop.sh \
--create 1 \
--image bigtop/puppet:trunk-centos-8 \
--memory 16g \
--repo file:///bigtop-home/output \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,hive,spark \
--smoke-tests spark
...
:bigtop-tests:smoke-tests:spark:test (Thread[Execution worker for ':' Thread 3,5,main]) completed. Took 1 mins 7.904 secs.
BUILD SUCCESSFUL in 1m 41s
34 actionable tasks: 7 executed, 27 up-to-date
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #833:
URL: https://github.com/apache/bigtop/pull/833#issuecomment-1000664868
Thanks, @elukey. I merged this to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #833: BIGTOP-3602. Bump Spark to 3.1.2.
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #833:
URL: https://github.com/apache/bigtop/pull/833#issuecomment-1000197027
My bad, I missed to mention that I was trying to run the smoke tests with the current version of Spark on Debian 11, and the above error popped up. I tried to install 3.1.2 on the docker provisioner (built on debian-11, with R 4.x) and the error went away.
Building + smoke tests succeeded on Debian 11, +1 to go!
Not sure what's up with the Travis checks though, can't find the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org