You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/05/18 04:38:37 UTC
[GitHub] [bigtop] sekikn opened a new pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
sekikn opened a new pull request #776:
URL: https://github.com/apache/bigtop/pull/776
https://issues.apache.org/jira/browse/BIGTOP-3540
Currently, `./gradlew toolchain` succeeds silently even if it fails to install R libraries.
With this PR, users can get aware of that failure from the message as follows (manually edited renv.pp for testing purpose, to specify non-existent package):
```
Debug: Exec[install_r_packages](provider=posix): Executing '/usr/bin/R -e 'pkgs <- c("foo"); install.packages(pkgs, repo="http://cran.r-project.org/"); for (pkg in pkgs[pkgs != "devtools"]) if (!library(pkg, character.only=TRUE, logical.return=TRUE)) q(save="no", status=1)''
Debug: Executing '/usr/bin/R -e 'pkgs <- c("foo"); install.packages(pkgs, repo="http://cran.r-project.org/"); for (pkg in pkgs[pkgs != "devtools"]) if (!library(pkg, character.only=TRUE, logical.return=TRUE)) q(save="no", status=1)''
...
Error: /usr/bin/R -e 'pkgs <- c("foo"); install.packages(pkgs, repo="http://cran.r-project.org/"); for (pkg in pkgs[pkgs != "devtools"]) if (!library(pkg, character.only=TRUE, logical.return=TRUE)) q(save="no", status=1)' returned 1 instead of one of [0]
Error: /Stage[main]/Bigtop_toolchain::Renv/Exec[install_r_packages]/returns: change from notrun to 0 failed: /usr/bin/R -e 'pkgs <- c("foo"); install.packages(pkgs, repo="http://cran.r-project.org/"); for (pkg in pkgs[pkgs != "devtools"]) if (!library(pkg, character.only=TRUE, logical.return=TRUE)) q(save="no", status=1)' returned 1 instead of one of [0]
...
BUILD SUCCESSFUL in 19s
1 actionable task: 1 executed
```
In the above example, `puppet apply` itself still succeeded, but it's already filed as https://issues.apache.org/jira/browse/BIGTOP-3402 and will be addressed separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865435666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #776: BIGTOP-3540. Fix toolchain to abort if R installation failed.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865614344
Thanks @iwasakims, just merged it with the updated issue title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn edited a comment on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
sekikn edited a comment on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865435666
@iwasakims This PR is not for fixing Spark build failure directly, but for noticing if the Docker image is wrongly built. So the original summary may be misleading.
A recent example is https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/714/COMPONENTS=spark,OS=fedora-33-ppc64le/console. This build was executed on the Docker image with id:2774aaa4f1d6, [which seems to be successfully built at first grance](https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/OS=fedora-33-ppc64le/169/console), but it silently failed to install R as follows.
```
jenkins@bigtop1slave:~$ docker images bigtop/slaves:trunk-fedora-33-ppc64le
REPOSITORY TAG IMAGE ID CREATED SIZE
bigtop/slaves trunk-fedora-33-ppc64le 2774aaa4f1d6 3 days ago 2.5GB
jenkins@bigtop1slave:~$ docker run -it bigtop/slaves:trunk-fedora-33-ppc64le bash
[root@d688f29b437c /]# R
bash: R: command not found
```
With this PR, we can get aware of the lack of R and its libraries when building Docker images before actual Spark build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865435666
@iwasakims This PR is not for fixing Spark build failure directly, but for noticing if the Docker image is wrongly built. So the original summary may be misleading.
A recent example is https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/714/COMPONENTS=spark,OS=fedora-33-ppc64le/console. This build was executed on the Docker image with id:2774aaa4f1d6, [which is successfully built at first grance](https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/OS=fedora-33-ppc64le/169/console), but it silently failed to install R as follows.
```
jenkins@bigtop1slave:~$ docker images bigtop/slaves:trunk-fedora-33-ppc64le
REPOSITORY TAG IMAGE ID CREATED SIZE
bigtop/slaves trunk-fedora-33-ppc64le 2774aaa4f1d6 3 days ago 2.5GB
jenkins@bigtop1slave:~$ docker run -it bigtop/slaves:trunk-fedora-33-ppc64le bash
[root@d688f29b437c /]# R
bash: R: command not found
```
With this PR, we can get aware of the lack of R and its libraries when building Docker images before actual Spark build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865437809
You can update the issue title based on the intent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn merged pull request #776: BIGTOP-3540. Fix toolchain to abort if R installation failed.
Posted by GitBox <gi...@apache.org>.
sekikn merged pull request #776:
URL: https://github.com/apache/bigtop/pull/776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn edited a comment on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
sekikn edited a comment on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865435666
@iwasakims This PR is not for fixing Spark build failure directly, but for noticing if the Docker image is wrongly built. So the original summary may be misleading.
A recent example is https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/714/COMPONENTS=spark,OS=fedora-33-ppc64le/console. This build was executed on the Docker image with id:2774aaa4f1d6, [which seems to be successfully built at first grance](https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk/OS=fedora-33-ppc64le/169/console), but it silently failed to install R as follows.
```
jenkins@bigtop1slave:~$ docker images bigtop/slaves:trunk-fedora-33-ppc64le
REPOSITORY TAG IMAGE ID CREATED SIZE
bigtop/slaves trunk-fedora-33-ppc64le 2774aaa4f1d6 3 days ago 2.5GB
jenkins@bigtop1slave:~$ docker run -it bigtop/slaves:trunk-fedora-33-ppc64le bash
[root@d688f29b437c /]# R
bash: R: command not found
```
With this PR, we can get aware of the lack of R and its libraries when building Docker images before actual Spark build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-843108293
@sekikn On my local CentOS 7, the `install_r_packages` worked regradless of the patch. The fix looks just making the command fail if expected packages are not found. Does this really fix the build failure of Spark in CI?
```
$ sudo /usr/bin/R -e "remove.packages(c('evaluate', 'rmarkdown', 'knitr', 'roxygen2', 'testthat', 'e1071'))"
$ /usr/bin/R -e 'library("knitr", character.only=TRUE, logical.return=TRUE)'
...
> library("knitr", character.only=TRUE, logical.return=TRUE)
[1] FALSE
Warning message:
In library("knitr", character.only = TRUE, logical.return = TRUE) :
there is no package called ‘knitr’
$ ./gradlew toolchain
...
Debug: Exec[install_r_packages](provider=posix): Executing '/usr/bin/R -e "install.packages(c('devtools', 'evaluate', 'rmarkdown', 'knitr', 'roxygen2', 'testthat', 'e1071'), repos = 'http://cran.r-project.org/')"'
Debug: Executing '/usr/bin/R -e "install.packages(c('devtools', 'evaluate', 'rmarkdown', 'knitr', 'roxygen2', 'testthat', 'e1071'), repos = 'http://cran.r-project.org/')"'
...
$ /usr/bin/R -e 'library("knitr", character.only=TRUE, logical.return=TRUE)'
...
> library("knitr", character.only=TRUE, logical.return=TRUE)
[1] TRUE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn merged pull request #776: BIGTOP-3540. Fix toolchain to abort if R installation failed.
Posted by GitBox <gi...@apache.org>.
sekikn merged pull request #776:
URL: https://github.com/apache/bigtop/pull/776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #776: BIGTOP-3540. Building Spark fails on CentOS 7.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #776:
URL: https://github.com/apache/bigtop/pull/776#issuecomment-865437809
You can update the issue title based on the intent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org