You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "amoeba (via GitHub)" <gi...@apache.org> on 2023/09/14 19:30:43 UTC

[GitHub] [arrow] amoeba opened a new pull request, #37727: GH-34105: [R] Provide extra output for failed builds

amoeba opened a new pull request, #37727:
URL: https://github.com/apache/arrow/pull/37727

   ### Rationale for this change
   
   This is a replacement for the previous PR https://github.com/apache/arrow/pull/37698. The rationale for this PR is providing extra output for R package builds where the C++ build fails
   
   
   ### What changes are included in this PR?
   
   Update the system call to save output when building Arrow C++ from the R package and output it if it's failed
   
   
   ### Are these changes tested?
   
   No automated tests but the changes have been tested manually.
   
   ### Are there any user-facing changes?
   
   Yes, but only for users building the R package from source which is hopefully not common.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1327648554


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   Gotcha, that makes sense now. I thought the pre-existing reliance on bash in nixlibs.R was odd, and we actually have this bit:
   
   https://github.com/apache/arrow/blob/d9ee3a7961df44a1e30d3fe615ee3a20af569025/r/tools/nixlibs.R#L625-L627
   
   and another spot where we make the `command` bash,
   
   https://github.com/apache/arrow/blob/d9ee3a7961df44a1e30d3fe615ee3a20af569025/r/tools/nixlibs.R#L630
   
   So this change adds some consistency. I'd be happy to changes things if/when CRAN has any issues. Thanks for catching it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1327648554


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   Gotcha, that makes sense now. I thought the pre-existing reliance on bash in nixlibs.R was odd, and we actually have this bit:
   
   https://github.com/apache/arrow/blob/d9ee3a7961df44a1e30d3fe615ee3a20af569025/r/tools/nixlibs.R#L625-L627
   
   and another spot where we make the `command` bash,
   
   https://github.com/apache/arrow/blob/d9ee3a7961df44a1e30d3fe615ee3a20af569025/r/tools/nixlibs.R#L630
   
   So this change adds some consistency. I'd be happy to changes things if/when CRAN has any issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on PR #37727:
URL: https://github.com/apache/arrow/pull/37727#issuecomment-1720297311

   I've updated the PR to address points brought up in https://github.com/apache/arrow/pull/37727#discussion_r1326477485 so this is ready for another look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1327157171


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   I was referring to the next line, looks like the mobile app attached the comment to the wrong line. I'm not sure that directly calling "bash" is necessary or a good idea--we didn't before, the script has a shebang at the top, and bash might not be the shell that's available, a point that CRAN has been pedantic about in the past. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] raulcd merged pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd merged PR #37727:
URL: https://github.com/apache/arrow/pull/37727


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] assignUser commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326565633


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   +1 on using a file and `cat`ing that in full if build fails. With the nature of CMake errors it is not always the case that the error is in the last lines (due to parallelism) and the issue with non-portable flag false positives is not an issue this way as the build has already failed...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326608502


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   @nealrichardson, Re:
   
   > I'm not sure you need this, we didn't before.
   
   Were you referring to how I broke up the args in the system2 call? Calling the script the old way does work with `system` but we switched to `system2 `to capture stdin/stdout and calling system2 the same way resulted in this error,
   
   > sh: SOURCE_DIR='../cpp' BUILD_DIR='/var/folders/vm/bbcxtl3s63l7gn9h1tvfh1mm0000gn/T//RtmpoTYnF4/file166a06be38857' DEST_DIR='libarrow/arrow-13.0.0.9000' CMAKE='/opt/homebrew/bin/cmake' EXTRA_CMAKE_FLAGS='' CC='clang -arch arm64' CXX='clang++ -arch arm64 -std=gnu++17' LDFLAGS='-L/opt/R/arm64/lib' N_JOBS='2' ARROW_S3='OFF' ARROW_GCS='OFF' inst/build_arrow_static.sh: No such file or directory
   
   I think this is because of what the man page for `system2` says, which is:
   
   > Unlike [system](http://127.0.0.1:17041/help/library/base/help/system), command is always quoted by [shQuote](http://127.0.0.1:17041/help/library/base/help/shQuote), so it must be a single command without arguments.
   
   So I felt like my change to the args made sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326477485


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   I'm not sure you need this, we didn't before. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326578253


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   My experience with build failures matches what @assignUser refers to with having to scroll up (sometimes quite far) to find the source of the failure. One trick I use to bring the error down is to re-build without parallelism but I'm not sure how best to implement that here.
   
   So I'd be in favor of (1) writing to a file instead of capturing and (2) cating the full file to hedge against the tempfile disappearing before someone can grab it. I'll do that now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] amoeba commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326615534


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   Alright, with https://github.com/apache/arrow/pull/37727/commits/4246503793ca46040c4b941469a5876268f7c965, we now redirect stdout/stderr to a tempfile and print that file out on a build failure. The way I set it up, it looks like
   
   > **** Error building Arrow C++.
   > **** Printing contents of build log because the build failed while ARROW_R_DEV was set to FALSE
   > /* contents of build log */
   > **** Complete build log is available at /var/folders/vm/bbcxtl3s63l7gn9h1tvfh1mm0000gn/T//RtmpKBAT6R/file75d2a7f76ba.log
   
   I printed the build log last in case we print so much information in the prior step that the user's terminal truncates the first part of the output so the user can still find the file (if it exists at this point).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37727:
URL: https://github.com/apache/arrow/pull/37727#issuecomment-1721982518

   After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit ac7e9a4c41b73242b2f7a15f13f3c8fde843416d.
   
   There was 1 benchmark result indicating a performance regression:
   
   - Commit Run on `ursa-i9-9960x` at [2023-09-15 20:33:04Z](https://conbench.ursa.dev/compare/runs/918c8c908a4849968b4f079cd9559b23...9a49bf1e48aa4c4ea3e5b221d82b796f/)
     - [`file-read` (R) with compression=lz4, dataset=fanniemae_2016Q4, file_type=feather, language=R, output_type=dataframe](https://conbench.ursa.dev/compare/benchmarks/0650485275c3798f800024a08e0bb75e...06504d205fc97be28000319128c25a07)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/16844495775) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] thisisnic commented on a diff in pull request #37727: GH-34105: [R] Provide extra output for failed builds

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on code in PR #37727:
URL: https://github.com/apache/arrow/pull/37727#discussion_r1326522658


##########
r/tools/nixlibs.R:
##########
@@ -473,17 +473,20 @@ build_libarrow <- function(src_dir, dst_dir) {
   env_vars <- env_vars_as_string(env_var_list)
 
   cat("**** arrow", ifelse(quietly, "", paste("with", env_vars)), "\n")
-  status <- suppressWarnings(system(
-    paste(env_vars, "inst/build_arrow_static.sh"),
-    ignore.stdout = quietly, ignore.stderr = quietly
+
+  status <- suppressWarnings(system2(
+    "bash",

Review Comment:
   Yeah, I think my earlier approach (which is what @amoeba has done an excellent job of fixing here!) which saves stuff to an R vector is not great, as it leaves a lot of room for R weirdness.  Perhaps @nealrichardson 's suggestion from Zulip might be worth trying instead: 
   
   > Probably a better way would be to redirect the build output to a file, and print the file name if the build fails so that you can inspect it (though an installation of a source package would put that in a temp dir that would probably be removed on exit)
   
   > Could also print the last N lines of that output on failure (would have to experiment to find a reasonable N), that might be enough guidance to debug something



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org