You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "assignUser (via GitHub)" <gi...@apache.org> on 2023/10/11 03:56:08 UTC
[PR] GH-38043: [R] Enable cloud support by default [arrow]
assignUser opened a new pull request, #38195:
URL: https://github.com/apache/arrow/pull/38195
### Rationale for this change
Previously GCS/S3 support would need to be explicitly enabled in source builds (when they are build without `NOT_CRAN`). As we want the macos binaries to be fully featured we should turn the features on when the dependencies exists.
### What changes are included in this PR?
This PR enables this behavior for both macOS and Linux as I don't see a reason not to give users the maximum possible features.
### Are these changes tested?
Crossbow and locally (thanks @paleolimbot )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758808112
I think this should be it, please review :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357150744
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
Review Comment:
I'm not sure I follow what you're saying here as to why it's redundant (or not used)? Could you explain a bit more for me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756735707
Revision: 08b463f2a1d3ac90725651ca53abde7eecd12aed
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1af1d91145](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1af1d91145)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6477847554/job/17588706740)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759758447
I'm also building locally to try this out, will keep y'all posted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356916932
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
Review Comment:
Why delete this comment? Is it no longer accurate or useful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756846285
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759813172
I think my local 11.7 build fails because it's mixing homebrew and bundled dependencies. I noticed that the configuration output showed that ARROW_DEPENDENCY_SOURCE was AUTO...should that be BUNDLED for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357007535
##########
r/configure:
##########
@@ -121,16 +121,25 @@ else
ARROW_USE_PKG_CONFIG="false"
fi
-# find openssl on macos. macOS ships with libressl. openssl is installable
-# with brew, but it is generally not linked. We can over-ride this and find
-# openssl but setting OPENSSL_ROOT_DIR (which cmake will pick up later in
-# the installation process). FWIW, arrow's cmake process uses this
-# same process to find openssl, but doing it now allows us to catch it in
-# nixlibs.R and throw a nicer error.
+## Find openssl
+# Arrow's cmake process uses this same process to find openssl,
+# but doing it now allows us to catch it in
+# nixlibs.R and activate S3 and GCS support for the source build.
+
+# macOS ships with libressl. openssl is installable with brew, but it is
+# generally not linked. We can over-ride this and find
+# openssl by setting OPENSSL_ROOT_DIR (which cmake will pick up later in
+# the installation process).
if [ "${OPENSSL_ROOT_DIR}" = "" ] && brew --prefix openssl >/dev/null 2>&1; then
export OPENSSL_ROOT_DIR="`brew --prefix openssl`"
export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
fi
+# Look for openssl with pkg-config for non-brew sources(e.g. CRAN) and Linux
Review Comment:
Yes this would of course work for any setup that has pkg-config available but we know that CRAN does not have brew but does have pkg-config so that's why I list it as an example :) (contrasting to the brew finding logic that's directly above which I would say is the default for the 'normal' mac dev)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357032188
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
- # This duplicates what we do with the test program above when we check
- # capabilities for using binaries. We could consider consolidating this
- # logic, though these use cmake in order to match exactly what we do in the
- # libarrow build, and maybe that increases the fidelity.
Review Comment:
see https://github.com/apache/arrow/pull/38195#discussion_r1357031841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758590026
Revision: d8aff44be00688ed3a4be59ce4d82bea8677f175
Submitted crossbow builds: [ursacomputing/crossbow @ actions-95192251cf](https://github.com/ursacomputing/crossbow/branches/all?query=actions-95192251cf)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6488322536/job/17620527762)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758006914
Some weirdness going on with the linux deps but mac works!
```
Arrow package version: 13.0.0.100000311
Capabilities:
acero TRUE
dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 TRUE
gcs TRUE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc TRUE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758255991
Revision: ab8c7f86c99194d3e45a69b43fd362b1e98b3355
Submitted crossbow builds: [ursacomputing/crossbow @ actions-548e152586](https://github.com/ursacomputing/crossbow/branches/all?query=actions-548e152586)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6486474195/job/17614756607)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759985919
Revision: 9d209a67c69a4805d2f73dee4ad82ce5bada7d3b
Submitted crossbow builds: [ursacomputing/crossbow @ actions-431cdc81fd](https://github.com/ursacomputing/crossbow/branches/all?query=actions-431cdc81fd)
|Task|Status|
|----|------|
|conda-linux-aarch64-cpu-r42|[](https://github.com/ursacomputing/crossbow/runs/17650022583)|
|conda-linux-aarch64-cpu-r43|[](https://github.com/ursacomputing/crossbow/runs/17650023413)|
|conda-linux-x64-cpu-r42|[](https://github.com/ursacomputing/crossbow/runs/17650023418)|
|conda-linux-x64-cpu-r43|[](https://github.com/ursacomputing/crossbow/runs/17650023479)|
|conda-osx-arm64-cpu-r42|[](https://github.com/ursacomputing/crossbow/runs/17650023825)|
|conda-osx-arm64-cpu-r43|[](https://github.com/ursacomputing/crossbow/runs/17650024047)|
|conda-osx-x64-cpu-r42|[](https://github.com/ursacomputing/crossbow/runs/17650022804)|
|conda-osx-x64-cpu-r43|[](https://github.com/ursacomputing/crossbow/runs/17650023340)|
|conda-win-x64-cpu-r41|[](https://github.com/ursacomputing/crossbow/runs/17650023942)|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521684/job/17650043177)|
|test-fedora-r-clang-sanitizer|[](https://github.com/ursacomputing/crossbow/runs/17650024736)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521798/job/17650023258)|
|test-r-depsource-bundled|[](https://github.com/ursacomputing/crossbow/runs/17650024233)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521677/job/17650022402)|
|test-r-dev-duckdb|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521691/job/17650022433)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521753/job/17650023366)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521719/job/17650022364)|
|test-r-gcc-12|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521737/job/17650022662)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521825/job/17650024032)|
|test-r-install-local-minsizerel|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521839/job/17650024456)|
|test-r-library-r-base-latest|[](https://github.com/ursacomputing/crossbow/runs/17650024263)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521826/job/17650025403)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521875/job/17650023723)|
|test-r-linux-valgrind|[](https://github.com/ursacomputing/crossbow/runs/17650023779)|
|test-r-minimal-build|[](https://github.com/ursacomputing/crossbow/runs/17650024301)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521851/job/17650023577)|
|test-r-offline-minimal|[](https://github.com/ursacomputing/crossbow/runs/17650023335)|
|test-r-rhub-debian-gcc-devel-lto-latest|[](https://github.com/ursacomputing/crossbow/runs/17650024709)|
|test-r-rhub-debian-gcc-release-custom-ccache|[](https://github.com/ursacomputing/crossbow/runs/17650025126)|
|test-r-rhub-ubuntu-gcc-release-latest|[](https://github.com/ursacomputing/crossbow/runs/17650025141)|
|test-r-rstudio-r-base-4.1-opensuse153|[](https://github.com/ursacomputing/crossbow/runs/17650023851)|
|test-r-rstudio-r-base-4.2-centos7-devtoolset-8|[](https://github.com/ursacomputing/crossbow/runs/17650025072)|
|test-r-rstudio-r-base-4.2-focal|[](https://github.com/ursacomputing/crossbow/runs/17650024769)|
|test-r-ubuntu-22.04|[](https://github.com/ursacomputing/crossbow/actions/runs/6498522124/job/17650025469)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions/runs/6498521947/job/17650024480)|
|test-ubuntu-r-sanitizer|[](https://github.com/ursacomputing/crossbow/runs/17650024734)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759882061
Thanks for testing @jonkeane !
> but setting this to use more cores might be nice + possible then.
In the follow up #38236 I automatically set not_cran when we are in a dev build, which should make the dev experience on especially linux but overall much smoother as you also won't have to fiddle with allow lists or such to get the binaries.
I am a bit hesitant to remove the 2 core cap by default for release builds as we could still have to build from source on cran (BDR special checks? Condition for use of binaries ...). Mac users will get the cran binary anyway and through the allow list a majority of the linux users will also get binaries, so I think the reward doesn't justify the risk of a policy violation. (I also think that the 2 core think is a philosophical stance not necessarily limited to builds on cran, at least going by recent r dev ml threads...)
> but the linking step seemed to use all of my cores
:scream: I didn't change anything there so I am pretty sure that was already happening (on linuc), so it looks like you are right and the short burst is within tolerance?
> I think my local 11.7 build fails because it's mixing homebrew and bundled dependencies
Hm that should not be a problem normally though unless there is some version mismatch? The 10.13 runner also has brew and even has the deps installed but no problem arises... as this is a definitive dev use case I think it's ok to have people manually set the deps source var if the problem happens. Could you try with `ARROW_DEPENDENCY_SOURCE= BUNDELD`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356915238
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
Could you make it a separate build step and use an `if:`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756727930
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356107981
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
We do need this for the 10.13 runner, v4 uses node 20 which errors on 10.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356116256
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
Review Comment:
The macos source job builds libarrow and the package from source (see `LIBARROW_BINARY=false` ), same as the previously existing linux job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356126967
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
Review Comment:
Ah, all Apache Arrow C++ build logs are suppressed. Sorry. You're right.
Can we improve the "Install arrow from nightly repo" step name? It seems that we don't use "nightly repo". (We use "local repo" that has a source package created by the "Source Package" job.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356919383
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
Review Comment:
It might be nice to add a comment here that explains what `self-hosted` is. It's an M* mac with macOS 14 on it? Or 13? Or? Or, if this is documented somewhere else that's public link to that so that folks can figure out what this is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser merged PR #38195:
URL: https://github.com/apache/arrow/pull/38195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758615897
> @jonkeane @nealrichardson we default to setting libarrow_minimal in nixlibs/build_r_static.sh if NOT_CRAN is not set. What is the main motivation for that? Keeping build times on CRAN linux builds small?
IIRC it was both to keep build times low and to not enable features we knew to be problematic on some architecture. Problematic here being both not being able to be built as well as having issues while testing (snappy + sanitizer issues is one I remember battling, though I believe we actually special case these tests to not run on the sanitizer environments and IIUC snappy has finally fixed the one we were running into)
Our extended cran like CI should alert if we try that on a branch (though we would definitely want to make sure the sanitizer build runs there too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356117307
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
Review Comment:
https://github.com/ursacomputing/crossbow/actions/runs/6490158381/job/17625543388#step:13:42 vs https://github.com/ursacomputing/crossbow/actions/runs/6490158381/job/17626149544#step:17:57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758177568
On 10.13 with OpenSSL installed from source I get:
```
40%] Bundling /var/folders/3f/t1t4zyl96dxb_vk0brpggvyr0000gn/T/RtmpR3p8bL/file3fd466b3e60/release/libarrow_bundled_dependencies.a
/Library/Developer/CommandLineTools/usr/bin/libtool: archive member: /var/folders/3f/t1t4zyl96dxb_vk0brpggvyr0000gn/T/RtmpR3p8bL/file3fd466b3e60/thrift_ep-install/lib/libthrift.a() size too large (archive member extends past the end of the file)
[ 40%] Built target arrow_dependencies
make[2]: *** [src/arrow/CMakeFiles/arrow_bundled_dependencies_merge] Error 1
make[1]: *** [src/arrow/CMakeFiles/arrow_bundled_dependencies_merge.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758779911
Revision: 66401fc6f827aadac8857a5e0c042a05ffc46605
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b239f3d014](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b239f3d014)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6490158381/job/17625544237)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1757924832
Revision: 4ddbeaf8d5df518f9a5d68a31021fc1b55eaa248
Submitted crossbow builds: [ursacomputing/crossbow @ actions-28d06e21a7](https://github.com/ursacomputing/crossbow/branches/all?query=actions-28d06e21a7)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6484537350/job/17608619029)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758291773
Revision: 36537e250aa92fb47a4d6239bdd5cef26a99941f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-7be1754f57](https://github.com/ursacomputing/crossbow/branches/all?query=actions-7be1754f57)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6486687900/job/17615390539)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756786002
Revision: 10ea3a31a863f9bcc2f8a7c9c20e082d344d927a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-8ad4975dfd](https://github.com/ursacomputing/crossbow/branches/all?query=actions-8ad4975dfd)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6478051633/job/17589232792)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356087524
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
Review Comment:
What is the difference with this "Test macOS source build" and "r-packages"?
It seems that both of them use pre-built Apache Arrow C++ and build Apache Arrow R from source.
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
- name: Install sccache
+ if: matrix.platform.name == 'Linux'
shell: bash
run: |
arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies
- run: |
- sudo arrow/ci/scripts/r_install_system_dependencies.sh
- env:
- ARROW_GCS: "ON"
- ARROW_S3: "ON"
- ARROW_SOURCE_HOME: arrow
+ - name: Install R package system dependencies (Linux)
+ if: matrix.platform.name == 'Linux'
+ run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev
Review Comment:
Why do we want to use `apt-get install` directly instead of using `arrow/ci/scripts/r_install_system_dependencies.sh`?
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
It seems that we don't need this for macOS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758723261
> on the 10.13 runner it works but the features are not all there which is probably why that error didn't occur?
Fine with me for now! Am I correct that it's our plan to submit using the downloaded static libraries and use this bundled source build only as a backup? I think this PR gets the build close enough that if/when we do have to go that direction it's not an unreasonable amount of work to get it where we need it.
On x86/BigSur 11.7, I get:
```
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so':
dlopen(/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/00LOCK-r/00new/arrow/libs/arrow.so, 6): Symbol not found: __ZN4absl12lts_2023012512RFC3339_fullE
```
...but again, I'm happy to sort out details when it seems more likely we will have to go that route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356925975
##########
r/configure:
##########
@@ -121,16 +121,25 @@ else
ARROW_USE_PKG_CONFIG="false"
fi
-# find openssl on macos. macOS ships with libressl. openssl is installable
-# with brew, but it is generally not linked. We can over-ride this and find
-# openssl but setting OPENSSL_ROOT_DIR (which cmake will pick up later in
-# the installation process). FWIW, arrow's cmake process uses this
-# same process to find openssl, but doing it now allows us to catch it in
-# nixlibs.R and throw a nicer error.
+## Find openssl
+# Arrow's cmake process uses this same process to find openssl,
+# but doing it now allows us to catch it in
+# nixlibs.R and activate S3 and GCS support for the source build.
+
+# macOS ships with libressl. openssl is installable with brew, but it is
+# generally not linked. We can over-ride this and find
+# openssl by setting OPENSSL_ROOT_DIR (which cmake will pick up later in
+# the installation process).
if [ "${OPENSSL_ROOT_DIR}" = "" ] && brew --prefix openssl >/dev/null 2>&1; then
export OPENSSL_ROOT_DIR="`brew --prefix openssl`"
export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
fi
+# Look for openssl with pkg-config for non-brew sources(e.g. CRAN) and Linux
Review Comment:
This is very minor
> for non-brew sources(e.g. CRAN)
I'm not sure I follow the connection between non-brew and CRAN here. This would work for anyone who has openssl installed in a way that pkg-config can find it, yeah? It might be clearer to just leave it at that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356927635
##########
r/tools/nixlibs.R:
##########
@@ -56,7 +56,8 @@ try_download <- function(from_url, to_file, hush = quietly) {
}
not_cran <- env_is("NOT_CRAN", "true")
-if (not_cran) {
+# enable full featured builds for macOS in case of CRAN source builds.
Review Comment:
```suggestion
# enable full featured builds and binaries for macOS (or if the NOT_CRAN variable has been set)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356103164
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
- name: Install sccache
+ if: matrix.platform.name == 'Linux'
shell: bash
run: |
arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies
- run: |
- sudo arrow/ci/scripts/r_install_system_dependencies.sh
- env:
- ARROW_GCS: "ON"
- ARROW_S3: "ON"
- ARROW_SOURCE_HOME: arrow
+ - name: Install R package system dependencies (Linux)
+ if: matrix.platform.name == 'Linux'
+ run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev
Review Comment:
We don't need test bench etc. for this build so this is clearer and directly visible what gets installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357031841
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
Review Comment:
I explained it in the commit message which seems redundant now as it doesn't show up unless you check ^^
The use of cmake here is correct in the fail fast way and matches the actual build, which the test compilation would not be.
On the other hand the test compilation does not require cmake and is used to get the binary so we would need cmake just to check for the deps which is an unnecessary dependency.
So I removed the comment because there is no change necessary!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756861600
Revision: 9d4018473900bf9127c6f52fbd18ab2b0d47bf35
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3b4a613597](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3b4a613597)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/6478607840/job/17590674095)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357042587
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
Review Comment:
Aaaah ok, got it. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356121108
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
Ah, I wanted to say that we don't need to checkout apache/arrow on macOS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356117307
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
Review Comment:
https://github.com/ursacomputing/crossbow/actions/runs/6490158381/job/17625543388#step:13:42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758778285
@github-actions crossbow submit r-binary-packages
I have restored the minimal build for linux builds. On macos we now also set LIBARROW_MINIMAL to false to default to a full build.(:crossed_fingers: )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357003833
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
Review Comment:
This is just one machine, the 10.13 runner but self-hosted runners can be identified by multiple tags that are group in an array. For example we have both linux and macos arm64 runners so we have to use `["self-hosted", "arm64", "macos"]` for the m1. Adding "self-hosted" is recommended by gh and the default tag all self-hosted runners have to avoid any collision with official runners. See https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow#using-default-labels-to-route-jobs for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1756729634
Revision: 845e6070a0441487f7d4208d017c64c41b503581
Submitted crossbow builds: [ursacomputing/crossbow @ actions-8cb947668c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-8cb947668c)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/tree/actions-8cb947668c-github-r-binary-packages)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356990928
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
we'd have to update the macro to accept input for the if clause, which is of course possible but the macro is used across all gh build so I'd rather not at the moment, as it saves 1 minute or so tops...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759993147
> Could you try with ARROW_DEPENDENCY_SOURCE= BUNDELD?
I'll debug my source build separately from this PR
> I am a bit hesitant to remove the 2 core cap
That primarily affects the development experience, and we're not planning to have that be the primary means by which any MacOS developer gets an Arrow build. If it turns out that it is, we can fix that incrementally after the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1757974892
(After r-binary-packages passes I'll give a shot locally!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1357295123
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
Review Comment:
I think it's reasonable to have some comment here explaing why this step exists; however, I think a wholistic review of comments on the entire script is better suited to https://github.com/apache/arrow/pull/38236 (after rebase) (and the existing comment doesn't make this clear to me, at least).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758026419
Ok I'll give it a go!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758249582
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758611126
@jonkeane @nealrichardson we default to setting `libarrow_minimal` in nixlibs/build_r_static.sh if `NOT_CRAN` is not set. What is the main motivation for that? Keeping build times on CRAN linux builds small?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356928445
##########
r/tools/nixlibs.R:
##########
@@ -736,11 +738,6 @@ with_cloud_support <- function(env_var_list) {
cat("**** ", start_msg, " support ", msg, "; building with ", off_flags, "\n")
}
- # Check the features
- # This duplicates what we do with the test program above when we check
- # capabilities for using binaries. We could consider consolidating this
- # logic, though these use cmake in order to match exactly what we do in the
- # libarrow build, and maybe that increases the fidelity.
Review Comment:
Any reason that we're removing this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable cloud support by default [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1758408819
@paleolimbot on the 10.13 runner it works but the features are not all there qhich is probably why that error didn't occure? :/ (though gcs and s3 are 🎉 https://github.com/ursacomputing/crossbow/actions/runs/6486687900/job/17615389474#step:14:43833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #38195:
URL: https://github.com/apache/arrow/pull/38195#discussion_r1356126583
##########
dev/tasks/r/github.packages.yml:
##########
@@ -375,29 +375,30 @@ jobs:
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
print(arrow_info())
- #TODO test macos source build?
- test-linux-source:
+ test-source:
needs: source
- name: Test linux source build
- runs-on: ubuntu-latest
+ name: Test {{ '${{ matrix.platform.name }}' }} source build
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - {runs_on: "ubuntu-latest", name: "Linux"}
+ - {runs_on: ["self-hosted", "macos-10.13"] , name: "macOS"}
steps:
- name: Install R
+ if: matrix.platform.name == 'Linux'
uses: r-lib/actions/setup-r@v2
- with:
- install-r: false
{{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
Review Comment:
Oh I see what you mean... I can't think of an elgant fix at the moment feel free to push something :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1759981206
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38043: [R] Enable all features by default on macOS [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38195:
URL: https://github.com/apache/arrow/pull/38195#issuecomment-1765835741
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 4bf777a6ad716e3d60f96446b0c37bc4752990b4.
There was 1 benchmark result indicating a performance regression:
- Commit Run on `ursa-i9-9960x` at [2023-10-13 20:07:12Z](https://conbench.ursa.dev/compare/runs/0122c1ead02d439f97e113243f87d337...7d324417a2f14070bb6a05ce184ae250/)
- [`tpch` (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-02, scale_factor=1](https://conbench.ursa.dev/compare/benchmarks/065299181f9b7ae2800049db78cb496d...06529b7bb57f75b980004080aebbba00)
The [full Conbench report](https://github.com/apache/arrow/runs/17769564704) has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org