You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "marberts (via GitHub)" <gi...@apache.org> on 2026/03/19 03:08:40 UTC
[PR] 32123 expose azure blob filesystem [arrow]
marberts opened a new pull request, #49553:
URL: https://github.com/apache/arrow/pull/49553
### Rationale for this change
This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.
This would close #32123.
### What changes are included in this PR?
- A new class `AzureFileSystem` that's analogous to `S3FileSystem`/`GcsFileSystem`, along with a helper function `az_container()` that's analogous to `s3_bucket()`/`gcs_bucket()`.
- Updates to `src/filesystem.cpp` to interact with the machinery in `arrow/filesystem/azurefs.h`.
- Updates to the configuration and build scripts to support building with support for Azure.
- Updates to the vignettes on cloud storage and installation.
### Are these changes tested?
Yes. See `tests/testthat/test-azure.R`.
### Are there any user-facing changes?
Yes. There is a new function `az_container()`, serving the analogous role to `s3_bucket()`/`gcs_bucket()`, along with an R6 class `AzureFileSystem`, again serving the same role as `S3FileSystem` and `GcsFileSystem`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4187434514
Thanks for persisting with this, I see a lot of rounds of iteration there!
It's a shame it's such a pain to support on Windows, but I think there's value including it even it can only work on MacOS and Linux.
I'm not an expert in this stuff, and so I did a :robot: analysis, which I'll paste below so I can verify the underlying assumptions on which I'm basing my opinions are correct.
----------------------------------------------------------------------------------------------------------------
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4935144646
Thank you for your continued patience on this! I've rebased this against main and am running our extended CI to confirm if those failures we saw before were related to things on main from the last branching point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4426998962
@jonkeane Thanks for your patience with this PR. It took longer than expected to make the changes you suggested a few weeks ago. But I think we're ready for review now! Please advise on the next steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4716292650
Looking at the PR, for some reason the GitHub UI is showing `dev/tasks/r/github.packages.yml` as having been completely rewritten, which is going to make it hard to review. Perhaps tabs and spaces git switched or something? Would you mind giving this a rebase and resolving that so it's a bit easier to see what changes have been made there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3456511372
##########
dev/tasks/r/github.packages.yml:
##########
@@ -1,435 +1,437 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{% import 'macros.jinja' as macros with context %}
-
-{% set is_fork = macros.is_fork %}
-
-{{ macros.github_header() }}
-
-jobs:
- source:
- # This job will change the version to either the custom_version param or YMD format.
- # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
- name: Source Package
- runs-on: ubuntu-latest
- outputs:
- pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
- - name: Save Version
- id: save-version
- shell: bash
- run: |
- echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
-
- - uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
-
- - name: Build R source package
- shell: bash
- run: |
- cd arrow/r
- # Copy in the Arrow C++ source
- make sync-cpp
- R CMD build --no-build-vignettes .
-
- - name: Upload package artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg__src__contrib
- path: arrow/r/arrow_*.tar.gz
-
- macos-cpp:
- name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: macos-15-intel, arch: "x86_64" }
- - { runs_on: macos-14, arch: "arm64" }
- env:
- PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
- PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - name: Install Deps
- run: |
- brew install sccache ninja
- brew install openssl@3.0
- - name: Build libarrow
- shell: bash
- env:
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- MACOSX_DEPLOYMENT_TARGET: "11.6"
- ARROW_GCS: ON
- ARROW_DEPENDENCY_SOURCE: BUNDLED
- CMAKE_GENERATOR: Ninja
- LIBARROW_MINIMAL: false
- run: |
- sccache --start-server
- export EXTRA_CMAKE_FLAGS="-DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3.0)"
- cd arrow
- r/inst/build_arrow_static.sh
- - name: Bundle libarrow
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- linux-cpp:
- name: C++ Binary Linux {{ '${{ matrix.arch }}' }}
- runs-on: {{ '${{ matrix.runs-on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- include:
- - arch: x86_64
- runs-on: ubuntu-latest
- ubuntu: "22.04"
- - arch: arm64
- runs-on: ubuntu-24.04-arm
- ubuntu: "22.04"
- env:
- PKG_ID: r-libarrow-linux-{{ '${{ matrix.arch }}' }}
- PKG_FILE: r-libarrow-linux-{{ '${{ matrix.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- {{ macros.github_install_archery()|indent }}
- - name: Build libarrow
- shell: bash
- env:
- ARCH: {{ "${{ matrix.arch == 'x86_64' && 'amd64' || 'arm64/v8' }}" }}
- UBUNTU: {{ '${{ matrix.ubuntu }}' }}
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: |
- source arrow/ci/scripts/util_enable_core_dumps.sh
- archery docker run ubuntu-cpp-static
- - name: Bundle libarrow
- shell: bash
- run: |
- # These files were created by the docker user so we have to chown them
- sudo chown -R $USER:$USER arrow/r/libarrow
-
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- windows-cpp:
- name: C++ Binary Windows RTools (40 only)
- needs: source
- runs-on: windows-latest
- env:
- PKG_ID: r-libarrow-windows-x86_64
- PKG_FILE: r-libarrow-windows-x86_64-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- - run: git config --global core.autocrlf false
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - uses: r-lib/actions/setup-r@v2
- with:
- rtools-version: 40
- r-version: "4.0"
- Ncpus: 2
- - name: Install sccache
- shell: bash
- run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
- - name: Build Arrow C++ with rtools40
- shell: bash
- env:
- ARROW_HOME: "arrow"
- MINGW_ARCH: "ucrt64"
- RTOOLS_VERSION: 40
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: arrow/ci/scripts/r_windows_build.sh
- - name: Create Checksum
- shell: bash
- run: |
- cd build
- sha512sum ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- r-packages:
- needs: [source, windows-cpp, macos-cpp]
- name: {{ '${{ matrix.platform.name }} ${{ matrix.r_version }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: 'windows-latest', name: "Windows"}
- - { runs_on: macos-15-intel, name: "macOS x86_64"}
- - { runs_on: macos-14, name: "macOS arm64" }
- r_version: [oldrel, release]
- steps:
- - uses: r-lib/actions/setup-r@v2
- with:
- r-version: {{ '${{ matrix.r_version }}' }}
- {{ macros.github_setup_local_r_repo(false, true, true)|indent }}
- - name: Prepare Dependency Installation
- shell: bash
- run: |
- tar -xzf repo/src/contrib/arrow_*.tar.gz arrow/DESCRIPTION
- - name: Install dependencies
- uses: r-lib/actions/setup-r-dependencies@v2
- with:
- working-directory: 'arrow'
- extra-packages: cpp11
- - name: Set CRAN like openssl
- if: contains(matrix.platform.name, 'macOS')
- # The -E forwards the GITHUB_* envvars
- shell: sudo -E Rscript {0}
- run: |
- # get the mac-recipes version of openssl from CRAN
- source("https://mac.R-project.org/bin/install.R")
- install.libs("openssl")
-
- # override our cmakes default setting of the brew --prefix as root dir to avoid version conflicts.
- if (Sys.info()[["machine"]] == "arm64"){
- cat("OPENSSL_ROOT_DIR=/opt/R/arm64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- } else {
- cat("OPENSSL_ROOT_DIR=/opt/R/x86_64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- }
- - name: Build Binary
- id: build
- shell: Rscript {0}
- env:
- NOT_CRAN: "false" # actions/setup-r sets this implicitly
- ARROW_R_DEV: "true"
- LIBARROW_BINARY: "true" # has to be set as long as allowlist not updated
- LIBARROW_BUILD: "false"
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- run: |
- on_windows <- tolower(Sys.info()[["sysname"]]) == "windows"
-
- # Build
- Sys.setenv(MAKEFLAGS = paste0("-j", parallel::detectCores()))
- INSTALL_opts <- "--build"
- if (!on_windows) {
- # Windows doesn't support the --strip arg
- INSTALL_opts <- c(INSTALL_opts, "--strip")
- }
-
- # always remove arrow (mainly for self-hosted runners)
- try(remove.packages("arrow"), silent = TRUE)
-
- cat("Install arrow from dev repo.\n")
- install.packages(
- "arrow",
- type = "source",
- # The sub is necessary to prevent an error on windows.
- repos = sub("file://", "file:", getOption("arrow.dev_repo")),,
- INSTALL_opts = INSTALL_opts
- )
-
-
- # Test
- library(arrow)
- arrow_info()
- read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
-
- # encode contrib.url for artifact name
- cmd <- paste0(
- "path=",
- gsub(
- "/", "__",
- contrib.url("", type = "binary")
- ),
- "\n"
- )
- cat(cmd, file = Sys.getenv("GITHUB_OUTPUT"), append = TRUE)
-
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg{{ '${{ steps.build.outputs.path }}' }}
- path: arrow_*
- test-linux-binary:
- needs: [source, linux-cpp]
- name: Test binary {{ '${{ matrix.config.image }}' }} {{ '${{ matrix.config.runner }}' }}
- runs-on: {{ '${{ matrix.config.runner }}' }}
- container: {{ '${{ matrix.config.image }}' }}
- strategy:
- fail-fast: false
- matrix:
- config:
- # If libarrow_binary is unset, we're testing that we're automatically
- # choosing a binary on this OS. If libarrow_binary is TRUE, we're on
- # an OS that is not in the allowlist, so we have to opt-in to use the
- # binary. Other env vars used in r_docker_configure.sh can be added
- # here and wired up in the later steps.
- # x86_64 tests
- - {image: "rhub/ubuntu-clang", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
- # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- - {image: "rhub/ubuntu-release", runner: "ubuntu-latest"} # currently ubuntu-24.04
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-latest"}
- # ARM64 tests
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-24.04-arm"}
- steps:
- # Get the arrow checkout just for the docker config scripts
- # Don't need submodules for this (hence false arg to macro): they fail on
- # actions/checkout for some reason in this context
- {{ macros.github_checkout_arrow(1, false, '3')|indent }}
-
- - name: Install system requirements
- env:
- ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
- shell: bash
- run: |
- # Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
- echo "${RPREFIX}/bin" >> $GITHUB_PATH
- export PATH="${RPREFIX}/bin:${PATH}"
- cd arrow && ARROW_SOURCE_HOME=$(pwd) ./ci/scripts/r_docker_configure.sh
- {{ macros.github_setup_local_r_repo(true, false)|indent }}
- - name: Install arrow from our repo
- env:
- ARROW_R_DEV: "TRUE"
- LIBARROW_BUILD: "FALSE"
- LIBARROW_BINARY: {{ '${{ matrix.config.libarrow_binary }}' }}
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- shell: bash
- run: |
- Rscript -e '
- {{ macros.github_test_r_src_pkg()|indent(8) }}
- '
- test-source:
- needs: source
- name: Test {{ '${{ matrix.platform.name }}' }} source build
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - {runs_on: "ubuntu-latest", name: "Linux"}
- - {runs_on: "macos-15-intel" , name: "macOS"}
- steps:
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- {{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- - name: Install sccache
- if: matrix.platform.name == 'Linux'
- shell: bash
- run: |
- arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies (Linux)
- if: matrix.platform.name == 'Linux'
- run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev libuv1-dev
- - name: Install R package system dependencies (macOS)
- if: matrix.platform.name == 'macOS'
- run: brew install sccache openssl curl libuv
- - name: Remove arrow/
- run: |
- rm -rf arrow/
- - name: Enable parallel build
- run: |
- cores=`nproc || sysctl -n hw.logicalcpu`
- echo "MAKEFLAGS=-j$cores" >> $GITHUB_ENV
- - name: Install arrow source package
- env:
- # Test source build so be sure not to download a binary
- LIBARROW_BINARY: "FALSE"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- - name: Retry with verbosity if that failed
- if: failure()
- env:
- LIBARROW_BINARY: "FALSE"
- ARROW_R_DEV: "TRUE"
- CMAKE_FIND_DEBUG_MODE: "ON"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- upload-binaries:
- # Only upload binaries if all tests pass.
- needs: [r-packages, test-source, test-linux-binary]
- name: Upload artifacts
- runs-on: ubuntu-latest
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- - name: Download Artifacts
- uses: actions/download-artifact@v4
- with:
- path: artifacts
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
- - name: Move libarrow artifacts
- run: |
- mkdir -p binaries/
- mv artifacts/r-libarrow-*/* binaries/
- - name: Rename artifacts
- shell: Rscript {0}
- run: |
- file_paths <- list.files("artifacts", include.dirs = FALSE, recursive = TRUE)
- new_names <- file.path("binaries", sub("/", "__", file_paths))
- dir.create("binaries", showWarnings = FALSE)
- file.copy(file.path("artifacts", file_paths), new_names)
-
- {{ macros.github_upload_releases("binaries/r-*")|indent }}
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{% set is_fork = macros.is_fork %}
+
+{{ macros.github_header() }}
+
+jobs:
+ source:
+ # This job will change the version to either the custom_version param or YMD format.
+ # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
+ name: Source Package
+ runs-on: ubuntu-latest
+ outputs:
+ pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
+ steps:
+ {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
+ - name: Save Version
+ id: save-version
+ shell: bash
+ run: |
+ echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
+
+ - uses: r-lib/actions/setup-r@v2
+ with:
+ install-r: true
+
+ - name: Build R source package
+ shell: bash
+ run: |
+ cd arrow/r
+ # Copy in the Arrow C++ source
+ make sync-cpp
+ R CMD build --no-build-vignettes .
+
+ - name: Upload package artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: r-pkg__src__contrib
+ path: arrow/r/arrow_*.tar.gz
+
+ macos-cpp:
+ name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ needs: source
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: macos-15-intel, arch: "x86_64" }
+ - { runs_on: macos-14, arch: "arm64" }
+ env:
+ PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
+ PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
+ steps:
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
+ - name: Install Deps
+ run: |
+ brew install sccache ninja
+ brew install openssl@3.0
+ brew install libxml2
+ - name: Build libarrow
+ shell: bash
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
+ MACOSX_DEPLOYMENT_TARGET: "11.6"
+ ARROW_GCS: ON
+ ARROW_AZURE: ON
Review Comment:
Second change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4775094869
I think I've messed up the history on my fork as I'm getting a lot of unexpected conflicts when I try to rebase. I'll need to look at this again later. Sorry for the mix up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4827221656
> Thank you for persisting with this. I know rebases can be a pain when the go sideways! A few questions / comments around installation + confirming that we have the right dependencies. The operative code looks ok, though I did not dig too too deeply on it. Do you happen to have a public azure bucket that I could point this at and try it out locally?
>
> I'm also going to run crossbow to run the tests.
Awesome, thanks for taking a look.
Unfortunately I don't have a public container for Azure. If you want to try it locally you can use azurite: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite. This is what we're using in the test suite. The only other tests I did was to try connecting to a made up storage container to check that I was getting the expected errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3488929783
##########
r/tools/nixlibs.R:
##########
@@ -942,16 +953,22 @@ with_cloud_support <- function(env_var_list) {
print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
} else if (!cmake_find_package("OpenSSL", "1.0.2", env_var_list)) {
print_warning("requires version >= 1.0.2 of openssl-devel (rpm), libssl-dev (deb), or openssl (brew)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
+ } else if (!cmake_find_package("libxml2", NULL, env_var_list)) {
+ print_warning("requires libxml2-devel (rpm), or libxml2-dev (deb), libxml2 (brew)", "AZURE")
+ arrow_azure <- FALSE
}
Review Comment:
Yes, good idea. Done in [1ea9150](https://github.com/apache/arrow/pull/49553/commits/1ea9150cc966af2a1fe066a4091ca2005353beb6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4820050909
I figured out the rebase and upstream changes are now correctly reflected in this PR.
The big diff for github.packages.yml was due to a change in line ending from my editor; it's all good now and the diff makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4154506562
Some of those CI failures are from things which have been fixed on main - you'll need to rebase your branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4713863158
@jonkeane @thisisnic Just a gentle nudge/reminder. I realize this is a big PR. Please let me know if any parts are unclear, and especially if the changes to the documentation clearly reflect the changes and limitations of using Azure for the R package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3487856362
##########
r/tools/nixlibs.R:
##########
@@ -942,16 +953,22 @@ with_cloud_support <- function(env_var_list) {
print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
} else if (!cmake_find_package("OpenSSL", "1.0.2", env_var_list)) {
print_warning("requires version >= 1.0.2 of openssl-devel (rpm), libssl-dev (deb), or openssl (brew)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
+ } else if (!cmake_find_package("libxml2", NULL, env_var_list)) {
+ print_warning("requires libxml2-devel (rpm), or libxml2-dev (deb), libxml2 (brew)", "AZURE")
+ arrow_azure <- FALSE
}
Review Comment:
If I'm following correctly, this check will run regardless of if someone is trying to setup Azure. Is it possible to make this only run if someone has requested Azure?
##########
r/configure.win:
##########
@@ -67,6 +67,7 @@ function configure_binaries() {
# pkg-config --libs libcurl
GCS_LIBS="-lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 \
-lz -lws2_32 -lnghttp2 -ldbghelp"
+ # AZURE_LIBS="-lcurl -lssl -lxml2"
Review Comment:
These are holdovers from when you were trying these on Windows, yeah?
##########
r/tools/nixlibs.R:
##########
@@ -942,16 +953,22 @@ with_cloud_support <- function(env_var_list) {
print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
Review Comment:
Is libcurl and openssl necessary for Azure as well? These additions seem to indicate yes, but I want to confirm that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4935139468
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4938864227
@github-actions crossbow submit r-binary-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane merged PR #49553:
URL: https://github.com/apache/arrow/pull/49553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4131460847
```
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23576717645
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3488373365
##########
r/configure.win:
##########
@@ -67,6 +67,7 @@ function configure_binaries() {
# pkg-config --libs libcurl
GCS_LIBS="-lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 \
-lz -lws2_32 -lnghttp2 -ldbghelp"
+ # AZURE_LIBS="-lcurl -lssl -lxml2"
Review Comment:
Yes, I left these commented out in case progress gets made in the future around the incompatibility with mingw and the azure SDK stuff. Happy to either remove them or make it clearer in the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4940950065
Awesome! Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3488403311
##########
r/tools/nixlibs.R:
##########
@@ -942,16 +953,22 @@ with_cloud_support <- function(env_var_list) {
print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
+ arrow_azure <- FALSE
Review Comment:
Although I did not test without openssl and libcurl, the azure c++ sdk requires them so I'm pretty sure they're needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3456511372
##########
dev/tasks/r/github.packages.yml:
##########
@@ -1,435 +1,437 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{% import 'macros.jinja' as macros with context %}
-
-{% set is_fork = macros.is_fork %}
-
-{{ macros.github_header() }}
-
-jobs:
- source:
- # This job will change the version to either the custom_version param or YMD format.
- # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
- name: Source Package
- runs-on: ubuntu-latest
- outputs:
- pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
- - name: Save Version
- id: save-version
- shell: bash
- run: |
- echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
-
- - uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
-
- - name: Build R source package
- shell: bash
- run: |
- cd arrow/r
- # Copy in the Arrow C++ source
- make sync-cpp
- R CMD build --no-build-vignettes .
-
- - name: Upload package artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg__src__contrib
- path: arrow/r/arrow_*.tar.gz
-
- macos-cpp:
- name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: macos-15-intel, arch: "x86_64" }
- - { runs_on: macos-14, arch: "arm64" }
- env:
- PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
- PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - name: Install Deps
- run: |
- brew install sccache ninja
- brew install openssl@3.0
- - name: Build libarrow
- shell: bash
- env:
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- MACOSX_DEPLOYMENT_TARGET: "11.6"
- ARROW_GCS: ON
- ARROW_DEPENDENCY_SOURCE: BUNDLED
- CMAKE_GENERATOR: Ninja
- LIBARROW_MINIMAL: false
- run: |
- sccache --start-server
- export EXTRA_CMAKE_FLAGS="-DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3.0)"
- cd arrow
- r/inst/build_arrow_static.sh
- - name: Bundle libarrow
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- linux-cpp:
- name: C++ Binary Linux {{ '${{ matrix.arch }}' }}
- runs-on: {{ '${{ matrix.runs-on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- include:
- - arch: x86_64
- runs-on: ubuntu-latest
- ubuntu: "22.04"
- - arch: arm64
- runs-on: ubuntu-24.04-arm
- ubuntu: "22.04"
- env:
- PKG_ID: r-libarrow-linux-{{ '${{ matrix.arch }}' }}
- PKG_FILE: r-libarrow-linux-{{ '${{ matrix.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- {{ macros.github_install_archery()|indent }}
- - name: Build libarrow
- shell: bash
- env:
- ARCH: {{ "${{ matrix.arch == 'x86_64' && 'amd64' || 'arm64/v8' }}" }}
- UBUNTU: {{ '${{ matrix.ubuntu }}' }}
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: |
- source arrow/ci/scripts/util_enable_core_dumps.sh
- archery docker run ubuntu-cpp-static
- - name: Bundle libarrow
- shell: bash
- run: |
- # These files were created by the docker user so we have to chown them
- sudo chown -R $USER:$USER arrow/r/libarrow
-
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- windows-cpp:
- name: C++ Binary Windows RTools (40 only)
- needs: source
- runs-on: windows-latest
- env:
- PKG_ID: r-libarrow-windows-x86_64
- PKG_FILE: r-libarrow-windows-x86_64-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- - run: git config --global core.autocrlf false
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - uses: r-lib/actions/setup-r@v2
- with:
- rtools-version: 40
- r-version: "4.0"
- Ncpus: 2
- - name: Install sccache
- shell: bash
- run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
- - name: Build Arrow C++ with rtools40
- shell: bash
- env:
- ARROW_HOME: "arrow"
- MINGW_ARCH: "ucrt64"
- RTOOLS_VERSION: 40
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: arrow/ci/scripts/r_windows_build.sh
- - name: Create Checksum
- shell: bash
- run: |
- cd build
- sha512sum ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- r-packages:
- needs: [source, windows-cpp, macos-cpp]
- name: {{ '${{ matrix.platform.name }} ${{ matrix.r_version }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: 'windows-latest', name: "Windows"}
- - { runs_on: macos-15-intel, name: "macOS x86_64"}
- - { runs_on: macos-14, name: "macOS arm64" }
- r_version: [oldrel, release]
- steps:
- - uses: r-lib/actions/setup-r@v2
- with:
- r-version: {{ '${{ matrix.r_version }}' }}
- {{ macros.github_setup_local_r_repo(false, true, true)|indent }}
- - name: Prepare Dependency Installation
- shell: bash
- run: |
- tar -xzf repo/src/contrib/arrow_*.tar.gz arrow/DESCRIPTION
- - name: Install dependencies
- uses: r-lib/actions/setup-r-dependencies@v2
- with:
- working-directory: 'arrow'
- extra-packages: cpp11
- - name: Set CRAN like openssl
- if: contains(matrix.platform.name, 'macOS')
- # The -E forwards the GITHUB_* envvars
- shell: sudo -E Rscript {0}
- run: |
- # get the mac-recipes version of openssl from CRAN
- source("https://mac.R-project.org/bin/install.R")
- install.libs("openssl")
-
- # override our cmakes default setting of the brew --prefix as root dir to avoid version conflicts.
- if (Sys.info()[["machine"]] == "arm64"){
- cat("OPENSSL_ROOT_DIR=/opt/R/arm64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- } else {
- cat("OPENSSL_ROOT_DIR=/opt/R/x86_64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- }
- - name: Build Binary
- id: build
- shell: Rscript {0}
- env:
- NOT_CRAN: "false" # actions/setup-r sets this implicitly
- ARROW_R_DEV: "true"
- LIBARROW_BINARY: "true" # has to be set as long as allowlist not updated
- LIBARROW_BUILD: "false"
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- run: |
- on_windows <- tolower(Sys.info()[["sysname"]]) == "windows"
-
- # Build
- Sys.setenv(MAKEFLAGS = paste0("-j", parallel::detectCores()))
- INSTALL_opts <- "--build"
- if (!on_windows) {
- # Windows doesn't support the --strip arg
- INSTALL_opts <- c(INSTALL_opts, "--strip")
- }
-
- # always remove arrow (mainly for self-hosted runners)
- try(remove.packages("arrow"), silent = TRUE)
-
- cat("Install arrow from dev repo.\n")
- install.packages(
- "arrow",
- type = "source",
- # The sub is necessary to prevent an error on windows.
- repos = sub("file://", "file:", getOption("arrow.dev_repo")),,
- INSTALL_opts = INSTALL_opts
- )
-
-
- # Test
- library(arrow)
- arrow_info()
- read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
-
- # encode contrib.url for artifact name
- cmd <- paste0(
- "path=",
- gsub(
- "/", "__",
- contrib.url("", type = "binary")
- ),
- "\n"
- )
- cat(cmd, file = Sys.getenv("GITHUB_OUTPUT"), append = TRUE)
-
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg{{ '${{ steps.build.outputs.path }}' }}
- path: arrow_*
- test-linux-binary:
- needs: [source, linux-cpp]
- name: Test binary {{ '${{ matrix.config.image }}' }} {{ '${{ matrix.config.runner }}' }}
- runs-on: {{ '${{ matrix.config.runner }}' }}
- container: {{ '${{ matrix.config.image }}' }}
- strategy:
- fail-fast: false
- matrix:
- config:
- # If libarrow_binary is unset, we're testing that we're automatically
- # choosing a binary on this OS. If libarrow_binary is TRUE, we're on
- # an OS that is not in the allowlist, so we have to opt-in to use the
- # binary. Other env vars used in r_docker_configure.sh can be added
- # here and wired up in the later steps.
- # x86_64 tests
- - {image: "rhub/ubuntu-clang", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
- # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- - {image: "rhub/ubuntu-release", runner: "ubuntu-latest"} # currently ubuntu-24.04
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-latest"}
- # ARM64 tests
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-24.04-arm"}
- steps:
- # Get the arrow checkout just for the docker config scripts
- # Don't need submodules for this (hence false arg to macro): they fail on
- # actions/checkout for some reason in this context
- {{ macros.github_checkout_arrow(1, false, '3')|indent }}
-
- - name: Install system requirements
- env:
- ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
- shell: bash
- run: |
- # Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
- echo "${RPREFIX}/bin" >> $GITHUB_PATH
- export PATH="${RPREFIX}/bin:${PATH}"
- cd arrow && ARROW_SOURCE_HOME=$(pwd) ./ci/scripts/r_docker_configure.sh
- {{ macros.github_setup_local_r_repo(true, false)|indent }}
- - name: Install arrow from our repo
- env:
- ARROW_R_DEV: "TRUE"
- LIBARROW_BUILD: "FALSE"
- LIBARROW_BINARY: {{ '${{ matrix.config.libarrow_binary }}' }}
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- shell: bash
- run: |
- Rscript -e '
- {{ macros.github_test_r_src_pkg()|indent(8) }}
- '
- test-source:
- needs: source
- name: Test {{ '${{ matrix.platform.name }}' }} source build
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - {runs_on: "ubuntu-latest", name: "Linux"}
- - {runs_on: "macos-15-intel" , name: "macOS"}
- steps:
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- {{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- - name: Install sccache
- if: matrix.platform.name == 'Linux'
- shell: bash
- run: |
- arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies (Linux)
- if: matrix.platform.name == 'Linux'
- run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev libuv1-dev
- - name: Install R package system dependencies (macOS)
- if: matrix.platform.name == 'macOS'
- run: brew install sccache openssl curl libuv
- - name: Remove arrow/
- run: |
- rm -rf arrow/
- - name: Enable parallel build
- run: |
- cores=`nproc || sysctl -n hw.logicalcpu`
- echo "MAKEFLAGS=-j$cores" >> $GITHUB_ENV
- - name: Install arrow source package
- env:
- # Test source build so be sure not to download a binary
- LIBARROW_BINARY: "FALSE"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- - name: Retry with verbosity if that failed
- if: failure()
- env:
- LIBARROW_BINARY: "FALSE"
- ARROW_R_DEV: "TRUE"
- CMAKE_FIND_DEBUG_MODE: "ON"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- upload-binaries:
- # Only upload binaries if all tests pass.
- needs: [r-packages, test-source, test-linux-binary]
- name: Upload artifacts
- runs-on: ubuntu-latest
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- - name: Download Artifacts
- uses: actions/download-artifact@v4
- with:
- path: artifacts
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
- - name: Move libarrow artifacts
- run: |
- mkdir -p binaries/
- mv artifacts/r-libarrow-*/* binaries/
- - name: Rename artifacts
- shell: Rscript {0}
- run: |
- file_paths <- list.files("artifacts", include.dirs = FALSE, recursive = TRUE)
- new_names <- file.path("binaries", sub("/", "__", file_paths))
- dir.create("binaries", showWarnings = FALSE)
- file.copy(file.path("artifacts", file_paths), new_names)
-
- {{ macros.github_upload_releases("binaries/r-*")|indent }}
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{% set is_fork = macros.is_fork %}
+
+{{ macros.github_header() }}
+
+jobs:
+ source:
+ # This job will change the version to either the custom_version param or YMD format.
+ # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
+ name: Source Package
+ runs-on: ubuntu-latest
+ outputs:
+ pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
+ steps:
+ {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
+ - name: Save Version
+ id: save-version
+ shell: bash
+ run: |
+ echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
+
+ - uses: r-lib/actions/setup-r@v2
+ with:
+ install-r: true
+
+ - name: Build R source package
+ shell: bash
+ run: |
+ cd arrow/r
+ # Copy in the Arrow C++ source
+ make sync-cpp
+ R CMD build --no-build-vignettes .
+
+ - name: Upload package artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: r-pkg__src__contrib
+ path: arrow/r/arrow_*.tar.gz
+
+ macos-cpp:
+ name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ needs: source
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: macos-15-intel, arch: "x86_64" }
+ - { runs_on: macos-14, arch: "arm64" }
+ env:
+ PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
+ PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
+ steps:
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
+ - name: Install Deps
+ run: |
+ brew install sccache ninja
+ brew install openssl@3.0
+ brew install libxml2
+ - name: Build libarrow
+ shell: bash
+ env:
+ {{ macros.github_set_sccache_envvars()|indent(8) }}
+ MACOSX_DEPLOYMENT_TARGET: "11.6"
+ ARROW_GCS: ON
+ ARROW_AZURE: ON
Review Comment:
Second change: `ARROW_AZURE: ON`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 32123 expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4087402327
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4114912679
It looks like the failing CI job is from a warning in R CMD check about a non-API call to a C function, due to all the changes in R-devel. I don't think this is due to our PR.
At this point, I'm fairly confident that we've implemented the Azure features correctly. The only thing I'm not sure about is the setup in `configure`, `configure.win`, and `tools/nixlibs.R`. The Azure C++ SDK used by Arrow requires `libxml2`, which is a new system dependency for this package. I don't fully understand how these configuration scripts work, so I wouldn't be surprised if they're not updated correctly to reflect the new system dependency.
@jonkeane, @thisisnic, @assignUser Please let me or @Collinbrown95 know if there are any changes needed to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4120956713
Great, thanks @thisisnic!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4273866934
> @jonkeane I might have misunderstood your comment in my initial response. Is your suggestion to build the R Arrow library with -DARROW_AZURE=ON with MSVC and host it outside of CRAN (possibly circumventing the MinGW restriction discussed earlier in this thread)?
Yeah, for example, for GCS we encourage folks to use the binaries hosted on R-universe and disable that on CRAN
> If this is your suggestion, could you kindly share a bit more information about how this would work?
>
> For some extra context, my current understanding is that there are low-level C++ ABI compatibility issues between binaries built with MSVC and binaries built with MinGW, which (I think) implies that an MSVC compiled binary would not interoperate with the rest of the R ecosystem (which I believe is built with the MinGW toolchain).
>
> That said, I am by no means an expert in R/Windows internals. I would be happy to learn more if you know of a way for R packages to interoperate with MSVC-built binaries without ABI issues.
Ah, that might make something like we do for GCS not a possibility. I'm not familiar enough with the MSVC / MinGW ecosystem to know if this is something that must be built within that ecosystem to even be used, or something we could build separately and then link to (or some other way of getting this to work!).
I did some exploration to see if anyone else is using and I couldn't find any (besides references [in arrow](https://github.com/search?q=org%3Acran%20azure-sdk-for-cpp&type=code)). Notable, even the official Azure packages don't rely on the azure sdk: https://github.com/Azure/AzureR
> The core issue seems to be an incompatibility between Windows Implementation Libraries (WIL) and MinGW (this is https://github.com/microsoft/wil/issues/117). Our understanding is that resolving this issue would resolve many of the downstream issues in building [azure-sdk-for-cpp](https://github.com/Azure/azure-sdk-for-cpp) with MinGW, which would open the door for a fully functioning build of Arrow on Windows with Azure support enabled.
>
> The most recent discussion on this topic I could find is https://github.com/microsoft/wil/pull/467. I can create an issue in either the WIL/MinGW repositories to flag that this compatibility issue is still relevant, but I get the impression that the scope/timeline of this work goes beyond what this PR is trying to achieve.
Ah ok, yeah those look like they aren't really being actively pursued right now so unlikely to change in the future 😕
> Based on our results from building azure-sdk-for-cpp from source, my understanding is that a limited version of Azure support for Arrow ought to be possible on Windows, but the Azure Identity features won't work on the Windows build due to the abovementioned difficulties of getting azure-identity-cpp to build with MinGW. I.e., users should still be able to use connection strings or SAS tokens in this scenario.
>
> If limited Azure support on Windows is preferable to disabling it, we will continue investigating what would be involved to disable the Azure Identity features and otherwise keep Azure support on the Windows build.
Yeah, having some support would be better than none. And overall, I do think having support on macos + linux alone is ok. Not ideal, but having that support there might give us signal that we need to come back and figure out how to get the full windows story settled. We should be open about that in the docs — and also follow GCS where we build this in our own builds + on R-universe but default it to off on CRAN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4938878844
Revision: 9dd6793d3749da8aa38335c6e73d7518aedaf691
Submitted crossbow builds: [ursacomputing/crossbow @ actions-a47c3da823](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a47c3da823)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/29118720629/job/86448589178)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4774922798
I'm not sure why the diff is showing that all of `dev/tasks/r/github.packages.yml` has changed. My last commit took the content of that file on the main branch and added the two lines I annotated above, and you can see there's no changes in that commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r2975018091
##########
r/configure:
##########
Review Comment:
TODO: `-lcurl -lssl` are already passed on this line, drop from line 364.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4149157998
```
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23698292050
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4826083402
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3456510720
##########
dev/tasks/r/github.packages.yml:
##########
@@ -1,435 +1,437 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{% import 'macros.jinja' as macros with context %}
-
-{% set is_fork = macros.is_fork %}
-
-{{ macros.github_header() }}
-
-jobs:
- source:
- # This job will change the version to either the custom_version param or YMD format.
- # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
- name: Source Package
- runs-on: ubuntu-latest
- outputs:
- pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
- - name: Save Version
- id: save-version
- shell: bash
- run: |
- echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
-
- - uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
-
- - name: Build R source package
- shell: bash
- run: |
- cd arrow/r
- # Copy in the Arrow C++ source
- make sync-cpp
- R CMD build --no-build-vignettes .
-
- - name: Upload package artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg__src__contrib
- path: arrow/r/arrow_*.tar.gz
-
- macos-cpp:
- name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: macos-15-intel, arch: "x86_64" }
- - { runs_on: macos-14, arch: "arm64" }
- env:
- PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
- PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - name: Install Deps
- run: |
- brew install sccache ninja
- brew install openssl@3.0
- - name: Build libarrow
- shell: bash
- env:
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- MACOSX_DEPLOYMENT_TARGET: "11.6"
- ARROW_GCS: ON
- ARROW_DEPENDENCY_SOURCE: BUNDLED
- CMAKE_GENERATOR: Ninja
- LIBARROW_MINIMAL: false
- run: |
- sccache --start-server
- export EXTRA_CMAKE_FLAGS="-DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3.0)"
- cd arrow
- r/inst/build_arrow_static.sh
- - name: Bundle libarrow
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- linux-cpp:
- name: C++ Binary Linux {{ '${{ matrix.arch }}' }}
- runs-on: {{ '${{ matrix.runs-on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- include:
- - arch: x86_64
- runs-on: ubuntu-latest
- ubuntu: "22.04"
- - arch: arm64
- runs-on: ubuntu-24.04-arm
- ubuntu: "22.04"
- env:
- PKG_ID: r-libarrow-linux-{{ '${{ matrix.arch }}' }}
- PKG_FILE: r-libarrow-linux-{{ '${{ matrix.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- {{ macros.github_install_archery()|indent }}
- - name: Build libarrow
- shell: bash
- env:
- ARCH: {{ "${{ matrix.arch == 'x86_64' && 'amd64' || 'arm64/v8' }}" }}
- UBUNTU: {{ '${{ matrix.ubuntu }}' }}
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: |
- source arrow/ci/scripts/util_enable_core_dumps.sh
- archery docker run ubuntu-cpp-static
- - name: Bundle libarrow
- shell: bash
- run: |
- # These files were created by the docker user so we have to chown them
- sudo chown -R $USER:$USER arrow/r/libarrow
-
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- windows-cpp:
- name: C++ Binary Windows RTools (40 only)
- needs: source
- runs-on: windows-latest
- env:
- PKG_ID: r-libarrow-windows-x86_64
- PKG_FILE: r-libarrow-windows-x86_64-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- - run: git config --global core.autocrlf false
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - uses: r-lib/actions/setup-r@v2
- with:
- rtools-version: 40
- r-version: "4.0"
- Ncpus: 2
- - name: Install sccache
- shell: bash
- run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
- - name: Build Arrow C++ with rtools40
- shell: bash
- env:
- ARROW_HOME: "arrow"
- MINGW_ARCH: "ucrt64"
- RTOOLS_VERSION: 40
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: arrow/ci/scripts/r_windows_build.sh
- - name: Create Checksum
- shell: bash
- run: |
- cd build
- sha512sum ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- r-packages:
- needs: [source, windows-cpp, macos-cpp]
- name: {{ '${{ matrix.platform.name }} ${{ matrix.r_version }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: 'windows-latest', name: "Windows"}
- - { runs_on: macos-15-intel, name: "macOS x86_64"}
- - { runs_on: macos-14, name: "macOS arm64" }
- r_version: [oldrel, release]
- steps:
- - uses: r-lib/actions/setup-r@v2
- with:
- r-version: {{ '${{ matrix.r_version }}' }}
- {{ macros.github_setup_local_r_repo(false, true, true)|indent }}
- - name: Prepare Dependency Installation
- shell: bash
- run: |
- tar -xzf repo/src/contrib/arrow_*.tar.gz arrow/DESCRIPTION
- - name: Install dependencies
- uses: r-lib/actions/setup-r-dependencies@v2
- with:
- working-directory: 'arrow'
- extra-packages: cpp11
- - name: Set CRAN like openssl
- if: contains(matrix.platform.name, 'macOS')
- # The -E forwards the GITHUB_* envvars
- shell: sudo -E Rscript {0}
- run: |
- # get the mac-recipes version of openssl from CRAN
- source("https://mac.R-project.org/bin/install.R")
- install.libs("openssl")
-
- # override our cmakes default setting of the brew --prefix as root dir to avoid version conflicts.
- if (Sys.info()[["machine"]] == "arm64"){
- cat("OPENSSL_ROOT_DIR=/opt/R/arm64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- } else {
- cat("OPENSSL_ROOT_DIR=/opt/R/x86_64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- }
- - name: Build Binary
- id: build
- shell: Rscript {0}
- env:
- NOT_CRAN: "false" # actions/setup-r sets this implicitly
- ARROW_R_DEV: "true"
- LIBARROW_BINARY: "true" # has to be set as long as allowlist not updated
- LIBARROW_BUILD: "false"
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- run: |
- on_windows <- tolower(Sys.info()[["sysname"]]) == "windows"
-
- # Build
- Sys.setenv(MAKEFLAGS = paste0("-j", parallel::detectCores()))
- INSTALL_opts <- "--build"
- if (!on_windows) {
- # Windows doesn't support the --strip arg
- INSTALL_opts <- c(INSTALL_opts, "--strip")
- }
-
- # always remove arrow (mainly for self-hosted runners)
- try(remove.packages("arrow"), silent = TRUE)
-
- cat("Install arrow from dev repo.\n")
- install.packages(
- "arrow",
- type = "source",
- # The sub is necessary to prevent an error on windows.
- repos = sub("file://", "file:", getOption("arrow.dev_repo")),,
- INSTALL_opts = INSTALL_opts
- )
-
-
- # Test
- library(arrow)
- arrow_info()
- read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
-
- # encode contrib.url for artifact name
- cmd <- paste0(
- "path=",
- gsub(
- "/", "__",
- contrib.url("", type = "binary")
- ),
- "\n"
- )
- cat(cmd, file = Sys.getenv("GITHUB_OUTPUT"), append = TRUE)
-
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg{{ '${{ steps.build.outputs.path }}' }}
- path: arrow_*
- test-linux-binary:
- needs: [source, linux-cpp]
- name: Test binary {{ '${{ matrix.config.image }}' }} {{ '${{ matrix.config.runner }}' }}
- runs-on: {{ '${{ matrix.config.runner }}' }}
- container: {{ '${{ matrix.config.image }}' }}
- strategy:
- fail-fast: false
- matrix:
- config:
- # If libarrow_binary is unset, we're testing that we're automatically
- # choosing a binary on this OS. If libarrow_binary is TRUE, we're on
- # an OS that is not in the allowlist, so we have to opt-in to use the
- # binary. Other env vars used in r_docker_configure.sh can be added
- # here and wired up in the later steps.
- # x86_64 tests
- - {image: "rhub/ubuntu-clang", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
- # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- - {image: "rhub/ubuntu-release", runner: "ubuntu-latest"} # currently ubuntu-24.04
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-latest"}
- # ARM64 tests
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-24.04-arm"}
- steps:
- # Get the arrow checkout just for the docker config scripts
- # Don't need submodules for this (hence false arg to macro): they fail on
- # actions/checkout for some reason in this context
- {{ macros.github_checkout_arrow(1, false, '3')|indent }}
-
- - name: Install system requirements
- env:
- ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
- shell: bash
- run: |
- # Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
- echo "${RPREFIX}/bin" >> $GITHUB_PATH
- export PATH="${RPREFIX}/bin:${PATH}"
- cd arrow && ARROW_SOURCE_HOME=$(pwd) ./ci/scripts/r_docker_configure.sh
- {{ macros.github_setup_local_r_repo(true, false)|indent }}
- - name: Install arrow from our repo
- env:
- ARROW_R_DEV: "TRUE"
- LIBARROW_BUILD: "FALSE"
- LIBARROW_BINARY: {{ '${{ matrix.config.libarrow_binary }}' }}
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- shell: bash
- run: |
- Rscript -e '
- {{ macros.github_test_r_src_pkg()|indent(8) }}
- '
- test-source:
- needs: source
- name: Test {{ '${{ matrix.platform.name }}' }} source build
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - {runs_on: "ubuntu-latest", name: "Linux"}
- - {runs_on: "macos-15-intel" , name: "macOS"}
- steps:
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- {{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- - name: Install sccache
- if: matrix.platform.name == 'Linux'
- shell: bash
- run: |
- arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies (Linux)
- if: matrix.platform.name == 'Linux'
- run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev libuv1-dev
- - name: Install R package system dependencies (macOS)
- if: matrix.platform.name == 'macOS'
- run: brew install sccache openssl curl libuv
- - name: Remove arrow/
- run: |
- rm -rf arrow/
- - name: Enable parallel build
- run: |
- cores=`nproc || sysctl -n hw.logicalcpu`
- echo "MAKEFLAGS=-j$cores" >> $GITHUB_ENV
- - name: Install arrow source package
- env:
- # Test source build so be sure not to download a binary
- LIBARROW_BINARY: "FALSE"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- - name: Retry with verbosity if that failed
- if: failure()
- env:
- LIBARROW_BINARY: "FALSE"
- ARROW_R_DEV: "TRUE"
- CMAKE_FIND_DEBUG_MODE: "ON"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- upload-binaries:
- # Only upload binaries if all tests pass.
- needs: [r-packages, test-source, test-linux-binary]
- name: Upload artifacts
- runs-on: ubuntu-latest
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- - name: Download Artifacts
- uses: actions/download-artifact@v4
- with:
- path: artifacts
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
- - name: Move libarrow artifacts
- run: |
- mkdir -p binaries/
- mv artifacts/r-libarrow-*/* binaries/
- - name: Rename artifacts
- shell: Rscript {0}
- run: |
- file_paths <- list.files("artifacts", include.dirs = FALSE, recursive = TRUE)
- new_names <- file.path("binaries", sub("/", "__", file_paths))
- dir.create("binaries", showWarnings = FALSE)
- file.copy(file.path("artifacts", file_paths), new_names)
-
- {{ macros.github_upload_releases("binaries/r-*")|indent }}
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{% set is_fork = macros.is_fork %}
+
+{{ macros.github_header() }}
+
+jobs:
+ source:
+ # This job will change the version to either the custom_version param or YMD format.
+ # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
+ name: Source Package
+ runs-on: ubuntu-latest
+ outputs:
+ pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
+ steps:
+ {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
+ - name: Save Version
+ id: save-version
+ shell: bash
+ run: |
+ echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
+
+ - uses: r-lib/actions/setup-r@v2
+ with:
+ install-r: true
+
+ - name: Build R source package
+ shell: bash
+ run: |
+ cd arrow/r
+ # Copy in the Arrow C++ source
+ make sync-cpp
+ R CMD build --no-build-vignettes .
+
+ - name: Upload package artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: r-pkg__src__contrib
+ path: arrow/r/arrow_*.tar.gz
+
+ macos-cpp:
+ name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ needs: source
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: macos-15-intel, arch: "x86_64" }
+ - { runs_on: macos-14, arch: "arm64" }
+ env:
+ PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
+ PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
+ steps:
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
+ - name: Install Deps
+ run: |
+ brew install sccache ninja
+ brew install openssl@3.0
+ brew install libxml2
Review Comment:
First change: `brew install libxml2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3456510720
##########
dev/tasks/r/github.packages.yml:
##########
@@ -1,435 +1,437 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{% import 'macros.jinja' as macros with context %}
-
-{% set is_fork = macros.is_fork %}
-
-{{ macros.github_header() }}
-
-jobs:
- source:
- # This job will change the version to either the custom_version param or YMD format.
- # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
- name: Source Package
- runs-on: ubuntu-latest
- outputs:
- pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
- - name: Save Version
- id: save-version
- shell: bash
- run: |
- echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
-
- - uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
-
- - name: Build R source package
- shell: bash
- run: |
- cd arrow/r
- # Copy in the Arrow C++ source
- make sync-cpp
- R CMD build --no-build-vignettes .
-
- - name: Upload package artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg__src__contrib
- path: arrow/r/arrow_*.tar.gz
-
- macos-cpp:
- name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: macos-15-intel, arch: "x86_64" }
- - { runs_on: macos-14, arch: "arm64" }
- env:
- PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
- PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - name: Install Deps
- run: |
- brew install sccache ninja
- brew install openssl@3.0
- - name: Build libarrow
- shell: bash
- env:
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- MACOSX_DEPLOYMENT_TARGET: "11.6"
- ARROW_GCS: ON
- ARROW_DEPENDENCY_SOURCE: BUNDLED
- CMAKE_GENERATOR: Ninja
- LIBARROW_MINIMAL: false
- run: |
- sccache --start-server
- export EXTRA_CMAKE_FLAGS="-DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3.0)"
- cd arrow
- r/inst/build_arrow_static.sh
- - name: Bundle libarrow
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- linux-cpp:
- name: C++ Binary Linux {{ '${{ matrix.arch }}' }}
- runs-on: {{ '${{ matrix.runs-on }}' }}
- needs: source
- strategy:
- fail-fast: false
- matrix:
- include:
- - arch: x86_64
- runs-on: ubuntu-latest
- ubuntu: "22.04"
- - arch: arm64
- runs-on: ubuntu-24.04-arm
- ubuntu: "22.04"
- env:
- PKG_ID: r-libarrow-linux-{{ '${{ matrix.arch }}' }}
- PKG_FILE: r-libarrow-linux-{{ '${{ matrix.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- {{ macros.github_install_archery()|indent }}
- - name: Build libarrow
- shell: bash
- env:
- ARCH: {{ "${{ matrix.arch == 'x86_64' && 'amd64' || 'arm64/v8' }}" }}
- UBUNTU: {{ '${{ matrix.ubuntu }}' }}
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: |
- source arrow/ci/scripts/util_enable_core_dumps.sh
- archery docker run ubuntu-cpp-static
- - name: Bundle libarrow
- shell: bash
- run: |
- # These files were created by the docker user so we have to chown them
- sudo chown -R $USER:$USER arrow/r/libarrow
-
- cd arrow/r/libarrow/dist
- zip -r $PKG_FILE lib/ include/
- - name: Create Checksum
- shell: bash
- run: |
- cd arrow/r/libarrow/dist
- shasum -a 512 ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../../../../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- windows-cpp:
- name: C++ Binary Windows RTools (40 only)
- needs: source
- runs-on: windows-latest
- env:
- PKG_ID: r-libarrow-windows-x86_64
- PKG_FILE: r-libarrow-windows-x86_64-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
- steps:
- - run: git config --global core.autocrlf false
- {{ macros.github_checkout_arrow()|indent }}
- {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
- - uses: r-lib/actions/setup-r@v2
- with:
- rtools-version: 40
- r-version: "4.0"
- Ncpus: 2
- - name: Install sccache
- shell: bash
- run: arrow/ci/scripts/install_sccache.sh pc-windows-msvc $(pwd)/sccache
- - name: Build Arrow C++ with rtools40
- shell: bash
- env:
- ARROW_HOME: "arrow"
- MINGW_ARCH: "ucrt64"
- RTOOLS_VERSION: 40
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- run: arrow/ci/scripts/r_windows_build.sh
- - name: Create Checksum
- shell: bash
- run: |
- cd build
- sha512sum ${PKG_FILE} > ${PKG_FILE}.sha512
- mv ${PKG_FILE}{,.sha512} ../
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: {{ '${{ env.PKG_ID }}' }}
- path: |
- {{ '${{ env.PKG_FILE }}' }}
- {{ '${{ env.PKG_FILE }}' }}.sha512
-
- r-packages:
- needs: [source, windows-cpp, macos-cpp]
- name: {{ '${{ matrix.platform.name }} ${{ matrix.r_version }}' }}
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - { runs_on: 'windows-latest', name: "Windows"}
- - { runs_on: macos-15-intel, name: "macOS x86_64"}
- - { runs_on: macos-14, name: "macOS arm64" }
- r_version: [oldrel, release]
- steps:
- - uses: r-lib/actions/setup-r@v2
- with:
- r-version: {{ '${{ matrix.r_version }}' }}
- {{ macros.github_setup_local_r_repo(false, true, true)|indent }}
- - name: Prepare Dependency Installation
- shell: bash
- run: |
- tar -xzf repo/src/contrib/arrow_*.tar.gz arrow/DESCRIPTION
- - name: Install dependencies
- uses: r-lib/actions/setup-r-dependencies@v2
- with:
- working-directory: 'arrow'
- extra-packages: cpp11
- - name: Set CRAN like openssl
- if: contains(matrix.platform.name, 'macOS')
- # The -E forwards the GITHUB_* envvars
- shell: sudo -E Rscript {0}
- run: |
- # get the mac-recipes version of openssl from CRAN
- source("https://mac.R-project.org/bin/install.R")
- install.libs("openssl")
-
- # override our cmakes default setting of the brew --prefix as root dir to avoid version conflicts.
- if (Sys.info()[["machine"]] == "arm64"){
- cat("OPENSSL_ROOT_DIR=/opt/R/arm64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- } else {
- cat("OPENSSL_ROOT_DIR=/opt/R/x86_64\n", file=Sys.getenv("GITHUB_ENV"), append = TRUE)
- }
- - name: Build Binary
- id: build
- shell: Rscript {0}
- env:
- NOT_CRAN: "false" # actions/setup-r sets this implicitly
- ARROW_R_DEV: "true"
- LIBARROW_BINARY: "true" # has to be set as long as allowlist not updated
- LIBARROW_BUILD: "false"
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- run: |
- on_windows <- tolower(Sys.info()[["sysname"]]) == "windows"
-
- # Build
- Sys.setenv(MAKEFLAGS = paste0("-j", parallel::detectCores()))
- INSTALL_opts <- "--build"
- if (!on_windows) {
- # Windows doesn't support the --strip arg
- INSTALL_opts <- c(INSTALL_opts, "--strip")
- }
-
- # always remove arrow (mainly for self-hosted runners)
- try(remove.packages("arrow"), silent = TRUE)
-
- cat("Install arrow from dev repo.\n")
- install.packages(
- "arrow",
- type = "source",
- # The sub is necessary to prevent an error on windows.
- repos = sub("file://", "file:", getOption("arrow.dev_repo")),,
- INSTALL_opts = INSTALL_opts
- )
-
-
- # Test
- library(arrow)
- arrow_info()
- read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
-
- # encode contrib.url for artifact name
- cmd <- paste0(
- "path=",
- gsub(
- "/", "__",
- contrib.url("", type = "binary")
- ),
- "\n"
- )
- cat(cmd, file = Sys.getenv("GITHUB_OUTPUT"), append = TRUE)
-
- - name: Upload binary artifact
- uses: actions/upload-artifact@v4
- with:
- name: r-pkg{{ '${{ steps.build.outputs.path }}' }}
- path: arrow_*
- test-linux-binary:
- needs: [source, linux-cpp]
- name: Test binary {{ '${{ matrix.config.image }}' }} {{ '${{ matrix.config.runner }}' }}
- runs-on: {{ '${{ matrix.config.runner }}' }}
- container: {{ '${{ matrix.config.image }}' }}
- strategy:
- fail-fast: false
- matrix:
- config:
- # If libarrow_binary is unset, we're testing that we're automatically
- # choosing a binary on this OS. If libarrow_binary is TRUE, we're on
- # an OS that is not in the allowlist, so we have to opt-in to use the
- # binary. Other env vars used in r_docker_configure.sh can be added
- # here and wired up in the later steps.
- # x86_64 tests
- - {image: "rhub/ubuntu-clang", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
- # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE", runner: "ubuntu-latest"}
- - {image: "rhub/ubuntu-release", runner: "ubuntu-latest"} # currently ubuntu-24.04
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-latest"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-latest"}
- # ARM64 tests
- - {image: "posit/r-base:4.3-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.4-noble", runner: "ubuntu-24.04-arm"}
- - {image: "posit/r-base:4.5-noble", runner: "ubuntu-24.04-arm"}
- steps:
- # Get the arrow checkout just for the docker config scripts
- # Don't need submodules for this (hence false arg to macro): they fail on
- # actions/checkout for some reason in this context
- {{ macros.github_checkout_arrow(1, false, '3')|indent }}
-
- - name: Install system requirements
- env:
- ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
- shell: bash
- run: |
- # Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
- echo "${RPREFIX}/bin" >> $GITHUB_PATH
- export PATH="${RPREFIX}/bin:${PATH}"
- cd arrow && ARROW_SOURCE_HOME=$(pwd) ./ci/scripts/r_docker_configure.sh
- {{ macros.github_setup_local_r_repo(true, false)|indent }}
- - name: Install arrow from our repo
- env:
- ARROW_R_DEV: "TRUE"
- LIBARROW_BUILD: "FALSE"
- LIBARROW_BINARY: {{ '${{ matrix.config.libarrow_binary }}' }}
- ARROW_R_ENFORCE_CHECKSUM: "true"
- ARROW_R_CHECKSUM_PATH: "{{ '${{ github.workspace }}' }}/repo/libarrow"
- shell: bash
- run: |
- Rscript -e '
- {{ macros.github_test_r_src_pkg()|indent(8) }}
- '
- test-source:
- needs: source
- name: Test {{ '${{ matrix.platform.name }}' }} source build
- runs-on: {{ '${{ matrix.platform.runs_on }}' }}
- strategy:
- fail-fast: false
- matrix:
- platform:
- - {runs_on: "ubuntu-latest", name: "Linux"}
- - {runs_on: "macos-15-intel" , name: "macOS"}
- steps:
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- {{ macros.github_setup_local_r_repo(false, false)|indent }}
- {{ macros.github_checkout_arrow(action_v="3")|indent }}
- - name: Install sccache
- if: matrix.platform.name == 'Linux'
- shell: bash
- run: |
- arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
- - name: Install R package system dependencies (Linux)
- if: matrix.platform.name == 'Linux'
- run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev libuv1-dev
- - name: Install R package system dependencies (macOS)
- if: matrix.platform.name == 'macOS'
- run: brew install sccache openssl curl libuv
- - name: Remove arrow/
- run: |
- rm -rf arrow/
- - name: Enable parallel build
- run: |
- cores=`nproc || sysctl -n hw.logicalcpu`
- echo "MAKEFLAGS=-j$cores" >> $GITHUB_ENV
- - name: Install arrow source package
- env:
- # Test source build so be sure not to download a binary
- LIBARROW_BINARY: "FALSE"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- - name: Retry with verbosity if that failed
- if: failure()
- env:
- LIBARROW_BINARY: "FALSE"
- ARROW_R_DEV: "TRUE"
- CMAKE_FIND_DEBUG_MODE: "ON"
- {{ macros.github_set_sccache_envvars()|indent(8) }}
- shell: Rscript {0}
- run: |
- {{ macros.github_test_r_src_pkg()|indent(8) }}
-
- upload-binaries:
- # Only upload binaries if all tests pass.
- needs: [r-packages, test-source, test-linux-binary]
- name: Upload artifacts
- runs-on: ubuntu-latest
- steps:
- {{ macros.github_checkout_arrow()|indent }}
- - name: Download Artifacts
- uses: actions/download-artifact@v4
- with:
- path: artifacts
- - name: Install R
- uses: r-lib/actions/setup-r@v2
- with:
- install-r: true
- - name: Move libarrow artifacts
- run: |
- mkdir -p binaries/
- mv artifacts/r-libarrow-*/* binaries/
- - name: Rename artifacts
- shell: Rscript {0}
- run: |
- file_paths <- list.files("artifacts", include.dirs = FALSE, recursive = TRUE)
- new_names <- file.path("binaries", sub("/", "__", file_paths))
- dir.create("binaries", showWarnings = FALSE)
- file.copy(file.path("artifacts", file_paths), new_names)
-
- {{ macros.github_upload_releases("binaries/r-*")|indent }}
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{% set is_fork = macros.is_fork %}
+
+{{ macros.github_header() }}
+
+jobs:
+ source:
+ # This job will change the version to either the custom_version param or YMD format.
+ # The output allows other steps to use the exact version to prevent issues (e.g. date changes during run)
+ name: Source Package
+ runs-on: ubuntu-latest
+ outputs:
+ pkg_version: {{ '${{ steps.save-version.outputs.pkg_version }}' }}
+ steps:
+ {{ macros.github_checkout_arrow()|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, arrow.no_rc_r_version)|indent }}
+ - name: Save Version
+ id: save-version
+ shell: bash
+ run: |
+ echo "pkg_version=$(grep ^Version arrow/r/DESCRIPTION | sed s/Version:\ //)" >> $GITHUB_OUTPUT
+
+ - uses: r-lib/actions/setup-r@v2
+ with:
+ install-r: true
+
+ - name: Build R source package
+ shell: bash
+ run: |
+ cd arrow/r
+ # Copy in the Arrow C++ source
+ make sync-cpp
+ R CMD build --no-build-vignettes .
+
+ - name: Upload package artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: r-pkg__src__contrib
+ path: arrow/r/arrow_*.tar.gz
+
+ macos-cpp:
+ name: C++ Binary macOS {{ '${{ matrix.platform.arch }}' }}
+ runs-on: {{ '${{ matrix.platform.runs_on }}' }}
+ needs: source
+ strategy:
+ fail-fast: false
+ matrix:
+ platform:
+ - { runs_on: macos-15-intel, arch: "x86_64" }
+ - { runs_on: macos-14, arch: "arm64" }
+ env:
+ PKG_ID: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}
+ PKG_FILE: r-libarrow-darwin-{{ '${{ matrix.platform.arch }}' }}-{{ '${{ needs.source.outputs.pkg_version }}' }}.zip
+ steps:
+ {{ macros.github_checkout_arrow(action_v="3")|indent }}
+ {{ macros.github_change_r_pkg_version(is_fork, '${{ needs.source.outputs.pkg_version }}')|indent }}
+ - name: Install Deps
+ run: |
+ brew install sccache ninja
+ brew install openssl@3.0
+ brew install libxml2
Review Comment:
First change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4935168891
Revision: 84f3b741ba5c803a3a4ad50fdf713f1bf13ef59c
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3cda80c98b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3cda80c98b)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/29092014671/job/86359486783)|
|r-recheck-most|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011854/job/86359353844)|
|test-r-alpine-linux-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011976/job/86359354210)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions/runs/29092013163/job/86359358404)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions/runs/29092013516/job/86359359375)|
|test-r-dev-duckdb|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011777/job/86359353524)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012064/job/86359354637)|
|test-r-extra-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/29092013113/job/86359357857)|
|test-r-fedora-clang|[](https://github.com/ursacomputing/crossbow/actions/runs/29092010946/job/86359350931)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012513/job/86359356230)|
|test-r-gcc-12|[](https://github.com/ursacomputing/crossbow/actions/runs/29092013280/job/86359358349)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012708/job/86359356760)|
|test-r-install-local-minsizerel|[](https://github.com/ursacomputing/crossbow/actions/runs/29092013240/job/86359358703)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012431/job/86359356018)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012348/job/86359355536)|
|test-r-linux-sanitizers|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012103/job/86359354731)|
|test-r-linux-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012986/job/86359357373)|
|test-r-m1-san|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011741/job/86359353467)|
|test-r-macos-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012618/job/86359356440)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions/runs/29092012604/job/86359356422)|
|test-r-ubuntu-22.04|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011934/job/86359354025)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011789/job/86359353489)|
|test-r-wasm|[](https://github.com/ursacomputing/crossbow/actions/runs/29092011738/job/86359353177)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4087406721
:warning: GitHub issue #32123 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4133723503
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4120961024
To do list.
- [ ] Rebase against `main` to fix failing CI job.
- [ ] Review `configure`, `configure.win` and `tools/nixlibs.R` against changes in https://github.com/apache/arrow/pull/13404. It's possible this will reveal some other changes that are needed.
- [ ] Add CI job to test changes. See again https://github.com/apache/arrow/pull/13404.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4119319230
Thanks for the PR @marberts - if you rebase from the main branch now, the non-API call stuff should be resolved.
Your best bet for the `configure` etc stuff is looking at the history of that file or older PRs to see how this kind of thing was handled there.
This PR where the GCS bindings were added might help: https://github.com/apache/arrow/pull/13404, but let us know if you have any questions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4774967810
Could you try `git rebase` not `git merge`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3488373365
##########
r/configure.win:
##########
@@ -67,6 +67,7 @@ function configure_binaries() {
# pkg-config --libs libcurl
GCS_LIBS="-lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 \
-lz -lws2_32 -lnghttp2 -ldbghelp"
+ # AZURE_LIBS="-lcurl -lssl -lxml2"
Review Comment:
Yes, I left these commented out in case progress gets made in the future around the incompatibility with mingw and the azure SDK stuff. Happy to either remove them or make it earlier in the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4187254788
Hi @thisisnic (CC: @marberts ),
Here's a quick update of where things are at.
Right now the `AMD64 Windows C++ RTools 40 ucrt64` job is failing due to the various Azure sdk dependencies (`azure-core-cpp`, `azure-identity-cpp`, `azure-storage-blobs-cpp`, `azure-storage-common-cpp`, and `azure-storage-files-datalake-cpp`). Specifically, these packages do not seem to be available on MSYS2 repositories (i.e., we can't `pacman install` them), and we haven't been able to install these packages with `vcpkg` using either `x64-mingw-dynamic` or `x64-mingw-static` triplets.
Our understanding is that the Arrow C++ code in the Arrow R package must be built with the MinGW toolchain (due to Rtools), so the Azure SDKs we depend on must also be built with MinGW. To the best of our understanding, the only way to do this is to build the [`azure-sdk-for-cpp`](https://github.com/Azure/azure-sdk-for-cpp) components we need from source with MinGW.
We were able to build `azure-core-cpp`, `azure-storage-blobs-cpp`, `azure-storage-common-cpp`, and `azure-storage-files-datalake-cpp` successfully with MinGW. Unfortunately, we were not able to get `azure-identity-cpp` to build as it seems to depend heavily on the Windows Implementation Library (WIL) which lacks stable support for MinGW.
**What this means for this PR**
Our understanding is that this `azure-identity-cpp` limitation means that either (1) certain Azure Identity features won't work on the Windows build of Arrow R or (2) Azure should be disabled on the Windows build of Arrow R to avoid confusion.
Before proceeding on this, we wanted to check if this limitation is a showstopper for this PR, and if not, do you want us to proceed by modifying the Windows build to ignore Azure identity, or should we just set `-DARROW_AZURE=OFF` on the Windows build?
Alternatively, if you think there's a way to get `azure-identity-cpp` build working with MinGW, we would be happy to accept any advice on this.
Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4131458966
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4167205512
It seems I made a mess of the rebase, although I'm not entirely sure how :disappointed:.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4224064287
> > should we just set -DARROW_AZURE=OFF on the Windows build?
> Yes, I think so, but would like to hear from @jonkeane on this as well.
Sorry this took so long to get back to. Yeah, I'm fine if we do this. Is there _any_ path forward to building this on windows? (i.e. would it be possible for us to build an arrow binary we host and ask folks to use that if they want support for this? Do we have upstream issues with the problematic dependencies?). I'm slightly uncomfortable shipping something that has _no possibility_ of support on windows (especially Azure, given the Microsoft connection I suspect there's slightly more windows based Arrow users on Azure than other clouds)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4725122109
> Looking at the PR, for some reason the GitHub UI is showing `dev/tasks/r/github.packages.yml` as having been completely rewritten, which is going to make it hard to review. Perhaps tabs and spaces git switched or something? Would you mind giving this a rebase and resolving that so it's a bit easier to see what changes have been made there?
:man_facepalming: Yes, there are only a handful of actual changes to that file. I'll fix it so the diff makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3418897981
##########
dev/tasks/tasks.yml:
##########
Review Comment:
What's the reason for deleting this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4977359821
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit b03b4c539a31dd2a5d93389ec6656f311ac60489.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/87279427722) has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4826090802
Revision: 004e7254a1acdb145d8a2cacf0bf03b8dec9c57f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f9bf4c7b3a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f9bf4c7b3a)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177805/job/83905965556)|
|r-recheck-most|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177421/job/83905902292)|
|test-r-alpine-linux-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177271/job/83905901827)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176800/job/83905900611)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177519/job/83905902385)|
|test-r-dev-duckdb|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176755/job/83905900446)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176968/job/83905901131)|
|test-r-extra-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177030/job/83905901294)|
|test-r-fedora-clang|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176901/job/83905900786)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177324/job/83905902048)|
|test-r-gcc-12|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177532/job/83905902515)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177152/job/83905901592)|
|test-r-install-local-minsizerel|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177002/job/83905901101)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177368/job/83905902186)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177163/job/83905901657)|
|test-r-linux-sanitizers|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176715/job/83905900370)|
|test-r-linux-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177070/job/83905901359)|
|test-r-m1-san|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177348/job/83905902084)|
|test-r-macos-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/28322177201/job/83905901690)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176793/job/83905900618)|
|test-r-ubuntu-22.04|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176868/job/83905900768)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176848/job/83905900730)|
|test-r-wasm|[](https://github.com/ursacomputing/crossbow/actions/runs/28322176601/job/83905900056)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4939142836
I pushed a change to make sure that the binary runners have `libxml`, etc. The other failures are unrelated to this branch I believe (though some aren't on https://crossbow.arrow-dev.org/ yet, so they might even be r-devel transitory actually!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4149157314
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r2975018091
##########
r/configure:
##########
Review Comment:
TODO: `-lcurl -lssl` are already passed on this line, drop from line 364.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4133741336
Revision: 4da7280fe7f015b17662e68cc87952a78f0f5ed2
Submitted crossbow builds: [ursacomputing/crossbow @ actions-92325d6543](https://github.com/ursacomputing/crossbow/branches/all?query=actions-92325d6543)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383615/job/68697321737)|
|r-recheck-most|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383860/job/68697203655)|
|test-r-alpine-linux-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382773/job/68697200237)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382372/job/68697199056)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383201/job/68697201311)|
|test-r-dev-duckdb|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382797/job/68697200168)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions/runs/23591384066/job/68697204163)|
|test-r-extra-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382625/job/68697199503)|
|test-r-fedora-clang|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382374/job/68697199075)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382590/job/68697199449)|
|test-r-gcc-12|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383699/job/68697203058)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383463/job/68697202403)|
|test-r-install-local-minsizerel|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383286/job/68697201661)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382785/job/68697200319)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383472/job/68697202338)|
|test-r-linux-sanitizers|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382424/job/68697199193)|
|test-r-linux-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383206/job/68697201243)|
|test-r-m1-san|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383217/job/68697201473)|
|test-r-macos-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383695/job/68697203097)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions/runs/23591383660/job/68697202909)|
|test-r-ubuntu-22.04|[](https://github.com/ursacomputing/crossbow/actions/runs/23591382391/job/68697199125)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions/runs/23591384146/job/68697204428)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4099762490
The CI is failing because the Azure C++ SDK depends on libxml2.
- [ ] Update `tools/nixlibs.R` to check for `libxml2` in `with_cloud_support()`, turning off the Azure feature flag is libxml2 isn't found.
- [ ] Update the vignettes to note that libxml2 is a dependency of Azure (same as with curl and openssl).
- [ ] Update DESCRIPTION to include libxml2 in SystemRequirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4119332925
We'll also need additional CI jobs to test this - the PR I linked to above contains examples of what we added there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49553:
URL: https://github.com/apache/arrow/pull/49553#discussion_r3425084868
##########
dev/tasks/tasks.yml:
##########
Review Comment:
This is out of place. I think it's an artifact from when I failed to correctly rebase. AFAICT this change was already merged in a few months ago: https://github.com/apache/arrow/commit/1a71493214d49db70f744e7cc1798579ec392f62.
The rest of the changes all correctly belong to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4412898086
Could we request the `AMD64 Windows C++ RTools 40 ucrt64` CI job to be re-run?
Reading through the [job logs](https://github.com/apache/arrow/actions/runs/25030129655/job/73309781840), the issue seems to be that a third party package failed to download (line 1419).
I wasn't able to reproduce this failure locally, so I'd like to see if this error was intermittent or if it occurs again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4224977799
Thank you both for the feedback!
Based on our results from building `azure-sdk-for-cpp` from source, my understanding is that a *limited* version of Azure support for Arrow ought to be possible on Windows, but the Azure Identity features won't work on the Windows build due to the abovementioned difficulties of getting `azure-identity-cpp` to build with MinGW. I.e., users should still be able to use connection strings or SAS tokens in this scenario.
If limited Azure support on Windows is preferable to disabling it, we will continue investigating what would be involved to disable the Azure Identity features and otherwise keep Azure support on the Windows build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-32123: [R] Expose azure blob filesystem [arrow]
Posted by "Collinbrown95 (via GitHub)" <gi...@apache.org>.
Collinbrown95 commented on PR #49553:
URL: https://github.com/apache/arrow/pull/49553#issuecomment-4229935684
Hi @jonkeane and @thisisnic,
I just had a discussion with @marberts, and we wanted to follow up on a couple of things.
1. @jonkeane I might have misunderstood your comment in my initial response. Is your suggestion to build the R Arrow library with `-DARROW_AZURE=ON` with MSVC and host it outside of CRAN (possibly circumventing the MinGW restriction discussed earlier in this thread)?
If this is your suggestion, could you kindly share a bit more information about how this would work?
For some extra context, my current understanding is that there are low-level C++ ABI compatibility issues between binaries built with MSVC and binaries built with MinGW, which (I think) implies that an MSVC compiled binary would not interoperate with the rest of the R ecosystem (which I believe is built with the MinGW toolchain).
That said, I am by no means an expert in R/Windows internals. I would be happy to learn more if you know of a way for R packages to interoperate with MSVC-built binaries without ABI issues.
2. I realize I never responded to your question about a path forward for fixing upstream issues.
The core issue seems to be an incompatibility between Windows Implementation Libraries (WIL) and MinGW (this is [the issue that @thisisnic cited earlier in this thread](https://github.com/microsoft/wil/issues/117)). Our understanding is that resolving this issue would resolve many of the downstream issues in building [`azure-sdk-for-cpp`](https://github.com/Azure/azure-sdk-for-cpp) with MinGW, which would open the door for a fully functioning build of Arrow on Windows with Azure support enabled.
The most recent discussion on this topic I could find is [this WIL issue from 2024](https://github.com/microsoft/wil/pull/467). I can create an issue in either the WIL/MinGW repositories to flag that this compatibility issue is still relevant, but I get the impression that the scope/timeline of this work goes beyond what this PR is trying to achieve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org