You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by np...@apache.org on 2022/07/06 14:16:37 UTC
[arrow] branch master updated: ARROW-16752: [R] Rework Linux binary installation (#13464)
This is an automated email from the ASF dual-hosted git repository.
npr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new c492ef497a ARROW-16752: [R] Rework Linux binary installation (#13464)
c492ef497a is described below
commit c492ef497a62e600c9436f2a92dace1190c7a465
Author: Neal Richardson <ne...@gmail.com>
AuthorDate: Wed Jul 6 10:16:31 2022 -0400
ARROW-16752: [R] Rework Linux binary installation (#13464)
See the jira for the main behavior changes here. Other changes of note:
* There are more brief messages printed to the installation log, even in the default "quiet" mode, that indicate which branch of the logic in nixlibs.R you've gone through. They're factual and generally connected to the tests that are being run, but they are worded somewhat ambiguously or coded, so as not to run afoul of censors should they appear in the wrong context. This should help us in the triaging of installation failures, even in circumstances where we can't enable greater verbosity.
* There is a start of a test suite for nixlibs.R, run separately from the package tests. It has been wired up to run in `ci/scripts/r_test.sh`.
Authored-by: Neal Richardson <ne...@gmail.com>
Signed-off-by: Neal Richardson <ne...@gmail.com>
---
ci/scripts/r_docker_configure.sh | 21 +--
ci/scripts/r_test.sh | 3 +
dev/release/rat_exclude_files.txt | 1 +
dev/tasks/macros.jinja | 18 +--
dev/tasks/r/github.packages.yml | 70 ++++++----
r/tools/nixlibs-allowlist.txt | 4 +
r/tools/nixlibs.R | 268 +++++++++++++++++++++++++++-----------
r/tools/test-nixlibs.R | 112 ++++++++++++++++
r/vignettes/install.Rmd | 112 ++++++++--------
9 files changed, 434 insertions(+), 175 deletions(-)
diff --git a/ci/scripts/r_docker_configure.sh b/ci/scripts/r_docker_configure.sh
index 9f93ba2b61..2bc5a4806f 100755
--- a/ci/scripts/r_docker_configure.sh
+++ b/ci/scripts/r_docker_configure.sh
@@ -19,12 +19,14 @@
set -ex
: ${R_BIN:=R}
+# This is where our docker setup puts things; set this to run outside of docker
+: ${ARROW_SOURCE_HOME:=/arrow}
# The Dockerfile should have put this file here
-if [ -f "/arrow/ci/etc/rprofile" ]; then
+if [ -f "${ARROW_SOURCE_HOME}/ci/etc/rprofile" ]; then
# Ensure parallel R package installation, set CRAN repo mirror,
# and use pre-built binaries where possible
- cat /arrow/ci/etc/rprofile >> $(${R_BIN} RHOME)/etc/Rprofile.site
+ cat ${ARROW_SOURCE_HOME}/ci/etc/rprofile >> $(${R_BIN} RHOME)/etc/Rprofile.site
fi
# Ensure parallel compilation of C/C++ code
@@ -74,6 +76,9 @@ if [ "$RHUB_PLATFORM" = "linux-x86_64-fedora-clang" ]; then
sed -i.bak -E -e 's/(CXX1?1? =.*)/\1 -stdlib=libc++/g' $(${R_BIN} RHOME)/etc/Makeconf
rm -rf $(${R_BIN} RHOME)/etc/Makeconf.bak
+ sed -i.bak -E -e 's/(\-std=gnu\+\+)/-std=c++/g' $(${R_BIN} RHOME)/etc/Makeconf
+ rm -rf $(${R_BIN} RHOME)/etc/Makeconf.bak
+
sed -i.bak -E -e 's/(CXXFLAGS = )(.*)/\1 -g -O3 -Wall -pedantic -frtti -fPIC/' $(${R_BIN} RHOME)/etc/Makeconf
rm -rf $(${R_BIN} RHOME)/etc/Makeconf.bak
@@ -88,8 +93,8 @@ if [[ "$DEVTOOLSET_VERSION" -gt 0 ]]; then
$PACKAGE_MANAGER install -y "devtoolset-$DEVTOOLSET_VERSION"
fi
-if [ "$ARROW_S3" == "ON" ] || [ "$ARROW_R_DEV" == "TRUE" ]; then
- # Install curl and openssl for S3 support
+if [ "$ARROW_S3" == "ON" ] || [ "$ARROW_GCS" == "ON" ] || [ "$ARROW_R_DEV" == "TRUE" ]; then
+ # Install curl and openssl for S3/GCS support
if [ "$PACKAGE_MANAGER" = "apt-get" ]; then
apt-get install -y libcurl4-openssl-dev libssl-dev
else
@@ -97,12 +102,12 @@ if [ "$ARROW_S3" == "ON" ] || [ "$ARROW_R_DEV" == "TRUE" ]; then
fi
# The Dockerfile should have put this file here
- if [ -f "/arrow/ci/scripts/install_minio.sh" ] && [ "`which wget`" ]; then
- /arrow/ci/scripts/install_minio.sh latest /usr/local
+ if [ -f "${ARROW_SOURCE_HOME}/ci/scripts/install_minio.sh" ] && [ "`which wget`" ]; then
+ ${ARROW_SOURCE_HOME}/ci/scripts/install_minio.sh latest /usr/local
fi
- if [ -f "/arrow/ci/scripts/install_gcs_testbench.sh" ] && [ "`which pip`" ]; then
- /arrow/ci/scripts/install_gcs_testbench.sh default
+ if [ -f "${ARROW_SOURCE_HOME}/ci/scripts/install_gcs_testbench.sh" ] && [ "`which pip`" ]; then
+ ${ARROW_SOURCE_HOME}/ci/scripts/install_gcs_testbench.sh default
fi
fi
diff --git a/ci/scripts/r_test.sh b/ci/scripts/r_test.sh
index 8429187d88..0328df2384 100755
--- a/ci/scripts/r_test.sh
+++ b/ci/scripts/r_test.sh
@@ -26,6 +26,9 @@ pushd ${source_dir}
printenv
+# Run the nixlibs.R test suite, which is not included in the installed package
+ ${R_BIN} -e 'setwd("tools"); testthat::test_dir(".")'
+
# Before release, we always copy the relevant parts of the cpp source into the
# package. In some CI checks, we will use this version of the source:
# this is done by setting ARROW_SOURCE_HOME to something other than "/arrow"
diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index c7ae282158..d9119f11e0 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -201,5 +201,6 @@ r/tests/testthat/test-*.txt
r/tests/testthat/_snaps/*.md
r/inst/include/cpp11.hpp
r/inst/include/cpp11/*.hpp
+r/tools/nixlibs-allowlist.txt
.gitattributes
ruby/red-arrow/.yardopts
diff --git a/dev/tasks/macros.jinja b/dev/tasks/macros.jinja
index e4628ad5d3..c7f7c647b7 100644
--- a/dev/tasks/macros.jinja
+++ b/dev/tasks/macros.jinja
@@ -237,10 +237,10 @@ on:
head DESCRIPTION
rm -f DESCRIPTION.bak
cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb tools/apache-arrow.rb
-
+
# Pin the git commit in the formula to match
cd tools
- if [ "{{ is_fork }}" == "true" ]; then
+ if [ "{{ is_fork }}" == "true" ]; then
sed -i.bak -E -e 's/apache\/arrow.git"$/{{ arrow.github_repo.split("/") | join("\/") }}.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow.rb
else
sed -i.bak -E -e 's/arrow.git"$/arrow.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow.rb
@@ -250,13 +250,13 @@ on:
{%- macro github_test_r_src_pkg() -%}
source("https://raw.githubusercontent.com/apache/arrow/master/ci/etc/rprofile")
-
+
install.packages(
"arrow",
- repos = c(getOption("arrow.dev_repo"), "https://cloud.r-project.org"),
+ repos = c(getOption("arrow.dev_repo"), getOption("repos")),
verbose = TRUE
)
-
+
library(arrow)
read_parquet(system.file("v0.7.1.parquet", package = "arrow"))
@@ -267,7 +267,7 @@ on:
{%- macro github_setup_local_r_repo(get_nix, get_win) -%}
- name: Setup local repo
shell: bash
- run: mkdir repo
+ run: mkdir repo
{% if get_win %}
- name: Get windows binary
uses: actions/download-artifact@v3
@@ -294,8 +294,8 @@ on:
run: |
# getwd() is necessary as this macro is used within jobs using a docker container
tools::write_PACKAGES(file.path(getwd(), "repo/src/contrib", fsep = "/"), type = "source", verbose = TRUE)
- - name: Show repo
- shell: bash
+ - name: Show repo
+ shell: bash
# tree not available in git-bash on windows
run: |
ls -R repo
@@ -306,6 +306,6 @@ on:
print(str)
profile_path <- file.path(getwd(), ".Rprofile")
write(str, file = profile_path, append = TRUE)
- # Set envvar for later steps by appending to $GITHUB_ENV
+ # Set envvar for later steps by appending to $GITHUB_ENV
write(paste0("R_PROFILE_USER=", profile_path), file = Sys.getenv("GITHUB_ENV"), append = TRUE)
{% endmacro %}
diff --git a/dev/tasks/r/github.packages.yml b/dev/tasks/r/github.packages.yml
index 76beb6400c..6e46b3ff43 100644
--- a/dev/tasks/r/github.packages.yml
+++ b/dev/tasks/r/github.packages.yml
@@ -212,47 +212,63 @@ jobs:
test-linux-binary:
needs: [source, linux-cpp]
- name: Test binary {{ '${{ matrix.image }}' }}
+ name: Test binary {{ '${{ matrix.config.image }}' }}
runs-on: ubuntu-latest
- container: {{ '${{ matrix.image }}' }}
+ container: {{ '${{ matrix.config.image }}' }}
strategy:
fail-fast: false
matrix:
- image:
- - "rhub/ubuntu-gcc-release" # ubuntu-20.04 (focal)
- - "rstudio/r-base:4.1-bionic"
- - "rstudio/r-base:4.2-centos7"
- - "rocker/r-ver:3.6.3" # for debian:buster (10)
- - "rocker/r-ver" # ubuntu-20.04
- - "rhub/fedora-clang-devel" # tests distro-map.csv, mapped to ubuntu-18.04
- - "rocker/r-ubuntu:22.04" # tests openssl3 compatibility
+ config:
+ # If libarrow_binary is unset, we're testing that we're automatically
+ # choosing a binary on this OS. If libarrow_binary is TRUE, we're on
+ # an OS that is not in the allowlist, so we have to opt-in to use the
+ # binary. Other env vars used in r_docker_configure.sh can be added
+ # here (like devtoolset) and wired up in the later steps.
+ - {image: "rhub/debian-clang-devel", libarrow_binary: "TRUE"}
+ # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
+ # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE"}
+ - {image: "rhub/ubuntu-gcc-release"} # currently ubuntu-20.04 (focal)
+ - {image: "rocker/r-ubuntu:22.04"} # openssl3
+ - {image: "rocker/r-ver"} # whatever is latest ubuntu LTS
+ - {image: "rocker/r-ver:4.0.0"} # ubuntu-20.04
+ - {image: "rocker/r-ver:3.6.3", libarrow_binary: "TRUE"} # debian:buster (10)
+ - {image: "rstudio/r-base:4.1-bionic"} # ubuntu-18.04
+ - {image: "rstudio/r-base:4.2-centos7"}
+ # Even with devtoolset-8, binaries fail because glibc is older than
+ # what we build with on ubuntu-18.04 (uncomment to see the error)
+ # TODO(ARROW-16976): build binaries with older glibc
+ # - {image: "rstudio/r-base:4.2-centos7", devtoolset: "8"}
steps:
+ # Get the arrow checkout just for the docker config scripts
+ # Don't need submodules for this (hence false arg to macro): they fail on
+ # actions/checkout for some reason in this context
+ {{ macros.github_checkout_arrow(1, false)|indent }}
- name: Install system requirements
+ env:
+ ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
+ DEVTOOLSET_VERSION: {{ '${{ matrix.config.devtoolset }}' }}
shell: bash
run: |
- if [ "`which dnf`" ]; then
- dnf install -y libcurl-devel openssl-devel
- elif [ "`which yum`" ]; then
- yum install -y libcurl-devel openssl-devel
- elif [ "`which zypper`" ]; then
- zypper install -y libcurl-devel libopenssl-devel
- else
- apt-get update
- apt-get install -y libcurl4-openssl-dev libssl-dev
- fi
-
- # Add R-devel to PATH
- echo "/opt/R-devel/bin" >> $GITHUB_PATH
-
+ # Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
+ echo "${RPREFIX}/bin" >> $GITHUB_PATH
+ export PATH="${RPREFIX}/bin:${PATH}"
+ cd arrow && ARROW_SOURCE_HOME=$(pwd) ./ci/scripts/r_docker_configure.sh
{{ macros.github_setup_local_r_repo(true, false)|indent }}
- name: Install arrow from our repo
env:
- ARROW_R_DEV: TRUE
+ ARROW_R_DEV: "TRUE"
LIBARROW_BUILD: "FALSE"
- LIBARROW_BINARY: "TRUE"
- shell: Rscript {0}
+ LIBARROW_BINARY: {{ '${{ matrix.config.libarrow_binary }}' }}
+ DEVTOOLSET_VERSION: {{ '${{ matrix.config.devtoolset }}' }}
+ shell: bash
run: |
+ if [[ "$DEVTOOLSET_VERSION" -gt 0 ]]; then
+ # enable the devtoolset version to use it
+ source /opt/rh/devtoolset-$DEVTOOLSET_VERSION/enable
+ fi
+ Rscript -e '
{{ macros.github_test_r_src_pkg()|indent(8) }}
+ '
test-source:
needs: source
diff --git a/r/tools/nixlibs-allowlist.txt b/r/tools/nixlibs-allowlist.txt
new file mode 100644
index 0000000000..bd9f0c1b2c
--- /dev/null
+++ b/r/tools/nixlibs-allowlist.txt
@@ -0,0 +1,4 @@
+ubuntu
+centos
+redhat
+rhel
diff --git a/r/tools/nixlibs.R b/r/tools/nixlibs.R
index 0dadaa0ef7..5b9cbddfb8 100644
--- a/r/tools/nixlibs.R
+++ b/r/tools/nixlibs.R
@@ -19,6 +19,9 @@ args <- commandArgs(TRUE)
VERSION <- args[1]
dst_dir <- paste0("libarrow/arrow-", VERSION)
+# TESTING is set in test-nixlibs.R; it won't be set when called from configure
+test_mode <- exists("TESTING")
+
arrow_repo <- paste0(getOption("arrow.dev_repo", "https://arrow-r-nightly.s3.amazonaws.com"), "/libarrow/")
options(.arrow.cleanup = character()) # To collect dirs to rm on exit
@@ -26,12 +29,12 @@ on.exit(unlink(getOption(".arrow.cleanup")))
env_is <- function(var, value) identical(tolower(Sys.getenv(var)), value)
-try_download <- function(from_url, to_file) {
+try_download <- function(from_url, to_file, hush = quietly) {
status <- try(
suppressWarnings(
- download.file(from_url, to_file, quiet = quietly)
+ download.file(from_url, to_file, quiet = hush)
),
- silent = quietly
+ silent = hush
)
# Return whether the download was successful
!inherits(status, "try-error") && status == 0
@@ -40,17 +43,15 @@ try_download <- function(from_url, to_file) {
# For local debugging, set ARROW_R_DEV=TRUE to make this script print more
quietly <- !env_is("ARROW_R_DEV", "true")
-# Default is build from source, not download a binary
+# The default will build from source as a fallback if a binary is not found or shouldn't be used
+# Set LIBARROW_BUILD=FALSE to ensure that we use a previously built libarrow
+# and don't fall back to a full source build
build_ok <- !env_is("LIBARROW_BUILD", "false")
-# For LIBARROW_BINARY we support "true" or the name of the OS to use to
-# locate the appropriate binary (e.g., 'ubuntu-18.04'). When NOT_CRAN=true, and
-# LIBARROW_BINARY is unset, configure script sets LIBARROW_BINARY=true.
-binary_ok <- !env_is("LIBARROW_BINARY", "false") || env_is("LIBARROW_BINARY", "")
# Check if we're doing an offline build.
# (Note that cmake will still be downloaded if necessary
# https://arrow.apache.org/docs/developers/cpp/building.html#offline-builds)
-download_ok <- !env_is("TEST_OFFLINE_BUILD", "true") && try_download("https://github.com", tempfile())
+download_ok <- !test_mode && !env_is("TEST_OFFLINE_BUILD", "true") && try_download("https://raw.githubusercontent.com/apache/arrow/master/r/DESCRIPTION", tempfile(), hush = TRUE)
# This "tools/thirdparty_dependencies" path, within the tar file, might exist if
# create_package_with_all_dependencies() was run, or if someone has created it
@@ -60,53 +61,174 @@ download_ok <- !env_is("TEST_OFFLINE_BUILD", "true") && try_download("https://gi
thirdparty_dependency_dir <- Sys.getenv("ARROW_THIRDPARTY_DEPENDENCY_DIR", "tools/thirdparty_dependencies")
-download_binary <- function(os = identify_os()) {
+download_binary <- function(lib) {
libfile <- tempfile()
- if (!is.null(os)) {
- # See if we can map this os-version to one we have binaries for
- os <- find_available_binary(os)
- binary_url <- paste0(arrow_repo, "bin/", os, "/arrow-", VERSION, ".zip")
- if (try_download(binary_url, libfile)) {
- cat(sprintf("*** Successfully retrieved C++ binaries for %s\n", os))
- if (!identical(os, "centos-7")) {
- # centos-7 uses gcc 4.8 so the binary doesn't have ARROW_S3=ON
- # or ARROW_GCS=ON but the others do
- # TODO: actually check for system requirements?
- cat("**** Binary package requires libcurl and openssl\n")
- cat("**** If installation fails, retry after installing those system requirements\n")
- }
- } else {
- cat(sprintf("*** No libarrow binary found for version %s on %s\n", VERSION, os))
- libfile <- NULL
+ binary_url <- paste0(arrow_repo, "bin/", lib, "/arrow-", VERSION, ".zip")
+ if (try_download(binary_url, libfile)) {
+ if (!quietly) {
+ cat(sprintf("*** Successfully retrieved C++ binaries (%s)\n", lib))
}
} else {
+ if (!quietly) {
+ cat(sprintf("*** No libarrow binary found for version %s (%s)\n", VERSION, lib))
+ }
libfile <- NULL
}
libfile
}
# Function to figure out which flavor of binary we should download, if at all.
-# By default (unset or "FALSE"), it will not download a precompiled library,
-# but you can override this by setting the env var LIBARROW_BINARY to:
-# * `TRUE` (not case-sensitive), to try to discover your current OS, or
-# * some other string, presumably a related "distro-version" that has binaries
-# built that work for your OS
-identify_os <- function(os = Sys.getenv("LIBARROW_BINARY")) {
- if (tolower(os) %in% c("", "false")) {
- # Env var says not to download a binary
- return(NULL)
- } else if (!identical(tolower(os), "true")) {
- # Env var provided an os-version to use--maybe you're on Ubuntu 18.10 but
- # we only build for 18.04 and that's fine--so use what the user set
- return(os)
+# LIBARROW_BINARY controls the behavior. If unset, it will determine a course
+# of action based on the current system. Other values you can set it to:
+# * "FALSE" (not case-sensitive), to skip this option altogether
+# * "TRUE" (not case-sensitive), to try to discover your current OS, or
+# * Some other string: a "distro-version" that corresponds to a binary that is
+# available, to override what this function may discover by default.
+# Possible values are:
+# * "centos-7" (gcc 4.8, no AWS/GCS support)
+# * "ubuntu-18.04" (gcc 8, openssl 1)
+# * "ubuntu-22.04" (openssl 3)
+# These string values, along with `NULL`, are the potential return values of
+# this function.
+identify_binary <- function(lib = Sys.getenv("LIBARROW_BINARY"), info = distro()) {
+ lib <- tolower(lib)
+ if (identical(lib, "")) {
+ # Not specified. Check the allowlist.
+ lib <- ifelse(check_allowlist(info$id), "true", "false")
+ }
+
+ if (identical(lib, "false")) {
+ # Do not download a binary
+ NULL
+ } else if (!identical(lib, "true")) {
+ # Env var provided an os-version to use, to override our logic.
+ # We don't validate that this exists. If it doesn't, the download will fail
+ # and the build will fall back to building from source
+ lib
+ } else {
+ # See if we can find a suitable binary
+ select_binary()
}
+}
+
+check_allowlist <- function(os, allowed = "https://raw.githubusercontent.com/apache/arrow/master/r/tools/nixlibs-allowlist.txt") {
+ allowlist <- tryCatch(
+ # Try a remote allowlist so that we can add/remove without a release
+ suppressWarnings(readLines(allowed)),
+ # Fallback to default: allowed only on Ubuntu and CentOS/RHEL
+ error = function(e) c("ubuntu", "centos", "redhat", "rhel")
+ )
+ # allowlist should contain valid regular expressions (plain strings ok too)
+ any(grepl(paste(allowlist, collapse = "|"), os))
+}
+
+select_binary <- function(os = tolower(Sys.info()[["sysname"]]),
+ arch = tolower(Sys.info()[["machine"]]),
+ compiler_version = compiler_version_string(),
+ test_program = test_for_curl_and_openssl) {
+ if (identical(os, "linux") && identical(arch, "x86_64")) {
+ # We only host x86 linux binaries today
+ is_gcc4 <- any(grepl("^g\\+\\+.*[^\\d.]4(\\.\\d){2}", compiler_version))
+ if (is_gcc4) {
+ cat("*** Some features are not available with gcc 4\n")
+ return("centos-7")
+ } else {
+ tryCatch(
+ # Somehow the test program system2 call errors on the sanitizer builds
+ # so globally handle the possibility that this could fail
+ {
+ errs <- compile_test_program(test_program)
+ determine_binary_from_stderr(errs)
+ },
+ error = function(e) {
+ cat("*** Unable to find libcurl and openssl\n")
+ NULL
+ }
+ )
+ }
+ } else {
+ # No binary available for arch
+ cat(sprintf("*** Building on %s %s\n", os, arch))
+ NULL
+ }
+}
+
+# This tests that curl and openssl are present (bc we can include their headers)
+# and it checks for other versions/features and raises errors that we grep for
+test_for_curl_and_openssl <- "
+#include <ciso646>
+#ifdef _LIBCPP_VERSION
+#error Using libc++
+#endif
+
+#if !( __GLIBC__ >= 2 && __GLIBC_MINOR__ >= 27)
+#error glibc version too old
+#endif
+
+#include <curl/curl.h>
+#include <openssl/opensslv.h>
+#if OPENSSL_VERSION_NUMBER < 0x10002000L
+#error OpenSSL version too old
+#endif
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+#error Using OpenSSL version 3
+#endif
+"
+
+compile_test_program <- function(code) {
+ # Note: if we wanted to check for openssl on macOS, we'd have to set the brew
+ # path as a -I directory. But since we (currently) only run this code to
+ # determine whether we can download a Linux binary, it's not relevant.
+ runner <- "`R CMD config CXX11` `R CMD config CPPFLAGS` `R CMD config CXX11FLAGS` `R CMD config CXX11STD` -E -xc++"
+ suppressWarnings(system2("echo", sprintf('"%s" | %s -', code, runner), stdout = FALSE, stderr = TRUE))
+}
- linux <- distro()
- if (is.null(linux)) {
- cat("*** Unable to identify current OS/version\n")
+# TODO(ARROW-16976): build "ubuntu-18.04" on centos7 with newer devtoolset (but glibc is 2.17) for broader compatibility (like manylinux2014)?
+determine_binary_from_stderr <- function(errs) {
+ if (is.null(attr(errs, "status"))) {
+ # There was no error in compiling: so we found libcurl and openssl > 1.0.2,
+ # openssl is < 3.0, glibc is >= 2.27, and we're not using a strict libc++
+ cat("*** Found libcurl and openssl >= 1.0.2\n")
+ return("ubuntu-18.04")
+ } else if (any(grepl("Using libc++", errs, fixed = TRUE))) {
+ # Our binaries are all built with GNU stdlib so they fail with libc++
+ cat("*** Found libc++\n")
return(NULL)
+ } else if (any(grepl("glibc version too old", errs))) {
+ # ubuntu-18.04 has glibc 2.27, so even if you install newer compilers
+ # (e.g. devtoolset on centos) and have curl/openssl, you run into problems
+ # TODO(ARROW-16976): build binaries with older glibc
+ cat("*** Checking glibc version\n")
+ # If we're here, we're on an older OS but with a newer compiler than gcc 4.8
+ # (we already checked), so it is possible to build with more features on.
+ # We just can't use our binaries because they were built with newer glibc.
+ return(NULL)
+ } else if (header_not_found("curl/curl", errs)) {
+ # TODO(ARROW-16985): should these next 3 NULL cases return centos-7? A source build
+ # won't be able to include more features.
+ # Could check if build_ok (also for glibc?)
+ cat("*** libcurl not found\n")
+ return(NULL)
+ } else if (header_not_found("openssl/opensslv", errs)) {
+ cat("*** openssl not found\n")
+ return(NULL)
+ } else if (any(grepl("OpenSSL version too old", errs))) {
+ cat("*** openssl found but version >= 1.0.2 is required for some features\n")
+ return(NULL)
+ } else if (any(grepl("Using OpenSSL version 3", errs))) {
+ cat("*** Found libcurl and openssl >= 3.0.0\n")
+ return("ubuntu-22.04")
}
- paste(linux$id, linux$short_version, sep = "-")
+ NULL
+}
+
+header_not_found <- function(header, errs) {
+ regex <- sprintf("[Ee]rror.*%s\\.h", header)
+ any(grepl(regex, errs))
+}
+
+compiler_version_string <- function(compiler = R_CMD_config("CXX11")) {
+ system(paste(compiler, "--version"), intern = TRUE)
}
#### start distro ####
@@ -212,20 +334,6 @@ read_system_release <- function() {
#### end distro ####
-find_available_binary <- function(os) {
- # Download a csv that maps one to the other, columns "actual" and "use_this"
- u <- "https://raw.githubusercontent.com/ursa-labs/arrow-r-nightly/master/linux/distro-map.csv"
- lookup <- try(utils::read.csv(u, stringsAsFactors = FALSE), silent = quietly)
- if (!inherits(lookup, "try-error") && os %in% lookup$actual) {
- new <- lookup$use_this[lookup$actual == os]
- if (length(new) == 1 && !is.na(new)) { # Just some sanity checking
- cat(sprintf("*** Using %s binary for %s\n", new, os))
- os <- new
- }
- }
- os
-}
-
find_local_source <- function() {
# We'll take the first of these that exists
# The first case probably occurs if we're in the arrow git repo
@@ -262,6 +370,10 @@ env_vars_as_string <- function(env_var_list) {
env_var_string
}
+R_CMD_config <- function(var) {
+ tools::Rcmd(paste("config", var), stdout = TRUE)
+}
+
build_libarrow <- function(src_dir, dst_dir) {
# We'll need to compile R bindings with these libs, so delete any .o files
system("rm src/*.o", ignore.stdout = TRUE, ignore.stderr = TRUE)
@@ -291,9 +403,6 @@ build_libarrow <- function(src_dir, dst_dir) {
}
options(.arrow.cleanup = c(getOption(".arrow.cleanup"), build_dir))
- R_CMD_config <- function(var) {
- tools::Rcmd(paste("config", var), stdout = TRUE)
- }
env_var_list <- c(
SOURCE_DIR = src_dir,
BUILD_DIR = build_dir,
@@ -356,11 +465,7 @@ build_libarrow <- function(src_dir, dst_dir) {
}
ensure_cmake <- function() {
- cmake <- find_cmake(c(
- Sys.getenv("CMAKE"),
- Sys.which("cmake"),
- Sys.which("cmake3")
- ))
+ cmake <- find_cmake()
if (is.null(cmake)) {
# If not found, download it
@@ -368,7 +473,7 @@ ensure_cmake <- function() {
CMAKE_VERSION <- Sys.getenv("CMAKE_VERSION", "3.21.4")
if (tolower(Sys.info()[["sysname"]]) %in% "darwin") {
postfix <- "-macos-universal.tar.gz"
- } else if (tolower(Sys.info()[["machine"]]) == "arm64") {
+ } else if (tolower(Sys.info()[["machine"]]) %in% c("arm64", "aarch64")) {
postfix <- "-linux-aarch64.tar.gz"
} else if (tolower(Sys.info()[["machine"]]) == "x86_64") {
postfix <- "-linux-x86_64.tar.gz"
@@ -401,11 +506,20 @@ ensure_cmake <- function() {
"/cmake-", CMAKE_VERSION, sub(".tar.gz", "", postfix, fixed = TRUE),
"/bin/cmake"
)
+ } else {
+ # Show which one we found
+ # Full source builds will always show "cmake" in the logs
+ cat(sprintf("**** cmake: %s\n", cmake))
}
cmake
}
-find_cmake <- function(paths, version_required = "3.10") {
+find_cmake <- function(paths = c(
+ Sys.getenv("CMAKE"),
+ Sys.which("cmake"),
+ Sys.which("cmake3")
+ ),
+ version_required = "3.10") {
# Given a list of possible cmake paths, return the first one that exists and is new enough
# version_required should be a string or packageVersion; numeric version
# can be misleading (e.g. 3.10 is actually 3.1)
@@ -559,13 +673,18 @@ with_cloud_support <- function(env_var_list) {
}
# Check the features
+ # This duplicates what we do with the test program above when we check
+ # capabilities for using binaries. We could consider consolidating this
+ # logic, though these use cmake in order to match exactly what we do in the
+ # libarrow build, and maybe that increases the fidelity.
if (isTRUE(cmake_gcc_version(env_var_list) < "4.9")) {
print_warning("not available for gcc < 4.9")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
} else if (!cmake_find_package("CURL", NULL, env_var_list)) {
# curl on macos should be installed, so no need to alter this for macos
- print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb")
+ # TODO: check for apt/yum/etc. and message the right thing?
+ print_warning("requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb)")
arrow_s3 <- FALSE
arrow_gcs <- FALSE
} else if (!cmake_find_package("OpenSSL", "1.0.2", env_var_list)) {
@@ -619,13 +738,17 @@ cmake_find_package <- function(pkg, version = NULL, env_var_list) {
#####
-if (!file.exists(paste0(dst_dir, "/include/arrow/api.h"))) {
+if (!test_mode && !file.exists(paste0(dst_dir, "/include/arrow/api.h"))) {
# If we're working in a local checkout and have already built the libs, we
# don't need to do anything. Otherwise,
# (1) Look for a prebuilt binary for this version
bin_file <- src_dir <- NULL
- if (download_ok && binary_ok) {
- bin_file <- download_binary()
+ if (download_ok) {
+ binary_flavor <- identify_binary()
+ if (!is.null(binary_flavor)) {
+ # The env vars say we can, and we've determined a lib that should work
+ bin_file <- download_binary(binary_flavor)
+ }
}
if (!is.null(bin_file)) {
# Extract them
@@ -638,8 +761,7 @@ if (!file.exists(paste0(dst_dir, "/include/arrow/api.h"))) {
if (!is.null(src_dir)) {
cat(paste0(
"*** Building libarrow from source\n",
- " For a faster, more complete installation, set the environment variable NOT_CRAN=true before installing\n",
- " See install vignette for details:\n",
+ " For build options and troubleshooting, see the install vignette:\n",
" https://cran.r-project.org/web/packages/arrow/vignettes/install.html\n"
))
build_libarrow(src_dir, dst_dir)
diff --git a/r/tools/test-nixlibs.R b/r/tools/test-nixlibs.R
new file mode 100644
index 0000000000..d5e83b1305
--- /dev/null
+++ b/r/tools/test-nixlibs.R
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# Usage: run testthat::test_dir(".") inside of this directory
+
+# Flag so that we just load the functions and don't evaluate them like we do
+# when called from configure.R
+TESTING <- TRUE
+
+source("nixlibs.R", local = TRUE)
+
+test_that("identify_binary() based on LIBARROW_BINARY", {
+ expect_null(identify_binary("FALSE"))
+ expect_identical(identify_binary("ubuntu-18.04"), "ubuntu-18.04")
+ expect_null(identify_binary("", info = list(id = "debian")))
+})
+
+test_that("select_binary() based on system", {
+ expect_output(
+ expect_null(select_binary("darwin", "x86_64")), # Not built today
+ "Building on darwin x86_64"
+ )
+ expect_output(
+ expect_null(select_binary("linux", arch = "aarch64")), # Not built today
+ "Building on linux aarch64"
+ )
+ gcc48 <- c(
+ "g++-4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4",
+ "Copyright (C) 2013 Free Software Foundation, Inc.",
+ "This is free software; see the source for copying conditions. There is NO",
+ "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+ )
+ expect_output(
+ expect_identical(
+ select_binary("linux", "x86_64", compiler_version = gcc48),
+ "centos-7"
+ ),
+ "Some features are not available with gcc 4"
+ )
+})
+
+test_that("compile_test_program()", {
+ expect_null(attr(compile_test_program("int a;"), "status"))
+ fail <- compile_test_program("#include <wrong/NOTAHEADER.h>")
+ expect_true(attr(fail, "status") > 0)
+ expect_true(header_not_found("wrong/NOTAHEADER", fail))
+})
+
+test_that("determine_binary_from_stderr", {
+ expect_output(
+ expect_identical(
+ determine_binary_from_stderr(compile_test_program("int a;")),
+ "ubuntu-18.04"
+ ),
+ "Found libcurl and openssl >= 1.0.2"
+ )
+ expect_output(
+ expect_identical(
+ determine_binary_from_stderr(compile_test_program("#error Using OpenSSL version 3")),
+ "ubuntu-22.04"
+ ),
+ "Found libcurl and openssl >= 3.0.0"
+ )
+ expect_output(
+ expect_null(
+ determine_binary_from_stderr(compile_test_program("#error OpenSSL version too old"))
+ ),
+ "openssl found but version >= 1.0.2 is required for some features"
+ )
+})
+
+test_that("select_binary() with test program", {
+ expect_output(
+ expect_identical(
+ select_binary("linux", "x86_64", "clang", "int a;"),
+ "ubuntu-18.04"
+ ),
+ "Found libcurl and openssl >= 1.0.2"
+ )
+ expect_output(
+ expect_identical(
+ select_binary("linux", "x86_64", "clang", "#error Using OpenSSL version 3"),
+ "ubuntu-22.04"
+ ),
+ "Found libcurl and openssl >= 3.0.0"
+ )
+})
+
+test_that("check_allowlist", {
+ tf <- tempfile()
+ cat("tu$\n^cent\n", file = tf)
+ expect_true(check_allowlist("ubuntu", tf))
+ expect_true(check_allowlist("centos", tf))
+ expect_false(check_allowlist("redhat", tf)) # remote allowlist doesn't have this
+ expect_true(check_allowlist("redhat", tempfile())) # remote allowlist doesn't exist, so we fall back to the default list, which contains redhat
+ expect_false(check_allowlist("debian", tempfile()))
+})
diff --git a/r/vignettes/install.Rmd b/r/vignettes/install.Rmd
index 257dd7b11d..156f06cce7 100644
--- a/r/vignettes/install.Rmd
+++ b/r/vignettes/install.Rmd
@@ -57,7 +57,7 @@ install.packages("arrow", repos = "https://packagemanager.rstudio.com/all/__linu
```
Note that the User Agent header must be specified as in the example above.
-Please check [the RStudio Package Manager: Admin Guide ](https://docs.rstudio.com/rspm/admin/serving-binaries/#using-linux-binary-packages) for more details.
+Please check [the RStudio Package Manager: Admin Guide](https://docs.rstudio.com/rspm/admin/serving-binaries/#using-linux-binary-packages) for more details.
For other Linux distributions, to get the relevant URL, you can visit
[the RSPM site](https://packagemanager.rstudio.com/client/#/repos/1/overview),
@@ -76,47 +76,16 @@ conda install -c conda-forge --strict-channel-priority r-arrow
knitr::include_graphics("./r_source_libarrow_binary.png")
```
-Another way of achieving faster installation with all key features enabled is to use our self-hosted libarrow binaries. You can do this by setting the `NOT_CRAN` environment variable before you call `install.packages()`:
+Another way of achieving faster installation with all key features enabled is to use static libarrow binaries we host. These are used automatically on many Linux distributions (x86_64 architecture only), according to the [allowlist](https://github.com/apache/arrow/blob/master/r/tools/nixlibs-allowlist.txt). If your distribution isn't in the list, you can opt-in by setting the `NOT_CRAN` environment variable before you call `install.packages()`:
```{r, eval = FALSE}
Sys.setenv("NOT_CRAN" = TRUE)
install.packages("arrow")
```
-This installs the source version of the R package, but during the installation process will check for compatible libarrow binaries that we host and use those if available. If no binary is available or can't be found, then this option falls back onto method 2 below, but results in a more fully-featured build than default.
+This installs the source version of the R package, but during the installation process will check for compatible libarrow binaries that we host and use those if available. If no binary is available or can't be found, then this option falls back onto method 2 below (full source build), but setting the environment variable results in a more fully-featured build than default.
-# Installing libarrow dependencies
-
-When you install libarrow, its dependencies will be automatically downloaded.
-The environment variable `ARROW_DEPENDENCY_SOURCE` controls whether the libarrow
-installation also downloads or installs all dependencies (when set to `BUNDLED`),
-uses only system-installed dependencies (when set to `SYSTEM`) or checks
-system-installed dependencies first and only installs dependencies which aren't
-already present (when set to `AUTO`).
-
-These dependencies vary by platform; however, if you wish to install these
-yourself prior to libarrow installation, we recommend that you take a look at
-the [docker file for whichever of our CI builds](https://github.com/apache/arrow/tree/master/ci/docker)
-(the ones ending in "cpp" are for building Arrow's C++ libaries aka libarrow)
-corresponds most closely to your setup. This will contain the most up-to-date
-information about dependencies and minimum versions.
-
-## Dependencies for S3 and GCS support
-
-The arrow package allows you to work with data in AWS S3 or in other cloud
-storage system that emulate S3, as well as Google Cloud Storage.
-However, support for working with S3 and GCS is not
-enabled in the default build, and it has additional system requirements. To
-enable it, set the environment variable `LIBARROW_MINIMAL=false` or
-`NOT_CRAN=true` to choose the full-featured build, or more selectively set
-`ARROW_S3=ON` and/or `ARROW_GCS=ON`.
-You also need the following system dependencies:
-
-* `gcc` >= 4.9 or `clang` >= 3.3; note that the default compiler on CentOS 7 is gcc 4.8.5, which is not sufficient
-* CURL: install `libcurl-devel` (rpm) or `libcurl4-openssl-dev` (deb)
-* OpenSSL >= 1.0.2: install `openssl-devel` (rpm) or `libssl-dev` (deb)
-
-The prebuilt libarrow binaries come with S3 and GCS support enabled, so you will need to meet these system requirements in order to use them--the package will not install without them (and will error with a message that explains this).If you're building everything from source, the install script will check for the presence of these dependencies and turn off S3 and GCS support in the build if the prerequisites are not met--installation will succeed but without S3 or GCS functionality. If [...]
+Except for the those built for gcc 4.8 (default on CentOS 7), the binaries include support for AWS S3 and Google Cloud Storage (GCS). These features require libcurl and openssl libraries installed separately; see below on how to install them. If you don't have these installed, the libarrow binary won't be used, and you will fall back to the full source build.
# Installing a release version (the less easy way)
@@ -126,12 +95,13 @@ The prebuilt libarrow binaries come with S3 and GCS support enabled, so you will
knitr::include_graphics("./r_source_libarrow_source.png")
```
-Generally compiling and installing R packages with C++ dependencies, requires
+Generally, compiling and installing R packages with C++ dependencies requires
either installing system packages, which you may not have privileges to do, or
building the C++ dependencies separately, which introduces all sorts of
-additional ways for things to go wrong, which is why we recommend method 1 above.
+additional ways for things to go wrong.
-However, if you wish to fine-tune or customise your Linux installation, the
+The full source build of arrow, compiling both C++ and R bindings, does handle most of the dependency management for you, but it is much slower.
+However, if using binaries isn't an option for you, or you wish to fine-tune or customize your Linux installation, the
instructions in this section explain how to do that.
### Basic configuration for building from source with fully featured installation
@@ -156,8 +126,8 @@ and alternative memory allocators.
Sys.setenv("LIBARROW_MINIMAL" = FALSE)
```
-By default this variable is unset; if set to `TRUE` a trimmed-down version of
-arrow is installed with many features disabled.
+By default this variable is unset, which builds many commonly used features such as Parquet support but disables some features that are more costly to build, like S3 and GCS support. If set to `TRUE`, a trimmed-down version of
+arrow is installed with all optional features disabled.
Note that in this guide, you will have seen us mention the environment variable
`NOT_CRAN` - this is a convenience variable, which when set to `TRUE`,
@@ -182,6 +152,41 @@ install.packages("arrow")
The section below discusses environment variables you can set before calling
`install.packages("arrow")` to build from source and customise your configuration.
+### Handling libarrow dependencies
+
+When you build libarrow from source, its dependencies will be automatically downloaded.
+The environment variable `ARROW_DEPENDENCY_SOURCE` controls whether the libarrow
+installation also downloads or installs all dependencies (when set to `BUNDLED`),
+uses only system-installed dependencies (when set to `SYSTEM`) or checks
+system-installed dependencies first and only installs dependencies which aren't
+already present (when set to `AUTO`, the default).
+
+These dependencies vary by platform; however, if you wish to install these
+yourself prior to libarrow installation, we recommend that you take a look at
+the [docker file for whichever of our CI builds](https://github.com/apache/arrow/tree/master/ci/docker)
+(the ones ending in "cpp" are for building Arrow's C++ libaries, aka libarrow)
+corresponds most closely to your setup. This will contain the most up-to-date
+information about dependencies and minimum versions.
+
+If downloading dependencies at build time is not an option, as when building on a system that is disconnected or behind a firewall, there are a few options. See "Offline builds" below.
+
+#### Dependencies for S3 and GCS support
+
+The arrow package allows you to work with data in AWS S3 or in other cloud
+storage system that emulate S3, as well as Google Cloud Storage.
+However, support for working with S3 and GCS is not
+enabled in the default source build, and it has additional system requirements. To
+enable it, set the environment variable `LIBARROW_MINIMAL=false` or
+`NOT_CRAN=true` to choose the full-featured build, or more selectively set
+`ARROW_S3=ON` and/or `ARROW_GCS=ON`.
+You also need the following system dependencies:
+
+* `gcc` >= 4.9 or `clang` >= 3.3; note that the default compiler on CentOS 7 is gcc 4.8.5, which is not sufficient
+* CURL: install `libcurl-devel` (rpm) or `libcurl4-openssl-dev` (deb)
+* OpenSSL >= 1.0.2: install `openssl-devel` (rpm) or `libssl-dev` (deb)
+
+The prebuilt libarrow binaries come with S3 and GCS support enabled, so you will need to meet these system requirements in order to use them. If you're building everything from source, the install script will check for the presence of these dependencies and turn off S3 and GCS support in the build if the prerequisites are not met--installation will succeed but without S3 or GCS functionality. If afterwards you install the missing system requirements, you'll need to reinstall the package [...]
+
### Advanced configuration for building from source
In this section, we describe how to fine-tune your installation at a more granular level.
@@ -219,7 +224,7 @@ the bundled build script. All boolean variables are case-insensitive.
| Name | Description | Default |
| --- | --- | :-: |
| `LIBARROW_BUILD` | Allow building from source | `true` |
-| `LIBARROW_BINARY` | Try to install `libarrow` binary instead of building from source | `true` |
+| `LIBARROW_BINARY` | Try to install `libarrow` binary instead of building from source | (unset) |
| `LIBARROW_MINIMAL` | Build with minimal features enabled | (unset) |
| `NOT_CRAN` | Set `LIBARROW_BINARY=true` and `LIBARROW_MINIMAL=false` | `false` |
| `ARROW_R_DEV` | More verbose messaging and regenerates some code | `false` |
@@ -229,8 +234,13 @@ the bundled build script. All boolean variables are case-insensitive.
See below for more in-depth explanations of these environment variables.
-* `LIBARROW_BINARY` : If set to `true`, the script will try to download a binary
- C++ library built for your operating system. You may also set it to some other string, a related "distro-version" that has binaries built that work for your OS. See the [distro map](https://raw.githubusercontent.com/ursa-labs/arrow-r-nightly/master/linux/distro-map.csv) for compatible binaries and OSs. If no binary is found, installation will fall back to building C++ dependencies from source.
+* `LIBARROW_BINARY` : By default on many distributions, or if explicitly set to
+ `true`, the script will determine whether there is a prebuilt libarrow that
+ will work with your system. You can set it to `false` to skip this option
+ altogether, or you can specify a string "distro-version" that corresponds to
+ a binary that is available, to override what this function may discover by
+ default. Possible values are: "centos-7" (gcc 4.8, no AWS/GCS support);
+ "ubuntu-18.04" (gcc 8, openssl 1); "ubuntu-22.04" (openssl 3).
* `LIBARROW_BUILD` : If set to `false`, the build script
will not attempt to build the C++ from source. This means you will only get
a working arrow R package if a prebuilt binary is found.
@@ -452,24 +462,10 @@ instead of downloading the prebuilt binary.
That should guarantee that the compiler settings match.
If a prebuilt libarrow binary wasn't found for your operating system but you think it should have been,
-check the logs for a message that says `*** Unable to identify current OS/version`,
-or a message that says `*** No C++ binaries found for` an invalid OS.
-If you see either, please [report an issue](https://issues.apache.org/jira/projects/ARROW/issues).
+please [report an issue](https://issues.apache.org/jira/projects/ARROW/issues) and share the console output.
You may also set the environment variable `ARROW_R_DEV=TRUE` for additional
debug messages.
-A workaround would be to set the environment variable `LIBARROW_BINARY`
-to a `distribution-version` that exists in the Ursa Labs repository.
-Setting `LIBARROW_BINARY` is also an option when there's not an exact match
-for your OS but a similar version would work,
-such as if you're on `ubuntu-18.10` and there's only a binary for `ubuntu-18.04`.
-
-If that workaround works for you, and you believe that it should work for everyone else too,
-you may propose [adding an entry to this lookup table](https://github.com/ursa-labs/arrow-r-nightly/edit/master/linux/distro-map.csv).
-This table is checked during the installation process
-and tells the script to use binaries built on a different operating system/version
-because they're known to work.
-
## Building libarrow from source
If building libarrow from source fails, check the error message.