You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ks...@apache.org on 2018/11/18 15:00:43 UTC
[arrow] branch master updated: ARROW-3366: [R] Dockerfile for
docker-compose setup
This is an automated email from the ASF dual-hosted git repository.
kszucs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new d23f4f2 ARROW-3366: [R] Dockerfile for docker-compose setup
d23f4f2 is described below
commit d23f4f25bffd1dfbedfed8c4c5bf18d77aed4c7f
Author: James Lamb <ja...@gmail.com>
AuthorDate: Sun Nov 18 16:00:27 2018 +0100
ARROW-3366: [R] Dockerfile for docker-compose setup
Thank you to @romainfrancois and others who have pushed forward the R side of this project!
This PR is my attempt to address [ARROW-3336](https://issues.apache.org/jira/projects/ARROW/issues/ARROW-3366), providing a testing container for the R package.
This follows up on work done by @kszucs in #2572 in an R-specific way.
**NOTE:** This PR is a WIP. `R CMD INSTALL` currently fails because it cannot find wherever I installed `arrow` to. But I felt that this is far enough along to put up for review.
Author: James Lamb <ja...@gmail.com>
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #2770 from jameslamb/r_dockerfile and squashes the following commits:
9fcebe53 <James Lamb> removed failing test in the R package
dd26d4b9 <James Lamb> fixed failing test in the R package
343456af <James Lamb> more changes
0db3369d <James Lamb> added bit64 dependency to R package
f7c050e2 <Krisztián Szűcs> R_CHECK_FORCE_SUGGESTS
db9ee5ba <Krisztián Szűcs> pass CXXFLAGS
8a8376d6 <James Lamb> Put R build steps into a separate CI script
64ea840f <James Lamb> Updated LD_LIBRARY_PATH in R container
4d7f02c7 <James Lamb> fixed a few issues in R container
94242b32 <James Lamb> Add explict cpp build in usage for R docker-compose
4a7112b2 <James Lamb> Fixed comment text in R Dockerfile
d8e55d7e <James Lamb> removed todo about R containers in docker-compose
210a7745 <James Lamb> ARROW-3366: Dockerfile for docker-compose setup
---
r/src/Makevars.in => ci/docker_build_r.sh | 26 +++++++++--
docker-compose.yml | 13 +++++-
r/.Rbuildignore | 3 +-
r/Dockerfile | 77 +++++++++++++++++++++++++++++++
r/src/Makevars.in | 3 +-
r/tests/testthat/test-RecordBatch.R | 1 -
6 files changed, 114 insertions(+), 9 deletions(-)
diff --git a/r/src/Makevars.in b/ci/docker_build_r.sh
old mode 100644
new mode 100755
similarity index 59%
copy from r/src/Makevars.in
copy to ci/docker_build_r.sh
index 8277c6c..9fb95bc
--- a/r/src/Makevars.in
+++ b/ci/docker_build_r.sh
@@ -1,3 +1,4 @@
+#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,7 +16,24 @@
# specific language governing permissions and limitations
# under the License.
-PKG_CPPFLAGS=@cflags@
-PKG_CXXFLAGS=$(C_VISIBILITY)
-CXX_STD=CXX11
-PKG_LIBS=@libs@ -Wl,-rpath,/usr/local/lib
+set -e
+
+export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
+export ARROW_HOME=$CONDA_PREFIX
+
+# For newer GCC per https://arrow.apache.org/docs/python/development.html#known-issues
+export CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
+export PKG_CXXFLAGS=$CXXFLAGS
+
+# Build arrow
+pushd /arrow/r
+
+rm src/RcppExports*
+Rscript -e "Rcpp::compileAttributes()"
+R CMD build --keep-empty-dirs .
+R CMD INSTALL $(ls | grep arrow_*.tar.gz)
+
+export _R_CHECK_FORCE_SUGGESTS_=false
+R CMD check $(ls | grep arrow_*.tar.gz) --as-cran --no-manual
+
+popd
diff --git a/docker-compose.yml b/docker-compose.yml
index c1f798d..a62eb33 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -28,8 +28,6 @@ services:
######################### Language Containers ###############################
- #TODO(kszucs): R
-
c_glib:
# Usage:
# docker-compose build cpp
@@ -101,6 +99,17 @@ services:
dockerfile: rust/Dockerfile
volumes: *volumes
+ r:
+ # Usage:
+ # docker-compose build cpp
+ # docker-compose build r
+ # docker-compose run r
+ image: arrow:r
+ build:
+ context: .
+ dockerfile: r/Dockerfile
+ volumes: *volumes
+
######################### Tools and Linters #################################
# TODO(kszucs): site
diff --git a/r/.Rbuildignore b/r/.Rbuildignore
index b0c42b5..be540f7 100644
--- a/r/.Rbuildignore
+++ b/r/.Rbuildignore
@@ -5,4 +5,5 @@ src/.clang-format
LICENSE.md
^data-raw$
lint.sh
-
+Dockerfile
+.*\.tar\.gz
diff --git a/r/Dockerfile b/r/Dockerfile
new file mode 100644
index 0000000..a357bcc
--- /dev/null
+++ b/r/Dockerfile
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM arrow:cpp
+
+# Configure
+ENV CC=gcc \
+ CXX=g++
+
+# r-base includes tzdata. Get around interactive stop in that package
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Build R
+# [1] https://www.digitalocean.com/community/tutorials/how-to-install-r-on-ubuntu-18-04
+# [2] https://linuxize.com/post/how-to-install-r-on-ubuntu-18-04/#installing-r-packages-from-cran
+RUN apt update && \
+ apt install -y \
+ apt-transport-https \
+ software-properties-common && \
+ apt-key adv \
+ --keyserver keyserver.ubuntu.com \
+ --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
+ add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/' && \
+ apt install -y r-base && \
+ # system libs needed by core R packages
+ apt install -y \
+ libgit2-dev \
+ libssl-dev && \
+ # install clang to mirror what was done on Travis
+ apt install -y \
+ clang \
+ clang-format \
+ clang-tidy && \
+ # R CMD CHECK --as-cran needs pdflatex to build the package manual
+ apt install -y \
+ texlive-latex-base && \
+ # Install vctrs from Github
+ Rscript -e "install.packages('devtools', repos = 'http://cran.rstudio.com')" && \
+ Rscript -e "devtools::install_github('romainfrancois/vctrs@bit64')" && \
+ Rscript -e "devtools::install_github('r-lib/withr')" && \
+ Rscript -e "devtools::install_github('RcppCore/Rcpp')" && \
+ # R is not good at picking up dependencies and installing them automatically
+ Rscript -e "install.packages(c( \
+ 'Rcpp', \
+ 'purrr', \
+ 'assertthat', \
+ 'fs', \
+ 'tibble', \
+ 'crayon', \
+ 'testthat', \
+ 'bit64', \
+ 'hms', \
+ 'lubridate'), \
+ repos = 'http://cran.rstudio.com')"
+
+# Tell R where it can find the source code for arrow
+ENV PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/build/cpp/src/arrow
+ENV LD_LIBRARY_PATH=/opt/conda/lib/:/build/cpp/src/arrow:/arrow/r/src
+ENV CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
+
+# build, install, test R package
+CMD /arrow/ci/docker_build_cpp.sh && \
+ /arrow/ci/docker_build_r.sh
diff --git a/r/src/Makevars.in b/r/src/Makevars.in
index 8277c6c..5e28551 100644
--- a/r/src/Makevars.in
+++ b/r/src/Makevars.in
@@ -16,6 +16,7 @@
# under the License.
PKG_CPPFLAGS=@cflags@
-PKG_CXXFLAGS=$(C_VISIBILITY)
+PKG_CXXFLAGS+=$(C_VISIBILITY)
CXX_STD=CXX11
PKG_LIBS=@libs@ -Wl,-rpath,/usr/local/lib
+#CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
diff --git a/r/tests/testthat/test-RecordBatch.R b/r/tests/testthat/test-RecordBatch.R
index e48be18..29a9ee8 100644
--- a/r/tests/testthat/test-RecordBatch.R
+++ b/r/tests/testthat/test-RecordBatch.R
@@ -137,7 +137,6 @@ test_that("read_record_batch handles various streams (ARROW-3450, ARROW-3505)",
batch4 <- read_record_batch(mmap_file)
batch5 <- read_record_batch(bytes)
batch6 <- read_record_batch(buf_reader)
- expect_error(read_record_batch(buf_reader))
stream_reader <- record_batch_stream_reader(bytes)
batch7 <- read_record_batch(stream_reader)