You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by fs...@apache.org on 2019/06/18 12:50:37 UTC
[arrow] branch master updated: ARROW-3759: [R][CI] Build and test
(no libarrow) on Windows in Appveyor
This is an automated email from the ASF dual-hosted git repository.
fsaintjacques pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 83c211c ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
83c211c is described below
commit 83c211c30cbd39a88603011b42c534519ec174e0
Author: Neal Richardson <ne...@gmail.com>
AuthorDate: Tue Jun 18 08:50:27 2019 -0400
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
This is very much a work-in-progress for running the R package tests on Appveyor. It's an attempt to bring in the "standard" R Appveyor build tooling (https://github.com/krlmlr/r-appveyor, Apache 2.0 licensed) into Arrow's configuration.
I currently have the C++ library building and installing, and R and all R package dependencies installing, and R checks beginning, but they fail. This is because https://github.com/apache/arrow/blob/master/r/tools/winlibs.R is configured only to build with a released [sic] version of the arrow C++ binaries pre-built and hosted on rwinlib. Two immediate to-dos are:
1. get the package to install and check without libarrow at all (as https://github.com/apache/arrow/pull/4471 now allows)
2. get the package to find the libarrow we built locally from source
After that, I can reverse-engineer what appveyor does and document it for any Windows developers (https://issues.apache.org/jira/browse/ARROW-3758).
There's obviously some mess I've made of appveyor.yml that I'll clean up before I'm done, and I'll do my best to document and refactor some of the non-obvious features of the setup that I've discovered (e.g. that `ci/appveyor-cpp-build-mingw.bat` also installs GLib and Ruby and runs their test suites too) while I'm at it so that the next person might be able to pick things up faster.
cc @romainfrancois @javierluraschi
Author: Neal Richardson <ne...@gmail.com>
Closes #4538 from nealrichardson/r-appveyor and squashes the following commits:
6d6a74e31 <Neal Richardson> PR feedback
84eef1b2d <Neal Richardson> Merge upstream/master
328bc376f <Neal Richardson> Merge remote-tracking branch 'upstream/master' into r-appveyor
53c3a2084 <Neal Richardson> Makevars.win is dynamically generated now, no need to tweak it in the release script
567f04142 <Neal Richardson> Cleanups
f4cf5c3dd <Neal Richardson> Some appveyor.yml cleanup
19b60df07 <Neal Richardson> Enable R package to install on windows even if rwinlib isn't found
fb199110a <Neal Richardson> Try this name hack
15c0342d1 <Neal Richardson> Try building first, see where that goes
18803c353 <Neal Richardson> cd this way
b6f6c8de7 <Neal Richardson> No popd?
1ba4839e2 <Neal Richardson> let the debugging begin
d5e8f4ed4 <Neal Richardson> Bootstrap within the package dir
f7327ed23 <Neal Richardson> Find shell script up a level
8c3e898f8 <Neal Richardson> nm we already fast finish
39236f3f7 <Neal Richardson> Put mine first and fast-finish
94c9a42df <Neal Richardson> Debug with fewer jobs in the matrix
af1852e79 <Neal Richardson> Throw this at appveyor and see how far off we are
---
appveyor.yml | 56 +++++++++++++++++++++++++++++++++++++++++-
ci/appveyor-filter-changes.bat | 7 ++++++
dev/release/00-prepare-test.rb | 14 -----------
dev/release/00-prepare.sh | 8 ------
r/configure.win | 47 +++++++++++++++++++++++++++++++++++
r/src/Makevars.win | 38 ----------------------------
r/tools/winlibs.R | 6 ++---
7 files changed, 112 insertions(+), 64 deletions(-)
diff --git a/appveyor.yml b/appveyor.yml
index cefa28e..a4995a8 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -19,7 +19,7 @@
os: Visual Studio 2015
only_commits:
- # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
+ # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
files:
- appveyor.yml
- c_glib/
@@ -29,6 +29,7 @@ only_commits:
- format/
- go/
- python/
+ - r/
- ruby/
- rust/
@@ -102,6 +103,8 @@ environment:
GO111MODULE: on
GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
+ - JOB: "R without libarrow"
+ USE_CLCACHE: false
MSVC_DEFAULT_OPTIONS: ON
APPVEYOR_SAVE_CACHE_ON_ERROR: true
@@ -119,6 +122,57 @@ before_build:
build_script:
- call ci\appveyor-build.bat
+for:
+-
+ matrix:
+ only:
+ - JOB: "R without libarrow"
+ init:
+ ps: |
+ $ErrorActionPreference = "Stop"
+ Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
+ Import-Module '..\appveyor-tool.ps1'
+
+ before_build:
+ - cd r
+ - ps: Bootstrap
+ - cd %APPVEYOR_BUILD_FOLDER%
+
+ build_script:
+ # TODO: for the version of the R tests that builds, probably can't use this
+ # ci script because we need to build with the Rtools toolchain
+ # - call ci\appveyor-build.bat
+ - pushd r
+ - travis-tool.sh install_deps
+
+ test_script:
+ - travis-tool.sh run_tests
+
+ on_failure:
+ - travis-tool.sh dump_logs
+
+ environment:
+ NOT_CRAN: true
+
+ artifacts:
+ - path: 'r\*.Rcheck\**\*.log'
+ name: Logs
+
+ - path: 'r\*.Rcheck\**\*.out'
+ name: Logs
+
+ - path: 'r\*.Rcheck\**\*.fail'
+ name: Logs
+
+ - path: 'r\*.Rcheck\**\*.Rout'
+ name: Logs
+
+ - path: 'r\*_*.tar.gz'
+ name: Bits
+
+ - path: 'r\*_*.zip'
+ name: Bits
+
# Disable test discovery
test: off
diff --git a/ci/appveyor-filter-changes.bat b/ci/appveyor-filter-changes.bat
index 04da517..bac8f4c 100644
--- a/ci/appveyor-filter-changes.bat
+++ b/ci/appveyor-filter-changes.bat
@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
echo ===
appveyor exit
)
+) else if "%JOB:~,2%" == "R " (
+ if "%ARROW_CI_R_AFFECTED%" == "0" (
+ echo ===
+ echo === No C++ or R changes, exiting job
+ echo ===
+ appveyor exit
+ )
) else (
if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
echo ===
diff --git a/dev/release/00-prepare-test.rb b/dev/release/00-prepare-test.rb
index 8a1936d..8ce8ca1 100644
--- a/dev/release/00-prepare-test.rb
+++ b/dev/release/00-prepare-test.rb
@@ -131,13 +131,6 @@ class PrepareTest < Test::Unit::TestCase
],
},
{
- path: "r/src/Makevars.win",
- hunks: [
- ["-VERSION = #{@previous_version}.9000",
- "+VERSION = #{@release_version}"],
- ],
- },
- {
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
hunks: [
["- VERSION = \"#{@snapshot_version}\"",
@@ -280,13 +273,6 @@ class PrepareTest < Test::Unit::TestCase
],
},
{
- path: "r/src/Makevars.win",
- hunks: [
- ["-VERSION = #{@release_version}",
- "+VERSION = #{@release_version}.9000"],
- ],
- },
- {
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
hunks: [
["- VERSION = \"#{@release_version}\"",
diff --git a/dev/release/00-prepare.sh b/dev/release/00-prepare.sh
index 724df83..f9b891f 100755
--- a/dev/release/00-prepare.sh
+++ b/dev/release/00-prepare.sh
@@ -101,14 +101,6 @@ update_versions() {
git add DESCRIPTION
cd -
- cd "${SOURCE_DIR}/../../r/src"
- sed -i.bak -E -e \
- "s/^VERSION = .+/VERSION = ${r_version}/" \
- Makevars.win
- rm -f Makevars.win.bak
- git add Makevars.win
- cd -
-
cd "${SOURCE_DIR}/../../ruby"
sed -i.bak -E -e \
"s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
diff --git a/r/configure.win b/r/configure.win
new file mode 100644
index 0000000..75702e6
--- /dev/null
+++ b/r/configure.win
@@ -0,0 +1,47 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# generate code
+if [ "$ARROW_R_DEV" == "TRUE" ]; then
+ echo "*** Generating code with data-raw/codegen.R"
+ "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
+fi
+
+VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
+# Try to find/download a C++ Arrow binary
+"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
+# If binary not found, script exits nonzero
+if [ $? -ne 0 ]; then
+ # We'll have to install without libarrow. Note to the user how to rectify.
+ echo "------------------------- NOTE ---------------------------"
+ echo "After installation, please run arrow::install_arrow() to install"
+ echo "required runtime libraries"
+ echo "---------------------------------------------------------"
+else
+ # Set the right flags to point to and enable arrow/parquet
+ RWINLIB="../windows/arrow-${VERSION}"
+ PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
+ PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
+fi
+
+echo "*** Writing Makevars.win"
+sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
+# Success
+exit 0
diff --git a/r/src/Makevars.win b/r/src/Makevars.win
deleted file mode 100644
index ebad1d5..0000000
--- a/r/src/Makevars.win
+++ /dev/null
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-VERSION = 0.13.0.9000
-RWINLIB = ../windows/arrow-$(VERSION)
-PKG_CPPFLAGS = -I$(RWINLIB)/include -DARROW_STATIC -DPARQUET_STATIC \
- -DARROW_R_WITH_PARQUET
-CXX_STD = CXX11
-
-PKG_LIBS = \
- -L$(RWINLIB)/lib$(subst gcc,,$(COMPILED_BY))$(R_ARCH) \
- -L$(RWINLIB)/lib$(R_ARCH) \
- -lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32
-
-#all: clean
-all: $(SHLIB)
-
-$(OBJECTS): winlibs
-
-winlibs:
- "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "../tools/winlibs.R" $(VERSION)
-
-clean:
- rm -f $(SHLIB) $(OBJECTS)
diff --git a/r/tools/winlibs.R b/r/tools/winlibs.R
index 12391c7..4f6b4fa 100644
--- a/r/tools/winlibs.R
+++ b/r/tools/winlibs.R
@@ -17,10 +17,10 @@
# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
-if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
+if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
if(getRversion() < "3.3.0") setInternet2()
download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
- dir.create("../windows", showWarnings = FALSE)
- unzip("lib.zip", exdir = "../windows")
+ dir.create("windows", showWarnings = FALSE)
+ unzip("lib.zip", exdir = "windows")
unlink("lib.zip")
}