You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2017/09/17 17:46:03 UTC

[1/6] arrow git commit: [Release] Update CHANGELOG.md for 0.7.0 [Forced Update!]

Repository: arrow
Updated Branches:
  refs/heads/master 40e72f5fb -> 59b24ba3b (forced update)


[Release] Update CHANGELOG.md for 0.7.0

Change-Id: I29a15bf02595ac5de6615b97597bb3ff3c493f9a


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/b671dccb
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/b671dccb
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/b671dccb

Branch: refs/heads/master
Commit: b671dccbffd69f4c2177ec469e3cd1369ede2af5
Parents: 576fb66
Author: Wes McKinney <we...@twosigma.com>
Authored: Tue Sep 12 17:58:08 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Tue Sep 12 17:58:08 2017 -0400

----------------------------------------------------------------------
 CHANGELOG.md | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/b671dccb/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8fa4e42..4858df8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,147 @@
   under the License.
 -->
 
+# Apache Arrow 0.7.0 (12 September 2017)
+
+## Bug
+
+* ARROW-1302 - C++: ${MAKE} variable not set sometimes on older MacOS installations
+* ARROW-1354 - [Python] Segfault in Table.from\_pandas with Mixed-Type Categories
+* ARROW-1357 - [Python] Data corruption in reading multi-file parquet dataset
+* ARROW-1363 - [C++] IPC writer sends buffer layout for dictionary rather than indices
+* ARROW-1365 - [Python] Remove usage of removed jemalloc\_memory\_pool in Python API docs
+* ARROW-1373 - [Java] Implement get<type>Buffer() methods at the ValueVector interface
+* ARROW-1375 - [C++] Visual Studio 2017 Appveyor builds failing
+* ARROW-1379 - [Java] maven dependency issues - both unused and undeclared
+* ARROW-1407 - Dictionaries can only hold a maximum of 4096 indices
+* ARROW-1411 - [Python] Booleans in Float Columns cause Segfault
+* ARROW-1414 - [GLib] Cast after status check
+* ARROW-1421 - [Python] pyarrow.serialize cannot serialize a Python dict input
+* ARROW-1426 - [Website] The title element of the top page is empty
+* ARROW-1429 - [Python] Error loading parquet file with \_metadata from HDFS
+* ARROW-1430 - [Python] flake8 warnings are not failing CI builds
+* ARROW-1434 - [C++/Python] pyarrow.Array.from\_pandas does not support datetime64[D] arrays
+* ARROW-1435 - [Python] PyArrow not propagating timezone information from Parquet to Python
+* ARROW-1439 - [Packaging] Automate updating RPM in RPM build
+* ARROW-1443 - [Java] Bug on ArrowBuf.setBytes with unsliced ByteBuffers
+* ARROW-1444 - BitVector.splitAndTransfer copies last byte incorrectly 
+* ARROW-1446 - Python: Writing more than 2^31 rows from pandas dataframe causes row count overflow error
+* ARROW-1450 - [Python] Raise proper error if custom serialization handler fails
+* ARROW-1452 - [C++] Make UNUSED macro name more unique so it does not conflict with thirdparty projects
+* ARROW-1453 - [Python] Implement WriteTensor for non-contiguous tensors
+* ARROW-1458 - [Python] Document that HadoopFileSystem.mkdir with create\_parents=False has no effect
+* ARROW-1459 - [Python] PyArrow fails to load partitioned parquet files with non-primitive types
+* ARROW-1461 - [C++] Disable builds using LLVM apt packages temporarily
+* ARROW-1467 - [JAVA]: Fix reset() and allocateNew() in Nullable Value Vectors template
+* ARROW-1490 - [Java] Allow Travis CI failures for JDK9 for now
+* ARROW-1493 - [C++] Flush the output stream at the end of each PrettyPrint function
+* ARROW-1495 - [C++] Store shared\_ptr to boxed arrays in RecordBatch
+* ARROW-1507 - [C++] arrow/compute/api.h can't be used without arrow/array.h
+* ARROW-1512 - [Docs] NumericArray has no member named 'raw\_data'
+* ARROW-1514 - [C++] Fix a typo in document
+* ARROW-1527 - Fix Travis JDK9 build
+* ARROW-1531 - [C++] Return ToBytes by value from Decimal128
+* ARROW-1532 - [Python] Referencing an Empty Schema causes a SegFault
+* ARROW-407 - BitVector.copyFromSafe() should re-allocate if necessary instead of returning false
+* ARROW-801 - [JAVA] Provide direct access to underlying buffer memory addresses in consistent way without generating garbage or large amount indirections
+
+## Improvement
+
+* ARROW-1307 - [Python] Add pandas serialization section + Feather API to Sphinx docs
+* ARROW-1317 - [Python] Add function to set Hadoop CLASSPATH 
+* ARROW-1331 - [Java] Refactor tests
+* ARROW-1339 - [C++] Use boost::filesystem for handling of platform-specific file path encodings
+* ARROW-1344 - [C++] Calling BufferOutputStream::Write after calling Finish crashes
+* ARROW-1348 - [C++/Python] Add release verification script for Windows
+* ARROW-1351 - Automate updating CHANGELOG.md as part of release scripts
+* ARROW-1352 - [Integration] Improve print formatting for producer, consumer line
+* ARROW-1355 - Make arrow buildable with java9
+* ARROW-1356 - [Website] Add new committers
+* ARROW-1358 - Update source release scripts to account for new SHA checksum policy
+* ARROW-1359 - [Python] Add Parquet writer option to normalize field names for use in Spark
+* ARROW-1366 - [Python] Add instructions for starting the Plasma store when installing pyarrow from wheels
+* ARROW-1372 - [Plasma] Support for storing data in huge pages
+* ARROW-1376 - [C++] RecordBatchStreamReader::Open API is inconsistent with writer
+* ARROW-1381 - [Python] Improve performance of SerializedPyObject.to\_buffer
+* ARROW-1383 - [C++] Support std::vector<bool> in builder vector appends
+* ARROW-1384 - [C++] Add convenience function for serializing a record batch to an IPC message 
+* ARROW-1386 - [C++] Unpin CMake version in MSVC build toolchain
+* ARROW-1395 - [C++] Remove APIs deprecated as of 0.5.0 and later versions
+* ARROW-1397 - [Packaging] Use Docker instead of Vagrant
+* ARROW-1401 - [C++] Add extra debugging context to failures in RETURN\_NOT\_OK in debug builds
+* ARROW-1402 - [C++] Possibly deprecate public APIs that use MutableBuffer
+* ARROW-1404 - [Packaging] Build .deb and .rpm on Travis CI
+* ARROW-1405 - [Python] Add logging option for verbose memory allocations
+* ARROW-1406 - [Python] Harden user API for generating serialized schema and record batch messages as memoryview-compatible objects
+* ARROW-1408 - [C++] Refactor and make IPC read / write APIs more consistent, add appropriate deprecations
+* ARROW-1410 - Plasma object store occasionally pauses for a long time
+* ARROW-1412 - [Plasma] Add higher level API for putting and getting Python objects
+* ARROW-1413 - [C++] Add include-what-you-use configuration
+* ARROW-1416 - [Format] Clarify example array in memory layout documentation
+* ARROW-1418 - [Python] Introduce SerializationContext to register custom serialization callbacks
+* ARROW-1419 - [GLib] Suppress sign-conversion warning on Clang
+* ARROW-1427 - [GLib] Add a link to readme of Arrow GLib
+* ARROW-1428 - [C++] Append steps to clone source code to README.mb
+* ARROW-1432 - [C++] Build bundled jemalloc functions with private prefix
+* ARROW-1433 - [C++] Simplify implementation of Array::Slice
+* ARROW-1438 - [Plasma] Pull SerializationContext through PlasmaClient put and get
+* ARROW-1441 - [Site] Add Ruby to Flexible section
+* ARROW-1442 - [Website] Add pointer to nightly conda packages on /install
+* ARROW-1447 - [C++] Round of include-what-you-use include cleanups
+* ARROW-1448 - [Packaging] Support uploading built .deb and .rpm to Bintray
+* ARROW-1449 - Implement Decimal using only Int128
+* ARROW-1451 - [C++] Create arrow/io/api.h
+* ARROW-1460 - [C++] Upgrade clang-format used to LLVM 4.0
+* ARROW-1466 - [C++] Support DecimalArray in arrow::PrettyPrint
+* ARROW-1468 - [C++] Append to PrimitiveBuilder from std::vector<CTYPE>
+* ARROW-1480 - [Python] Improve performance of serializing sets
+* ARROW-1494 - [C++] Document that shared\_ptr returned by RecordBatch::column needs to be retained
+* ARROW-1499 - [Python] Consider adding option to parquet.write\_table that sets options for maximum Spark compatibility
+* ARROW-1505 - [GLib] Simplify arguments check
+* ARROW-1506 - [C++] Support pkg-config for compute modules
+* ARROW-1508 - C++: Add support for FixedSizeBinaryType in DictionaryBuilder
+* ARROW-1511 - [C++] Deprecate arrow::MakePrimitiveArray
+* ARROW-1513 - C++: Add cast from Dictionary to plain arrays
+* ARROW-1515 - [GLib] Detect version directly
+* ARROW-1516 - [GLib] Update document
+* ARROW-1517 - Remove unnecessary temporary in DecimalUtil::ToString function
+* ARROW-1519 - [C++] Move DecimalUtil functions to methods on the Int128 class
+* ARROW-1528 - [GLib] Resolve include dependency
+* ARROW-1530 - [C++] Install arrow/util/parallel.h
+* ARROW-594 - [Python] Provide interface to write pyarrow.Table to a stream
+* ARROW-786 - [Format] In-memory format for 128-bit Decimals, handling of sign bit
+* ARROW-837 - [Python] Expose buffer allocation, FixedSizeBufferWriter
+* ARROW-941 - [Docs] Improve "cold start" integration testing instructions
+
+## New Feature
+
+* ARROW-1156 - [Python] pyarrow.Array.from\_pandas should take a type parameter
+* ARROW-1238 - [Java] Add JSON read/write support for decimals for integration tests
+* ARROW-1364 - [C++] IPC reader and writer specialized for GPU device memory
+* ARROW-1377 - [Python] Add function to assist with benchmarking Parquet scan performance
+* ARROW-1387 - [C++] Set up GPU leaf library build toolchain
+* ARROW-1392 - [C++] Implement reader and writer IO interfaces for GPU buffers
+* ARROW-1396 - [C++] Add PrettyPrint function for Schemas, which also outputs any dictionaries
+* ARROW-1399 - [C++] Add CUDA build version in a public header to help prevent ABI conflicts
+* ARROW-1400 - [Python] Ability to create partitions when writing to Parquet
+* ARROW-1415 - [GLib] Support date32 and date64
+* ARROW-1417 - [Python] Allow more generic filesystem objects to be passed to ParquetDataset
+* ARROW-1462 - [GLib] Support time array
+* ARROW-1479 - [JS] Expand JavaScript implementation
+* ARROW-1481 - [C++] Expose type casts as generic callable object that can write into pre-allocated memory
+* ARROW-1504 - [GLib] Support timestamp
+* ARROW-1510 - [C++] Support cast
+* ARROW-229 - [C++] Implement safe casts for primitive types
+* ARROW-592 - [C++] Provide .deb and .rpm packages
+* ARROW-695 - Integration tests for Decimal types
+* ARROW-696 - [C++] Add JSON read/write support for decimals for integration tests
+* ARROW-759 - [Python] Implement a transient list serialization function that can handle a mix of scalars, lists, ndarrays, dicts
+* ARROW-989 - [Python] Write pyarrow.Table to FileWriter or StreamWriter
+
+## Test
+
+* ARROW-1390 - [Python] Extend tests for python serialization
+
 # Apache Arrow 0.6.0 (14 August 2017)
 
 ## Bug
@@ -1102,3 +1243,4 @@
 
 * ARROW-260 - TestValueVector.testFixedVectorReallocation and testVariableVectorReallocation are flaky
 * ARROW-83 - Add basic test infrastructure for DecimalType
+


[5/6] arrow git commit: ARROW-559: Add release verification script for Linux

Posted by we...@apache.org.
ARROW-559: Add release verification script for Linux

Since we're accumulating a bunch of components, I started this script which we can refine to make verifying releases easier for others.

I bootstrapped some pieces off https://github.com/apache/parquet-cpp/blob/master/dev/release/verify-release-candidate, very helpful!

This script:

* Checks GPG signature, checksums
* Installs temporary Python install for the duration of these tests
* Builds/install C++ and runs tests (with Python and Plasma)
* Builds parquet-cpp against the Arrow RC
* Python (with Parquet and Plasma extensions)
* C GLib (requires Ruby in PATH and the gems indicated in README)
* Integration tests
* JavaScript (requires NodeJS >= 6.0.0)

There are some potentially snowflake-y aspects to my environment:

* BOOST_ROOT is set to a Boost install location containing libraries built with `-fPIC`. I'm not sure what to do about this one. One maybe better option is to use system level boost and shared libraries
* Maven 3.3.9 is in PATH
* NodeJS 6.11.3 is in PATH

There are probably some other things that Linux users will run into as they run this script.

I had to compile GLib libraries in this since the ones at system level (Ubuntu 14.04) are too old.

cc @kou @xhochy

Author: Wes McKinney <we...@twosigma.com>

Closes #1102 from wesm/ARROW-559 and squashes the following commits:

8fd6530 [Wes McKinney] Use Boost shared libraries
3531927 [Wes McKinney] Add note to dev/README.md
079b5e4 [Wes McKinney] Fix comments
17f7ac0 [Wes McKinney] More fixes, finally works
adb3146 [Wes McKinney] More work on release verification script
86ef171 [Wes McKinney] Start Linux release verification script


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/59b24ba3
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/59b24ba3
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/59b24ba3

Branch: refs/heads/master
Commit: 59b24ba3b81bb8ead4fa74ff15ebe633d408415e
Parents: 72ad07e
Author: Wes McKinney <we...@twosigma.com>
Authored: Fri Sep 15 12:29:10 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Sun Sep 17 13:45:20 2017 -0400

----------------------------------------------------------------------
 dev/README.md                           |  18 ++
 dev/release/verify-release-candidate.sh | 244 +++++++++++++++++++++++++++
 2 files changed, 262 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/59b24ba3/dev/README.md
----------------------------------------------------------------------
diff --git a/dev/README.md b/dev/README.md
index e986abe..91d688d 100644
--- a/dev/README.md
+++ b/dev/README.md
@@ -92,3 +92,21 @@ Merge hash: 485658a5
 Would you like to pick 485658a5 into another branch? (y/n):
 ```
 For now just say n as we have 1 branch
+
+## Verifying Release Candidates
+
+We have provided a script to assist with verifying release candidates:
+
+```shell
+bash dev/release/verify-release-candidate.sh 0.7.0 0
+```
+
+Currently this only works on Linux (patches to expand to macOS welcome!). Read
+the script for information about system dependencies.
+
+On Windows, we have a script that verifies C++ and Python (requires Visual
+Studio 2015):
+
+```
+dev/release/verify-release-candidate.bat apache-arrow-0.7.0.tar.gz
+```
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/arrow/blob/59b24ba3/dev/release/verify-release-candidate.sh
----------------------------------------------------------------------
diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
new file mode 100755
index 0000000..38680f4
--- /dev/null
+++ b/dev/release/verify-release-candidate.sh
@@ -0,0 +1,244 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Requirements
+# - Ruby 2.x
+#   - Plus gem dependencies, see c_glib/README
+# - Maven >= 3.3.9
+# - JDK >=7
+# - gcc >= 4.8
+# - nodejs >= 6.0.0 (best way is to use nvm)
+#
+# If using a non-system Boost, set BOOST_ROOT and add Boost libraries to
+# LD_LIBRARY_PATH
+
+case $# in
+  2) VERSION="$1"
+     RC_NUMBER="$2"
+     ;;
+
+  *) echo "Usage: $0 X.Y.Z RC_NUMBER"
+     exit 1
+     ;;
+esac
+
+set -ex
+
+HERE=$(cd `dirname "${BASH_SOURCE[0]:-$0}"` && pwd)
+
+ARROW_DIST_URL='https://dist.apache.org/repos/dist/dev/arrow'
+
+download_dist_file() {
+  curl -f -O $ARROW_DIST_URL/$1
+}
+
+download_rc_file() {
+  download_dist_file apache-arrow-${VERSION}-rc${RC_NUMBER}/$1
+}
+
+import_gpg_keys() {
+  download_dist_file KEYS
+  gpg --import KEYS
+}
+
+fetch_archive() {
+  local dist_name=$1
+  download_rc_file ${dist_name}.tar.gz
+  download_rc_file ${dist_name}.tar.gz.asc
+  download_rc_file ${dist_name}.tar.gz.md5
+  download_rc_file ${dist_name}.tar.gz.sha512
+  gpg --verify ${dist_name}.tar.gz.asc ${dist_name}.tar.gz
+  gpg --print-md MD5 ${dist_name}.tar.gz | diff - ${dist_name}.tar.gz.md5
+  sha512sum ${dist_name}.tar.gz | diff - ${dist_name}.tar.gz.sha512
+}
+
+setup_tempdir() {
+  cleanup() {
+    rm -fr "$TMPDIR"
+  }
+  trap cleanup EXIT
+  TMPDIR=$(mktemp -d -t "$1.XXXXX")
+}
+
+
+setup_miniconda() {
+  # Setup short-lived miniconda for Python and integration tests
+  MINICONDA_URL=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
+
+  MINICONDA=`pwd`/test-miniconda
+
+  wget -O miniconda.sh $MINICONDA_URL
+  bash miniconda.sh -b -p $MINICONDA
+  rm -f miniconda.sh
+
+  export PATH=$MINICONDA/bin:$PATH
+
+  conda create -n arrow-test -y -q python=3.6 \
+        nomkl numpy pandas six cython
+  source activate arrow-test
+}
+
+# Build and test C++
+
+test_and_install_cpp() {
+  mkdir cpp/build
+  pushd cpp/build
+
+  cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
+        -DARROW_PLASMA=on \
+        -DARROW_PYTHON=on \
+        -DARROW_BOOST_USE_SHARED=on \
+        -DCMAKE_BUILD_TYPE=release \
+        -DARROW_BUILD_BENCHMARKS=on \
+        ..
+
+  make -j$NPROC
+  make install
+
+  ctest -L unittest
+  popd
+}
+
+# Build and install Parquet master so we can test the Python bindings
+
+install_parquet_cpp() {
+  git clone git@github.com:apache/parquet-cpp.git
+
+  mkdir parquet-cpp/build
+  pushd parquet-cpp/build
+
+  cmake -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
+        -DCMAKE_BUILD_TYPE=release \
+        -DPARQUET_BOOST_USE_SHARED=on \
+        -DPARQUET_BUILD_TESTS=off \
+        ..
+
+  make -j$NPROC
+  make install
+
+  popd
+}
+
+# Build and test Python
+
+test_python() {
+  pushd python
+
+  pip install -r requirements.txt
+
+  python setup.py build_ext --inplace --with-parquet --with-plasma
+  py.test pyarrow -v --pdb
+
+  popd
+}
+
+
+test_glib() {
+  # Build and test GLib, requires GLib >= 2.44 , so install that
+  # here
+  GLIB_VERSION=glib-2.53.7
+  GLIB_URL=https://gensho.ftp.acc.umu.se/pub/gnome/sources/glib/2.53/$GLIB_VERSION.tar.xz
+  curl -f -O $GLIB_URL
+  tar xf $GLIB_VERSION.tar.xz
+  pushd $GLIB_VERSION
+
+  ./configure --disable-libelf --enable-libmount=no --prefix=$ARROW_HOME
+  make -j$NPROC
+  make install
+
+  popd
+
+  pushd c_glib
+
+  ./configure --prefix=$ARROW_HOME
+  make -j$NPROC
+  make install
+
+  NO_MAKE=yes test/run-test.sh
+
+  popd
+}
+
+test_js() {
+  pushd js
+  npm install
+  npm run validate
+  popd
+}
+
+# Build and test Java (Requires newer Maven -- I used 3.3.9)
+
+test_package_java() {
+  pushd java
+
+  mvn test
+  mvn package
+
+  popd
+}
+
+# Run integration tests
+test_integration() {
+  JAVA_DIR=`pwd`/java
+  CPP_BUILD_DIR=`pwd`/cpp/build
+
+  export ARROW_JAVA_INTEGRATION_JAR=$JAVA_DIR/tools/target/arrow-tools-$VERSION-jar-with-dependencies.jar
+  export ARROW_CPP_EXE_PATH=$CPP_BUILD_DIR/release
+
+  pushd integration
+
+  python integration_test.py
+
+  popd
+}
+
+setup_tempdir "arrow-$VERSION"
+echo "Working in sandbox $TMPDIR"
+cd $TMPDIR
+
+export ARROW_HOME=$TMPDIR/install
+export PARQUET_HOME=$TMPDIR/install
+export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
+export PKG_CONFIG_PATH=$ARROW_HOME/lib/pkgconfig:$PKG_CONFIG_PATH
+
+NPROC=$(nproc)
+VERSION=$1
+RC_NUMBER=$2
+
+TARBALL=apache-arrow-$1.tar.gz
+
+import_gpg_keys
+
+DIST_NAME="apache-arrow-${VERSION}"
+fetch_archive $DIST_NAME
+tar xvzf ${DIST_NAME}.tar.gz
+cd ${DIST_NAME}
+
+setup_miniconda
+test_and_install_cpp
+install_parquet_cpp
+test_python
+test_glib
+test_package_java
+test_integration
+test_js
+
+echo 'Release candidate looks good!'
+exit 0


[4/6] arrow git commit: ARROW-1542: [C++] Install packages in temporary directory in MSVC build verification script

Posted by we...@apache.org.
ARROW-1542: [C++] Install packages in temporary directory in MSVC build verification script

I found that the script did not work due to the remnants of the last time I ran it.

Author: Wes McKinney <we...@twosigma.com>

Closes #1101 from wesm/ARROW-1542 and squashes the following commits:

0718370 [Wes McKinney] Install packages in temporary directory in MSVC build verification script


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/72ad07e9
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/72ad07e9
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/72ad07e9

Branch: refs/heads/master
Commit: 72ad07e96648cdc3bc7d2c6f48c732dbe3133080
Parents: 96d451f
Author: Wes McKinney <we...@twosigma.com>
Authored: Fri Sep 15 10:48:22 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Sun Sep 17 13:45:20 2017 -0400

----------------------------------------------------------------------
 dev/release/verify-release-candidate.bat | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/72ad07e9/dev/release/verify-release-candidate.bat
----------------------------------------------------------------------
diff --git a/dev/release/verify-release-candidate.bat b/dev/release/verify-release-candidate.bat
index 503f1fd..27157da 100644
--- a/dev/release/verify-release-candidate.bat
+++ b/dev/release/verify-release-candidate.bat
@@ -45,15 +45,18 @@ tar xvf %1.tar.gz -C "C:/tmp/"
 set GENERATOR=Visual Studio 14 2015 Win64
 set CONFIGURATION=release
 set ARROW_SOURCE=C:\tmp\%1
+set INSTALL_DIR=C:\tmp\%1\install
 
 pushd %ARROW_SOURCE%
 
 call activate arrow-verify-release
 
 set ARROW_BUILD_TOOLCHAIN=%CONDA_PREFIX%\Library
-set ARROW_HOME=%CONDA_PREFIX%\Library
 set PARQUET_BUILD_TOOLCHAIN=%CONDA_PREFIX%\Library
-set PARQUET_HOME=%CONDA_PREFIX%\Library
+
+set ARROW_HOME=%INSTALL_DIR%
+set PARQUET_HOME=%INSTALL_DIR%
+set PATH=%INSTALL_DIR%\bin;%PATH%
 
 @rem Build and test Arrow C++ libraries
 mkdir cpp\build


[3/6] arrow git commit: ARROW-1537: [C++] Support building with full path install_name on macOS

Posted by we...@apache.org.
ARROW-1537: [C++] Support building with full path install_name on macOS

If you use `@rpath` for install_name (default), you can use the
DYLD_LIBRARY_PATH environment variable to find libarrow.dylib. But the
DYLD_LIBRARY_PATH environment variable isn't inherited to sub process by
System Integration Protection (SIP). It's difficult to use
libarrow.dylib.

You can use full path install_name by -DARROW_INSTALL_NAME_RPATH=OFF
CMake option. If you use it, you can find libarrow.dylib without
DYLD_LIBRARY_PATH environment variable.

Author: Kouhei Sutou <ko...@clear-code.com>

Closes #1100 from kou/cpp-macos-support-install-name and squashes the following commits:

8207ace [Kouhei Sutou] [C++] Support building with full path install_name on macOS


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/96d451f3
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/96d451f3
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/96d451f3

Branch: refs/heads/master
Commit: 96d451f37eb23d9c9cdc997032d7534a585537ff
Parents: d0b0b74
Author: Kouhei Sutou <ko...@clear-code.com>
Authored: Fri Sep 15 10:47:04 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Sun Sep 17 13:45:20 2017 -0400

----------------------------------------------------------------------
 ci/travis_before_script_c_glib.sh  | 6 ------
 ci/travis_before_script_cpp.sh     | 3 ++-
 cpp/CMakeLists.txt                 | 4 ++++
 cpp/cmake_modules/BuildUtils.cmake | 9 +++++++--
 4 files changed, 13 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/96d451f3/ci/travis_before_script_c_glib.sh
----------------------------------------------------------------------
diff --git a/ci/travis_before_script_c_glib.sh b/ci/travis_before_script_c_glib.sh
index 533c892..52bfe87 100755
--- a/ci/travis_before_script_c_glib.sh
+++ b/ci/travis_before_script_c_glib.sh
@@ -77,12 +77,6 @@ pushd $ARROW_C_GLIB_DIR
 export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ARROW_CPP_INSTALL/lib/pkgconfig
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ARROW_CPP_INSTALL/lib
 
-if [ $TRAVIS_OS_NAME == "osx" ]; then
-  install_name_tool \
-    -id $ARROW_CPP_INSTALL/lib/libarrow.dylib \
-    $ARROW_CPP_INSTALL/lib/libarrow.dylib
-fi
-
 CONFIGURE_OPTIONS="--prefix=$ARROW_C_GLIB_INSTALL"
 if [ $TRAVIS_OS_NAME != "osx" ]; then
   CONFIGURE_OPTIONS="$CONFIGURE_OPTIONS --enable-gtk-doc"

http://git-wip-us.apache.org/repos/asf/arrow/blob/96d451f3/ci/travis_before_script_cpp.sh
----------------------------------------------------------------------
diff --git a/ci/travis_before_script_cpp.sh b/ci/travis_before_script_cpp.sh
index a613957..d46fa2f 100755
--- a/ci/travis_before_script_cpp.sh
+++ b/ci/travis_before_script_cpp.sh
@@ -70,7 +70,8 @@ if [ $only_library_mode == "yes" ]; then
   CMAKE_COMMON_FLAGS="\
 $CMAKE_COMMON_FLAGS \
 -DARROW_BUILD_TESTS=OFF \
--DARROW_BUILD_UTILITIES=OFF"
+-DARROW_BUILD_UTILITIES=OFF \
+-DARROW_INSTALL_NAME_RPATH=OFF"
 fi
 
 # Use Ninja for faster builds when using toolchain

http://git-wip-us.apache.org/repos/asf/arrow/blob/96d451f3/cpp/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt
index 577a4bb..972132f 100644
--- a/cpp/CMakeLists.txt
+++ b/cpp/CMakeLists.txt
@@ -150,6 +150,10 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
     "Build Arrow libraries with RATH set to \$ORIGIN"
     OFF)
 
+  option(ARROW_INSTALL_NAME_RPATH
+    "Build Arrow libraries with install_name set to @rpath"
+    ON)
+
   option(ARROW_PLASMA
     "Build the plasma object store along with Arrow"
     OFF)

http://git-wip-us.apache.org/repos/asf/arrow/blob/96d451f3/cpp/cmake_modules/BuildUtils.cmake
----------------------------------------------------------------------
diff --git a/cpp/cmake_modules/BuildUtils.cmake b/cpp/cmake_modules/BuildUtils.cmake
index 6b2be41..8f92d73 100644
--- a/cpp/cmake_modules/BuildUtils.cmake
+++ b/cpp/cmake_modules/BuildUtils.cmake
@@ -176,10 +176,15 @@ function(ADD_ARROW_LIB LIB_NAME)
   endif()
 
   if (APPLE)
-      set_target_properties(${LIB_NAME}_shared
+    if (ARROW_INSTALL_NAME_RPATH)
+      set(_lib_install_name "@rpath")
+    else()
+      set(_lib_install_name "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
+    endif()
+    set_target_properties(${LIB_NAME}_shared
       PROPERTIES
       BUILD_WITH_INSTALL_RPATH ON
-      INSTALL_NAME_DIR "@rpath")
+      INSTALL_NAME_DIR "${_lib_install_name}")
   endif()
 
 endfunction()


[6/6] arrow git commit: ARROW-1529: [GLib] Use Xcode 8.3 on Travis CI

Posted by we...@apache.org.
ARROW-1529: [GLib] Use Xcode 8.3 on Travis CI

Author: Kouhei Sutou <ko...@clear-code.com>

Closes #1092 from kou/glib-travis-macos and squashes the following commits:

291808b2 [Kouhei Sutou] [GLib] Use Xcode 8.3 on Travis CI


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/d0b0b741
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/d0b0b741
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/d0b0b741

Branch: refs/heads/master
Commit: d0b0b74181fd94c22996014b9a13e953ed7bbadd
Parents: 97f9029
Author: Kouhei Sutou <ko...@clear-code.com>
Authored: Wed Sep 13 09:09:38 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Sun Sep 17 13:45:20 2017 -0400

----------------------------------------------------------------------
 .travis.yml                       |  2 +-
 ci/travis_before_script_c_glib.sh | 13 +++++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/d0b0b741/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index c4a8f41..f6f6400 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -128,7 +128,7 @@ matrix:
     script:
     - $TRAVIS_BUILD_DIR/ci/travis_script_c_glib.sh
   - compiler: clang
-    osx_image: xcode6.4
+    osx_image: xcode8.3
     os: osx
     cache:
     addons:

http://git-wip-us.apache.org/repos/asf/arrow/blob/d0b0b741/ci/travis_before_script_c_glib.sh
----------------------------------------------------------------------
diff --git a/ci/travis_before_script_c_glib.sh b/ci/travis_before_script_c_glib.sh
index 6799ae4..533c892 100755
--- a/ci/travis_before_script_c_glib.sh
+++ b/ci/travis_before_script_c_glib.sh
@@ -23,7 +23,9 @@ source $TRAVIS_BUILD_DIR/ci/travis_env_common.sh
 
 if [ $TRAVIS_OS_NAME == "osx" ]; then
   brew install gtk-doc autoconf-archive gobject-introspection
-  brew upgrade git cmake wget libtool
+  brew upgrade git cmake
+  brew outdated || brew upgrade wget
+  brew outdated || brew upgrade libtool
 
   export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/opt/libffi/lib/pkgconfig
 else
@@ -36,6 +38,7 @@ gem install test-unit gobject-introspection
 
 if [ $TRAVIS_OS_NAME == "osx" ]; then
   brew install lua
+  sudo env PKG_CONFIG_PATH=$PKG_CONFIG_PATH luarocks install lgi
 else
   git clone \
     --quiet \
@@ -47,8 +50,8 @@ else
   echo "yes" | ./install.sh > /dev/null
   . ~/torch/install/bin/torch-activate
   popd
+  luarocks install lgi
 fi
-luarocks install lgi
 
 go get github.com/linuxdeepin/go-gir-generator || :
 pushd $GOPATH/src/github.com/linuxdeepin/go-gir-generator
@@ -74,6 +77,12 @@ pushd $ARROW_C_GLIB_DIR
 export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ARROW_CPP_INSTALL/lib/pkgconfig
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ARROW_CPP_INSTALL/lib
 
+if [ $TRAVIS_OS_NAME == "osx" ]; then
+  install_name_tool \
+    -id $ARROW_CPP_INSTALL/lib/libarrow.dylib \
+    $ARROW_CPP_INSTALL/lib/libarrow.dylib
+fi
+
 CONFIGURE_OPTIONS="--prefix=$ARROW_C_GLIB_INSTALL"
 if [ $TRAVIS_OS_NAME != "osx" ]; then
   CONFIGURE_OPTIONS="$CONFIGURE_OPTIONS --enable-gtk-doc"


[2/6] arrow git commit: [maven-release-plugin] prepare release apache-arrow-0.7.0

Posted by we...@apache.org.
[maven-release-plugin] prepare release apache-arrow-0.7.0

Change-Id: Ib082c383446182630abbb6c8e2e154006ff4560f


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/97f9029c
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/97f9029c
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/97f9029c

Branch: refs/heads/master
Commit: 97f9029ce835dfc2655ca91b9820a2e6aed89107
Parents: b671dcc
Author: Wes McKinney <we...@twosigma.com>
Authored: Tue Sep 12 18:01:14 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Tue Sep 12 18:01:14 2017 -0400

----------------------------------------------------------------------
 java/format/pom.xml | 2 +-
 java/memory/pom.xml | 2 +-
 java/pom.xml        | 4 ++--
 java/tools/pom.xml  | 2 +-
 java/vector/pom.xml | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/97f9029c/java/format/pom.xml
----------------------------------------------------------------------
diff --git a/java/format/pom.xml b/java/format/pom.xml
index 647b716..684ed3c 100644
--- a/java/format/pom.xml
+++ b/java/format/pom.xml
@@ -15,7 +15,7 @@
 <parent>
   <artifactId>arrow-java-root</artifactId>
   <groupId>org.apache.arrow</groupId>
-  <version>0.7.0-SNAPSHOT</version>
+  <version>0.7.0</version>
 </parent>
 
 <artifactId>arrow-format</artifactId>

http://git-wip-us.apache.org/repos/asf/arrow/blob/97f9029c/java/memory/pom.xml
----------------------------------------------------------------------
diff --git a/java/memory/pom.xml b/java/memory/pom.xml
index 1c4eef7..dcd3572 100644
--- a/java/memory/pom.xml
+++ b/java/memory/pom.xml
@@ -14,7 +14,7 @@
   <parent>
     <groupId>org.apache.arrow</groupId>
     <artifactId>arrow-java-root</artifactId>
-    <version>0.7.0-SNAPSHOT</version>
+    <version>0.7.0</version>
   </parent>
   <artifactId>arrow-memory</artifactId>
   <name>Arrow Memory</name>

http://git-wip-us.apache.org/repos/asf/arrow/blob/97f9029c/java/pom.xml
----------------------------------------------------------------------
diff --git a/java/pom.xml b/java/pom.xml
index c4c301d..acd5626 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -20,7 +20,7 @@
 
   <groupId>org.apache.arrow</groupId>
   <artifactId>arrow-java-root</artifactId>
-  <version>0.7.0-SNAPSHOT</version>
+  <version>0.7.0</version>
   <packaging>pom</packaging>
 
   <name>Apache Arrow Java Root POM</name>
@@ -44,7 +44,7 @@
     <connection>scm:git:https://git-wip-us.apache.org/repos/asf/arrow.git</connection>
     <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/arrow.git</developerConnection>
     <url>https://github.com/apache/arrow</url>
-    <tag>HEAD</tag>
+    <tag>apache-arrow-0.7.0</tag>
   </scm>
 
   <mailingLists>

http://git-wip-us.apache.org/repos/asf/arrow/blob/97f9029c/java/tools/pom.xml
----------------------------------------------------------------------
diff --git a/java/tools/pom.xml b/java/tools/pom.xml
index 48b507a..a1a6308 100644
--- a/java/tools/pom.xml
+++ b/java/tools/pom.xml
@@ -14,7 +14,7 @@
     <parent>
         <groupId>org.apache.arrow</groupId>
         <artifactId>arrow-java-root</artifactId>
-        <version>0.7.0-SNAPSHOT</version>
+        <version>0.7.0</version>
     </parent>
     <artifactId>arrow-tools</artifactId>
     <name>Arrow Tools</name>

http://git-wip-us.apache.org/repos/asf/arrow/blob/97f9029c/java/vector/pom.xml
----------------------------------------------------------------------
diff --git a/java/vector/pom.xml b/java/vector/pom.xml
index bf33823..dd1a0ea 100644
--- a/java/vector/pom.xml
+++ b/java/vector/pom.xml
@@ -14,7 +14,7 @@
   <parent>
     <groupId>org.apache.arrow</groupId>
     <artifactId>arrow-java-root</artifactId>
-    <version>0.7.0-SNAPSHOT</version>
+    <version>0.7.0</version>
   </parent>
   <artifactId>arrow-vector</artifactId>
   <name>Arrow Vectors</name>