You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@parquet.apache.org by ju...@apache.org on 2016/02/19 00:29:49 UTC
parquet-cpp git commit: PARQUET-471: Use common build environment
setup for sandbox and Travis CI. Travis CI cleanup
Repository: parquet-cpp
Updated Branches:
refs/heads/master 4ae1b921b -> 17a204555
PARQUET-471: Use common build environment setup for sandbox and Travis CI. Travis CI cleanup
This also adds better build instructions for OS X users using Homebrew (googletest is not available and needs to be build from `thirdparty/`).
Closes #53
Author: Wes McKinney <we...@apache.org>
Closes #54 from wesm/PARQUET-471 and squashes the following commits:
162f523 [Wes McKinney] Refactor .travis.yml and associated scripts to have a common sandbox setup script. Add a build matrix and disable unneeded OS X gcc build. Better thirdparty documentation for googletest and build setup
Project: http://git-wip-us.apache.org/repos/asf/parquet-cpp/repo
Commit: http://git-wip-us.apache.org/repos/asf/parquet-cpp/commit/17a20455
Tree: http://git-wip-us.apache.org/repos/asf/parquet-cpp/tree/17a20455
Diff: http://git-wip-us.apache.org/repos/asf/parquet-cpp/diff/17a20455
Branch: refs/heads/master
Commit: 17a204555bede41d1f68e59a540f3bbef072c821
Parents: 4ae1b92
Author: Wes McKinney <we...@apache.org>
Authored: Thu Feb 18 15:29:46 2016 -0800
Committer: Julien Le Dem <ju...@dremio.com>
Committed: Thu Feb 18 15:29:46 2016 -0800
----------------------------------------------------------------------
.travis.yml | 49 +++++++++++++++++++++--------------
README.md | 45 ++++++++++++++++++++++++++------
ci/before_script_travis.sh | 20 ++++----------
ci/run_tests.sh | 5 ----
setup_build_env.sh | 22 +++++++---------
thirdparty/set_thirdparty_env.sh | 18 +++++++++++++
6 files changed, 98 insertions(+), 61 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 09f4705..ad71e29 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,15 +1,5 @@
sudo: required
dist: trusty
-language: cpp
-
-compiler:
- - gcc
- - clang
-
-os:
- - linux
- - osx
-
addons:
apt:
sources:
@@ -28,17 +18,36 @@ addons:
- bison #needed for thrift cpp compilation
- flex #needed for thrift cpp compilation
- pkg-config #needed for thrift cpp compilation
+cache:
+ directories:
+ - $HOME/parquet-build
+matrix:
+ include:
+ - compiler: gcc
+ os: linux
+ script:
+ - make lint
+ - make -j4 || exit 1
+ - valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest
+ - compiler: clang
+ os: linux
+ script:
+ - make -j4 || exit 1
+ - ctest
+ - os: osx
+ compiler: clang
+ addons:
+ script:
+ - make -j4 || exit 1
+ - ctest
+
+language: cpp
before_install:
- - mkdir $HOME/build_dir
- - cd $HOME/build_dir
+- mkdir $HOME/parquet-build
+- pushd $HOME/parquet-build
before_script:
- - source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh
- - cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR
- - export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data
-
-script:
-- make
-- source $TRAVIS_BUILD_DIR/ci/run_tests.sh
-- make lint
+- source $TRAVIS_BUILD_DIR/ci/before_script_travis.sh
+- cmake -DCMAKE_CXX_FLAGS="-Werror" $TRAVIS_BUILD_DIR
+- export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 1578d4e..b6ec89e 100644
--- a/README.md
+++ b/README.md
@@ -1,35 +1,64 @@
-Parquet-cpp [](https://travis-ci.org/apache/parquet-cpp)
-===========
-A C++ library to read parquet files.
+## parquet-cpp: a C++ library to read and write the Apache Parquet columnar data format.
+
+[](https://travis-ci.org/apache/parquet-cpp)
## Third Party Dependencies
+
- snappy
- lz4
+- zlib
- thrift 0.7+ [install instructions](https://thrift.apache.org/docs/install/)
+- googletest 1.7.0 (cannot be installed with package managers)
+
+You can install these dependencies using a package manager or using the
+`thirdparty/` scripts in this repository. On Homebrew, you can run:
-Many package managers support some or all of these dependencies. E.g.:
```shell
-ubuntu$ sudo apt-get install libboost-dev libsnappy-dev liblz4-dev
+ brew install snappy lz4 thrift zlib
```
+
+To build the thirdparty libraries in-tree, run:
+
```shell
-mac$ brew install snappy lz4 thrift
+./thirdparty/download_thirdparty.sh
+./thirdparty/build_thirdparty.sh
+source thirdparty/set_thirdparty_env.sh
```
-`setup_build_env.sh` tries to automate setting up a build environment for you
+The provided script `setup_build_env.sh` sets up a build environment for you
with third party dependencies. You use it by running `source
setup_build_env.sh`. By default, it will create a build directory `build/`.
You can override the build directory by setting the BUILD_DIR env variable to
another location.
-Also feel free to take a look at our [.travis.yml](.travis.yml) to see how that build env is set up.
+After building the thirdparty libraries, for future development iteration you
+can set the dependency environment variables (detailed below) by running
+
+`source $BUILD_DIR/thirdparty/set_thirdparty_env.sh`
+
+Note, the environment variables are set automatically the first time you run
+`setup_build_env.sh`.
+The unit tests depend on `googletest` which cannot be installed with Homebrew
+or normal package managers. If you wish to use system dependencies, we
+recommend that you build googletest in-tree by running:
+
+```
+./thirdparty/download_thirdparty.sh
+./thirdparty/build_thirdparty.sh gtest
+source thirdparty/versions.sh
+export GTEST_HOME=`pwd`/thirdparty/$GTEST_BASEDIR
+```
## Build
+
- `cmake .`
+
- You can customize dependent library locations through various environment variables:
- THRIFT_HOME customizes the thrift installed location.
- SNAPPY_HOME customizes the snappy installed location.
- LZ4_HOME customizes the lz4 installed location.
+
- `make`
The binaries will be built to ./debug which contains the libraries to link against as
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/before_script_travis.sh
----------------------------------------------------------------------
diff --git a/ci/before_script_travis.sh b/ci/before_script_travis.sh
index b1f31c5..c467c67 100755
--- a/ci/before_script_travis.sh
+++ b/ci/before_script_travis.sh
@@ -3,25 +3,15 @@
# Build an isolated thirdparty
cp -r $TRAVIS_BUILD_DIR/thirdparty .
./thirdparty/download_thirdparty.sh
-source thirdparty/versions.sh
if [ $TRAVIS_OS_NAME == "osx" ]; then
- brew update
- brew install thrift lz4 snappy
-
- # Only build gtest
- ./thirdparty/build_thirdparty.sh gtest
-fi
-
-if [ $TRAVIS_OS_NAME == "linux" ]; then
- ./thirdparty/build_thirdparty.sh
- export THRIFT_HOME=$HOME/build_dir/thirdparty/installed
- export SNAPPY_HOME=$HOME/build_dir/thirdparty/installed
- export LZ4_HOME=$HOME/build_dir/thirdparty/installed
-
+ brew update -q
+ brew install thrift
+else
# Use a C++11 compiler on Linux
export CC="gcc-4.9"
export CXX="g++-4.9"
fi
-export GTEST_HOME=$HOME/build_dir/thirdparty/$GTEST_BASEDIR
+export PARQUET_TEST_DATA=$TRAVIS_BUILD_DIR/data
+source $TRAVIS_BUILD_DIR/setup_build_env.sh
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/ci/run_tests.sh
----------------------------------------------------------------------
diff --git a/ci/run_tests.sh b/ci/run_tests.sh
deleted file mode 100755
index aa7b2f6..0000000
--- a/ci/run_tests.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-if [ $TRAVIS_OS_NAME == "linux" ]; then
- valgrind --tool=memcheck --leak-check=yes --error-exitcode=1 ctest;
-else
- ctest;
-fi
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/setup_build_env.sh
----------------------------------------------------------------------
diff --git a/setup_build_env.sh b/setup_build_env.sh
index 6df1f49..60ce7cf 100755
--- a/setup_build_env.sh
+++ b/setup_build_env.sh
@@ -3,27 +3,23 @@
SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
: ${BUILD_DIR:=$SOURCE_DIR/build}
+# Create an isolated thirdparty
mkdir -p $BUILD_DIR
cp -r $SOURCE_DIR/thirdparty $BUILD_DIR
-cd $BUILD_DIR
+pushd $BUILD_DIR
+
./thirdparty/download_thirdparty.sh
./thirdparty/build_thirdparty.sh
-source thirdparty/versions.sh
-
-export SNAPPY_HOME=$BUILD_DIR/thirdparty/installed
-export LZ4_HOME=$BUILD_DIR/thirdparty/installed
-export ZLIB_HOME=$BUILD_DIR/thirdparty/installed
-# build script doesn't support building thrift on OSX
-if [ "$(uname)" != "Darwin" ]; then
- export THRIFT_HOME=$BUILD_DIR/thirdparty/installed
-fi
+source thirdparty/set_thirdparty_env.sh
export PARQUET_TEST_DATA=$SOURCE_DIR/data
-export GTEST_HOME=$BUILD_DIR/thirdparty/$GTEST_BASEDIR
-cmake $SOURCE_DIR
+# Thrift needs to be installed from brew on OS X
+if [ "$(uname)" == "Darwin" ]; then
+ export -n THRIFT_HOME
+fi
-cd $SOURCE_DIR
+popd
echo
echo "Build env initialized in $BUILD_DIR."
http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/17a20455/thirdparty/set_thirdparty_env.sh
----------------------------------------------------------------------
diff --git a/thirdparty/set_thirdparty_env.sh b/thirdparty/set_thirdparty_env.sh
new file mode 100644
index 0000000..50c5731
--- /dev/null
+++ b/thirdparty/set_thirdparty_env.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+SOURCE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
+source $SOURCE_DIR/versions.sh
+
+if [ -z "$THIRDPARTY_DIR" ]; then
+ THIRDPARTY_DIR=$SOURCE_DIR
+fi
+
+export SNAPPY_HOME=$THIRDPARTY_DIR/installed
+export LZ4_HOME=$THIRDPARTY_DIR/installed
+export ZLIB_HOME=$THIRDPARTY_DIR/installed
+# build script doesn't support building thrift on OSX
+if [ "$(uname)" != "Darwin" ]; then
+ export THRIFT_HOME=$THIRDPARTY_DIR/installed
+fi
+
+export GTEST_HOME=$THIRDPARTY_DIR/$GTEST_BASEDIR