You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ap...@apache.org on 2019/05/09 16:45:00 UTC
[arrow] branch master updated: ARROW-5222: [Python] Revise pyarrow
installation instructions for macOS
This is an automated email from the ASF dual-hosted git repository.
apitrou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 0034ef2 ARROW-5222: [Python] Revise pyarrow installation instructions for macOS
0034ef2 is described below
commit 0034ef290ea2683f5b6009684edcdbf92a222915
Author: Neal Richardson <ne...@gmail.com>
AuthorDate: Thu May 9 18:44:49 2019 +0200
ARROW-5222: [Python] Revise pyarrow installation instructions for macOS
This patch adds some docs and some links to previously existing documentation to help clarify the problems outlined on https://issues.apache.org/jira/browse/ARROW-5222. It also resolves the multiple-Brewfile issue by factoring out a common cpp/Brewfile.
There are also some assorted formatting cleanups here too.
Author: Neal Richardson <ne...@gmail.com>
Closes #4231 from nealrichardson/pyarrow-macos-dev-instructions and squashes the following commits:
716f2607f <Neal Richardson> Brewfile tweak
6014fac37 <Neal Richardson> Tweak language about conda on macos
13385853e <Neal Richardson> Update .gitignore
33a041c7f <Neal Richardson> Factor out a cpp/Brewfile and document it (thanks @kou)
9d333e517 <Neal Richardson> Use :ref: to link to other sections
4158d70be <Neal Richardson> Revise pyarrow installation instructions for macOS
---
.gitignore | 2 ++
c_glib/Brewfile | 11 ------
ci/travis_install_osx.sh | 1 +
{python => cpp}/Brewfile | 10 +++++-
docs/source/developers/cpp.rst | 4 ++-
docs/source/developers/documentation.rst | 2 +-
docs/source/developers/python.rst | 58 +++++++++++++++++++-------------
7 files changed, 50 insertions(+), 38 deletions(-)
diff --git a/.gitignore b/.gitignore
index 2622519..b1a414e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -61,3 +61,5 @@ pkgs
arrow.Rcheck/
docker_cache
.gdb_history
+.DS_Store
+*.orig
diff --git a/c_glib/Brewfile b/c_glib/Brewfile
index ffe0509..66eee91 100644
--- a/c_glib/Brewfile
+++ b/c_glib/Brewfile
@@ -16,19 +16,8 @@
# under the License.
brew "autoconf-archive"
-brew "automake"
-brew "boost"
-brew "ccache"
-brew "cmake"
-brew "double-conversion"
-brew "git"
-brew "glog"
brew "gobject-introspection"
brew "gtk-doc"
brew "libtool"
-brew "llvm@7"
brew "lua"
brew "luarocks"
-brew "ninja"
-brew "thrift"
-brew "wget"
diff --git a/ci/travis_install_osx.sh b/ci/travis_install_osx.sh
index 6b6a4b2..932a2d0 100755
--- a/ci/travis_install_osx.sh
+++ b/ci/travis_install_osx.sh
@@ -45,6 +45,7 @@ if [ "$ARROW_CI_RUBY_AFFECTED" = "1" ]; then
run_brew update
run_brew upgrade python
run_brew uninstall postgis
+ run_brew bundle --file=$TRAVIS_BUILD_DIR/cpp/Brewfile --verbose
run_brew bundle --file=$TRAVIS_BUILD_DIR/c_glib/Brewfile --verbose
rm ${brew_log_path}
fi
diff --git a/python/Brewfile b/cpp/Brewfile
similarity index 87%
rename from python/Brewfile
rename to cpp/Brewfile
index f8cdf22..a5ee9d1 100644
--- a/python/Brewfile
+++ b/cpp/Brewfile
@@ -15,6 +15,14 @@
# specific language governing permissions and limitations
# under the License.
-brew "ccache"
+brew "automake"
brew "boost"
+brew "ccache"
+brew "cmake"
+brew "double-conversion"
+brew "git"
+brew "glog"
+brew "llvm@7"
+brew "ninja"
brew "thrift"
+brew "wget"
diff --git a/docs/source/developers/cpp.rst b/docs/source/developers/cpp.rst
index 71ee003..ab71062 100644
--- a/docs/source/developers/cpp.rst
+++ b/docs/source/developers/cpp.rst
@@ -82,7 +82,7 @@ On macOS, you can use `Homebrew <https://brew.sh/>`_.
git clone https://github.com/apache/arrow.git
cd arrow
- brew update && brew bundle --file=c_glib/Brewfile
+ brew update && brew bundle --file=cpp/Brewfile
Building
========
@@ -180,6 +180,8 @@ We are currently using LLVM 7 for library builds and for other developer tools
such as code formatting with ``clang-format``. LLVM can be installed via most
modern package managers (apt, yum, conda, Homebrew, chocolatey).
+.. _cpp-build-dependency-management:
+
Build Dependency Management
===========================
diff --git a/docs/source/developers/documentation.rst b/docs/source/developers/documentation.rst
index 452305b..3a8eb07 100644
--- a/docs/source/developers/documentation.rst
+++ b/docs/source/developers/documentation.rst
@@ -66,7 +66,7 @@ These two steps are mandatory and must be executed in order.
This step requires the the pyarrow library is installed
in your python environment. One way to accomplish
this is to follow the build instructions at :ref:`python-development`
- and then run `python setup.py install` in arrow/python
+ and then run ``python setup.py install`` in arrow/python
(it is best to do this in a dedicated conda/virtual environment).
.. code-block:: shell
diff --git a/docs/source/developers/python.rst b/docs/source/developers/python.rst
index e1ace04..c6e68be 100644
--- a/docs/source/developers/python.rst
+++ b/docs/source/developers/python.rst
@@ -98,7 +98,7 @@ Building on Linux and MacOS
System Requirements
-------------------
-On macOS, any modern XCode (6.4 or higher; the current version is 8.3.1) is
+On macOS, any modern XCode (6.4 or higher; the current version is 10) is
sufficient.
On Linux, for this guide, we require a minimum of gcc 4.8, or clang 3.7 or
@@ -138,10 +138,18 @@ You should now see
Using Conda
~~~~~~~~~~~
+.. note::
+
+ Using conda to build Arrow on macOS is complicated by the
+ fact that the `conda-forge compilers require an older macOS SDK <https://stackoverflow.com/a/55798942>`_.
+ Conda offers some `installation instructions <https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#macos-sdk>`_;
+ the alternative would be to use :ref:`Homebrew <python-homebrew>` and
+ ``pip`` instead.
+
Let's create a conda environment with all the C++ build and Python dependencies
from conda-forge, targeting development for Python 3.7:
-On Linux and OSX:
+On Linux and macOS:
.. code-block:: shell
@@ -152,10 +160,13 @@ On Linux and OSX:
compilers \
python=3.7
-As of January 2019, the `compilers` package is needed on many Linux distributions to use packages from conda-forge.
+As of January 2019, the ``compilers`` package is needed on many Linux
+distributions to use packages from conda-forge.
With this out of the way, you can now activate the conda environment
+.. code-block:: shell
+
conda activate pyarrow-dev
For Windows, see the `Building on Windows`_ section below.
@@ -177,12 +188,17 @@ Using pip
to manage your development. Please follow the conda-based development
instructions instead.
-On macOS, install all dependencies through Homebrew that are required for
+.. _python-homebrew:
+
+On macOS, use Homebrew to install all dependencies required for
building Arrow C++:
.. code-block:: shell
- brew update && brew bundle --file=arrow/python/Brewfile
+ brew update && brew bundle --file=arrow/cpp/Brewfile
+
+See :ref:`here <cpp-build-dependency-management>` for a list of dependencies you
+may need.
On Debian/Ubuntu, you need the following minimal set of dependencies. All other
dependencies will be automatically built by Arrow's third-party toolchain.
@@ -266,7 +282,7 @@ to ``OFF``:
If multiple versions of Python are installed in your environment, you may have
to pass additional parameters to cmake so that it can find the right
executable, headers and libraries. For example, specifying
-`-DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python` (assuming that you're in
+``-DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python`` (assuming that you're in
virtualenv) enables cmake to choose the python executable which you are using.
.. note::
@@ -276,6 +292,16 @@ virtualenv) enables cmake to choose the python executable which you are using.
this reason we recommend passing ``-DCMAKE_INSTALL_LIBDIR=lib`` because the
Python build scripts assume the library directory is ``lib``
+.. note::
+
+ If you have conda installed but are not using it to manage dependencies,
+ and you have trouble building the C++ library, you may need to set
+ ``-DARROW_DEPENDENCY_SOURCE=AUTO`` or some other value (described
+ :ref:`here <cpp-build-dependency-management>`)
+ to explicitly tell CMake not to use conda.
+
+For any other C++ build challenges, see :ref:`cpp-development`.
+
Now, build pyarrow:
.. code-block:: shell
@@ -291,24 +317,8 @@ Now, build pyarrow:
If you did not build one of the optional components, set the corresponding
``PYARROW_WITH_$COMPONENT`` environment variable to 0.
-You should be able to run the unit tests with:
-
-.. code-block:: shell
-
- $ py.test pyarrow
- ================================ test session starts ====================
- platform linux -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
- rootdir: /home/wesm/arrow-clone/python, inifile:
-
- collected 1061 items / 1 skipped
-
- [... test output not shown here ...]
-
- ============================== warnings summary ===============================
-
- [... many warnings not shown here ...]
-
- ====== 1000 passed, 56 skipped, 6 xfailed, 19 warnings in 26.52 seconds =======
+Now you are ready to install test dependencies and run `Unit Testing`_, as
+described above.
To build a self-contained wheel (including the Arrow and Parquet C++
libraries), one can set ``--bundle-arrow-cpp``: