You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/04/23 15:59:42 UTC
[GitHub] [arrow] kszucs opened a new pull request #7021: Wrap docker-compose commands with archery [WIP]
kszucs opened a new pull request #7021:
URL: https://github.com/apache/arrow/pull/7021
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621957531
@github-actions crossbow submit test-debian-10-cpp
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621884756
Revision: 5c0b02dd7947e1e61da701169cc5fafb9135a6e5
Submitted crossbow builds: [ursa-labs/crossbow @ actions-183](https://github.com/ursa-labs/crossbow/branches/all?query=actions-183)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-183-github-test-conda-python-3.7)|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-183-circle-test-debian-10-cpp)|
|test-debian-10-go-1.12|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-183-azure-test-debian-10-go-1.12)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420023826
##########
File path: dev/archery/archery/testing.py
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from contextlib import contextmanager
+import os
+from unittest import mock
+import re
+
+
+class PartialEnv(dict):
+
+ def __eq__(self, other):
+ return self.items() <= other.items()
Review comment:
I use it to check only the relevant subset of environment variables in the mocked `subprocess.run(..., env=env)` call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621347949
https://issues.apache.org/jira/browse/ARROW-8628
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419722389
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+``archery`` requires ``python>=3.5``. It is recommended to install archery in
+``editable`` mode with the ``-e`` flag to automatically update the intallation
+by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+For the available commands and options invoke the installed archery commands
+with the ``--help`` flag:
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+
+Examples
+~~~~~~~~
+
+**List the available images:**
+
+.. code:: bash
+
+ archery docker images
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**Show the docker-compose commands instead of executing them:**
+
+.. code:: bash
+
+ archery docker run --dry-run conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+In case of the example below the command builds the
+``conda-cpp > conda-python > conda-python-pandas`` branch of the image tree
+where the leaf image is ``conda-python-pandas``.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doesn't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+The layer caching mechanism of docker-compose is less reliable than docker's
+depending on the version, ``cache_from`` build entry and the used backend
+(docker-py, docker-cli, docker-cli and buildkit). This can lead to different
+layer hashes - even when executing the same build command repeatedly -
+eventually causing cache misses full image rebuilds.
+
+If the image has been already built but the cache doesn't work properly, it can
+be useful to skip the build phases:
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+**Pass environment variables to the container:**
+
+Most of the build scripts used within the containers can be configured through
+environment variables. Pass them using ``--env`` or ``-e`` CLI options -
+similarly to ``docker run`` and ``docker-compose run`` interface.
Review comment:
```suggestion
similar to the ``docker run`` and ``docker-compose run`` interface.
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+``archery`` requires ``python>=3.5``. It is recommended to install archery in
+``editable`` mode with the ``-e`` flag to automatically update the intallation
+by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+For the available commands and options invoke the installed archery commands
+with the ``--help`` flag:
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+
+Examples
+~~~~~~~~
+
+**List the available images:**
+
+.. code:: bash
+
+ archery docker images
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**Show the docker-compose commands instead of executing them:**
+
+.. code:: bash
+
+ archery docker run --dry-run conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+In case of the example below the command builds the
+``conda-cpp > conda-python > conda-python-pandas`` branch of the image tree
+where the leaf image is ``conda-python-pandas``.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doesn't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+The layer caching mechanism of docker-compose is less reliable than docker's
+depending on the version, ``cache_from`` build entry and the used backend
Review comment:
```suggestion
The layer-caching mechanism of docker-compose can be less reliable than docker's,
depending on the version, the ``cache_from`` build entry, and the backend used
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+``archery`` requires ``python>=3.5``. It is recommended to install archery in
+``editable`` mode with the ``-e`` flag to automatically update the intallation
+by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+For the available commands and options invoke the installed archery commands
+with the ``--help`` flag:
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+
+Examples
+~~~~~~~~
+
+**List the available images:**
+
+.. code:: bash
+
+ archery docker images
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**Show the docker-compose commands instead of executing them:**
+
+.. code:: bash
+
+ archery docker run --dry-run conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+In case of the example below the command builds the
+``conda-cpp > conda-python > conda-python-pandas`` branch of the image tree
+where the leaf image is ``conda-python-pandas``.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doesn't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+The layer caching mechanism of docker-compose is less reliable than docker's
+depending on the version, ``cache_from`` build entry and the used backend
+(docker-py, docker-cli, docker-cli and buildkit). This can lead to different
+layer hashes - even when executing the same build command repeatedly -
+eventually causing cache misses full image rebuilds.
+
+If the image has been already built but the cache doesn't work properly, it can
+be useful to skip the build phases:
Review comment:
This sounds backwards to me but probably just needs a little explanation. If caching isn't working properly, doesn't that mean I *would* need to build, rather than? Because it can't take it from the cache. I would think that `--no-build` means "use the cache".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419995058
##########
File path: .github/workflows/cpp_cron.yml
##########
@@ -36,170 +36,69 @@ env:
jobs:
- debian:
- name: AMD64 Debian ${{ matrix.debian }} C++
+ docker:
+ name: ${{ matrix.title }}
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
strategy:
fail-fast: false
matrix:
- debian: [10]
+ name:
+ - debian-10-cpp
+ - fedora-30-cpp
+ - ubuntu-16.04-cpp
+ - ubuntu-18.04-cpp
+ - ubuntu-18.04-cpp-cmake32
+ include:
+ - name: debian-10-cpp
+ image: debian-cpp
+ title: AMD64 Debian 10 C++
+ debian: 10
+ - name: fedora-30-cpp
+ image: fedora-cpp
+ title: AMD64 Fedora 30 C++
+ fedora: 30
+ - name: ubuntu-16.04-cpp
+ image: ubuntu-cpp
+ title: AMD64 Ubuntu 16.04 C++
+ ubuntu: 16.04
+ - name: ubuntu-18.04-cpp
+ image: ubuntu-cpp
+ title: AMD64 Ubuntu 18.04 C++
+ ubuntu: 18.04
+ - name: ubuntu-18.04-cpp-cmake32
+ image: ubuntu-cpp-cmake32
+ title: AMD64 Ubuntu 18.04 C++ CMake 3.2
+ ubuntu: 18.04
env:
- DEBIAN: ${{ matrix.debian }}
+ # the defaults here should correspond to the values in .env
Review comment:
Why are the defaults required, since each matrix entry lists the required value explicitly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420014427
##########
File path: .github/workflows/archery.yml
##########
@@ -48,13 +50,15 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v1
with:
- python-version: '3.7'
- - name: Install
+ python-version: '3.5'
Review comment:
On older linux distros this is the default, so I was trying to make it more convenient for non-python developers. We can of course increase the python requirement later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-622057148
@github-actions crossbow submit -g test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r418622281
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
Can this link to the main Archery docs? (*Are there Archery docs?*)
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
Review comment:
```suggestion
There are multiple ways to execute the docker based builds. The recommended is
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
Review comment:
Can I run these docker-compose commands myself?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
Review comment:
Would I ever call `archery docker` and not immediately put `run` next? If not, why not just `archery run conda-python`?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
Among the questions I'd have for the archery docs specifically right here:
* What is archery?
* What system requirements does it have? (like, python version?)
* What do I do if the simple instructions don't work?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
Review comment:
What's a leaf image?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
Review comment:
Is there `archery docker --help` or something that tell me all of the flags like this?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
Review comment:
This needs more explanation, particularly to distinguish build-time parameters from environment variables
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
+
+A good example is cpp_build.sh build script which forwards environment
+variables as CMake options - so the same scripts can be invoked in various
+configurations without the necessity of chaning it. For examples see how the
Review comment:
chaning?
##########
File path: docs/source/developers/integration.rst
##########
@@ -62,9 +62,9 @@ simplify this, we provide a Makefile.
.. code-block:: shell
# Build and run manually
- docker-compose build cpp
- docker-compose build python
- docker-compose run python
+ docker-compose build conda-cpp
+ docker-compose build conda-python
+ docker-compose run conda-python
- # Using the makefile with proper image dependency resolution
- make -f Makefile.docker python
+ # Using the archery with proper image dependency resolution
Review comment:
```suggestion
# Using archery with proper image dependency resolution
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
Review comment:
That's what `-e` means?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
Review comment:
For C++, can I pass all cmake flags that way?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
+
+A good example is cpp_build.sh build script which forwards environment
+variables as CMake options - so the same scripts can be invoked in various
+configurations without the necessity of chaning it. For examples see how the
+environment variables are passed in the docker-compose.yml's C++ images.
+
+Adding New Images
+~~~~~~~~~~~~~~~~~
+
+See more in the docker-compose.yml
Review comment:
I don't understand
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
Review comment:
```suggestion
possible to be responsible for only a subset of tasks.
```
What subset? Why minimal?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
Review comment:
Ok, these are the build parameters. Can you link to this section where you mention that above?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
Review comment:
```suggestion
The parametrization is done through environment variables with useful defaults to
```
What parametrization?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
Review comment:
```suggestion
keep the build configurations declarative. Note that these parameters are
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
Review comment:
... by doing what? (Please be explicit so I don't have to diff the examples to figure out what to do)
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
Review comment:
```suggestion
behavior of the builds scripts within the containers.
```
Also this sentence is not clear to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-622062939
Revision: 3cd96ea48a2116322b2fec06207fb1d624e0f969
Submitted crossbow builds: [ursa-labs/crossbow @ actions-189](https://github.com/ursa-labs/crossbow/branches/all?query=actions-189)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-cpp-valgrind)|
|test-conda-python-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.6)|
|test-conda-python-3.6-pandas-0.23|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.6-pandas-0.23)|
|test-conda-python-3.7|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7)|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-pandas-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-pandas-master)|
|test-conda-python-3.7-spark-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-spark-master)|
|test-conda-python-3.7-turbodbc-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-turbodbc-latest)|
|test-conda-python-3.7-turbodbc-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.7-turbodbc-master)|
|test-conda-python-3.8|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.8)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.8-dask-master)|
|test-conda-python-3.8-jpype|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.8-jpype)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-python-3.8-pandas-latest)|
|test-conda-r-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-conda-r-3.6)|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-debian-10-cpp)|
|test-debian-10-go-1.12|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-debian-10-go-1.12)|
|test-debian-10-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-debian-10-python-3)|
|test-debian-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-debian-c-glib)|
|test-debian-ruby|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-debian-ruby)|
|test-fedora-30-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-fedora-30-cpp)|
|test-fedora-30-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-fedora-30-python-3)|
|test-r-linux-as-cran|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-189-github-test-r-linux-as-cran)|
|test-r-rhub-ubuntu-gcc-release|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rhub-ubuntu-gcc-release)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rstudio-r-base-3.6-centos6)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-ubuntu-16.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-16.04-cpp)|
|test-ubuntu-18.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-cmake32|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-18.04-cpp-cmake32)|
|test-ubuntu-18.04-cpp-release|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-docs|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-ubuntu-18.04-docs)|
|test-ubuntu-18.04-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-ubuntu-18.04-python-3)|
|test-ubuntu-18.04-r-3.6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-ubuntu-18.04-r-3.6)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-189-circle-test-ubuntu-c-glib)|
|test-ubuntu-ruby|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-189-azure-test-ubuntu-ruby)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419339195
##########
File path: .github/workflows/java.yml
##########
@@ -38,6 +38,8 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
Review comment:
Missed that. I'm not sure what would the best way to reduce the maintenance cost of this yml jungle. One possibility to use a crossbow like yml generator, but it has numerous disadvantages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420020854
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
+@click.option('--dry-run/--execute', default=False,
+ help='Display the docker-compose commands instead of executing '
+ 'them')
+@click.pass_obj
+def docker_compose_run(obj, image, command, env, build, cache, cache_leaf,
+ dry_run):
+ """Execute docker-compose builds.
+
+ To see the available builds run `archery docker list`.
+
+ Examples:
+
+ # execute a single build
+ archery docker run conda-python
+
+ # execute the builds but disable the image pulling
+ archery docker run --no-cache conda-python
+
+ # pass a docker-compose parameter, like the python version
+ PYTHON=3.8 archery docker run conda-python
+
+ # disable the cache only for the leaf image
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+ # entirely skip building the image
+ archery docker run --no-build conda-python
+
+ # pass runtime parameters via docker environment variables
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+ # starting an interactive bash session for debugging
+ archery docker run ubuntu-cpp bash
+ """
+ from .docker import UndefinedImage
+
+ compose = obj['compose']
+
+ if dry_run:
+ from types import MethodType
+
+ def _print_command(self, *args, **kwargs):
+ params = ['{}={}'.format(k, v) for k, v in self.params.items()]
+ command = ' '.join(params + ['docker-compose'] + list(args))
+ click.echo(command)
+
+ compose._execute = MethodType(_print_command, compose)
+
+ try:
+ if build:
+ compose.build(image, cache=cache, cache_leaf=cache_leaf)
+ compose.run(image, command=command)
+ except UndefinedImage as e:
+ raise click.ClickException(
+ "There is no service/image defined in docker-compose.yml with "
+ "name: {}".format(str(e))
+ )
+ except RuntimeError as e:
+ raise click.ClickException(str(e))
+
+
+@docker_compose.command('push')
+@click.argument('image')
+@click.option('--user', '-u', required=True, envvar='ARCHERY_DOCKER_USER',
+ help='Docker login user')
+@click.option('--password', '-p', required=True,
+ envvar='ARCHERY_DOCKER_PASSWORD',
+ help='Docker login user')
+@click.pass_obj
+def docker_compose_push(obj, image, user, password):
+ """Push the generated docker-compose image."""
+ compose = obj['compose']
+ compose.push(image, user=user, password=password)
+
+
+@docker_compose.command('images')
+@click.pass_obj
+def docker_compose_images(obj):
+ """Push the generated docker-compose image."""
Review comment:
Updating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419353296
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
Review comment:
I kept that open for features like listing available build(er)s. I can change this to something like `archery build conda-python` if you find it more convenient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-622578059
Thanks for the reviews, addressing them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621272162
Revision: 356984236aa4f34059eb770831a49197e454a12c
Submitted crossbow builds: [ursa-labs/crossbow @ actions-174](https://github.com/ursa-labs/crossbow/branches/all?query=actions-174)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-174-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-174-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-174-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420000011
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
+@click.option('--dry-run/--execute', default=False,
+ help='Display the docker-compose commands instead of executing '
+ 'them')
+@click.pass_obj
+def docker_compose_run(obj, image, command, env, build, cache, cache_leaf,
+ dry_run):
+ """Execute docker-compose builds.
+
+ To see the available builds run `archery docker list`.
+
+ Examples:
+
+ # execute a single build
+ archery docker run conda-python
+
+ # execute the builds but disable the image pulling
+ archery docker run --no-cache conda-python
+
+ # pass a docker-compose parameter, like the python version
+ PYTHON=3.8 archery docker run conda-python
+
+ # disable the cache only for the leaf image
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+ # entirely skip building the image
+ archery docker run --no-build conda-python
+
+ # pass runtime parameters via docker environment variables
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+ # starting an interactive bash session for debugging
+ archery docker run ubuntu-cpp bash
+ """
+ from .docker import UndefinedImage
+
+ compose = obj['compose']
+
+ if dry_run:
+ from types import MethodType
+
+ def _print_command(self, *args, **kwargs):
+ params = ['{}={}'.format(k, v) for k, v in self.params.items()]
+ command = ' '.join(params + ['docker-compose'] + list(args))
+ click.echo(command)
+
+ compose._execute = MethodType(_print_command, compose)
+
+ try:
+ if build:
+ compose.build(image, cache=cache, cache_leaf=cache_leaf)
+ compose.run(image, command=command)
+ except UndefinedImage as e:
+ raise click.ClickException(
+ "There is no service/image defined in docker-compose.yml with "
+ "name: {}".format(str(e))
+ )
+ except RuntimeError as e:
+ raise click.ClickException(str(e))
+
+
+@docker_compose.command('push')
+@click.argument('image')
+@click.option('--user', '-u', required=True, envvar='ARCHERY_DOCKER_USER',
+ help='Docker login user')
+@click.option('--password', '-p', required=True,
+ envvar='ARCHERY_DOCKER_PASSWORD',
+ help='Docker login user')
+@click.pass_obj
+def docker_compose_push(obj, image, user, password):
+ """Push the generated docker-compose image."""
+ compose = obj['compose']
+ compose.push(image, user=user, password=password)
+
+
+@docker_compose.command('images')
+@click.pass_obj
+def docker_compose_images(obj):
+ """Push the generated docker-compose image."""
Review comment:
This docstring seems off.
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
+@click.option('--dry-run/--execute', default=False,
+ help='Display the docker-compose commands instead of executing '
+ 'them')
+@click.pass_obj
+def docker_compose_run(obj, image, command, env, build, cache, cache_leaf,
+ dry_run):
+ """Execute docker-compose builds.
+
+ To see the available builds run `archery docker list`.
+
+ Examples:
+
+ # execute a single build
+ archery docker run conda-python
+
+ # execute the builds but disable the image pulling
+ archery docker run --no-cache conda-python
+
+ # pass a docker-compose parameter, like the python version
+ PYTHON=3.8 archery docker run conda-python
+
+ # disable the cache only for the leaf image
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+ # entirely skip building the image
+ archery docker run --no-build conda-python
+
+ # pass runtime parameters via docker environment variables
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+ # starting an interactive bash session for debugging
+ archery docker run ubuntu-cpp bash
+ """
+ from .docker import UndefinedImage
+
+ compose = obj['compose']
+
+ if dry_run:
+ from types import MethodType
+
+ def _print_command(self, *args, **kwargs):
+ params = ['{}={}'.format(k, v) for k, v in self.params.items()]
+ command = ' '.join(params + ['docker-compose'] + list(args))
+ click.echo(command)
+
+ compose._execute = MethodType(_print_command, compose)
+
+ try:
+ if build:
+ compose.build(image, cache=cache, cache_leaf=cache_leaf)
+ compose.run(image, command=command)
+ except UndefinedImage as e:
+ raise click.ClickException(
+ "There is no service/image defined in docker-compose.yml with "
+ "name: {}".format(str(e))
+ )
+ except RuntimeError as e:
+ raise click.ClickException(str(e))
+
+
+@docker_compose.command('push')
+@click.argument('image')
+@click.option('--user', '-u', required=True, envvar='ARCHERY_DOCKER_USER',
+ help='Docker login user')
+@click.option('--password', '-p', required=True,
+ envvar='ARCHERY_DOCKER_PASSWORD',
+ help='Docker login user')
Review comment:
This description seems off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420001361
##########
File path: dev/archery/archery/testing.py
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from contextlib import contextmanager
+import os
+from unittest import mock
+import re
+
+
+class PartialEnv(dict):
+
+ def __eq__(self, other):
+ return self.items() <= other.items()
Review comment:
Why do you need such a weird `__eq__` method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621946996
@github-actions crossbow submit test-debian-10-cpp
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419541129
##########
File path: docs/source/example1.dat
##########
@@ -0,0 +1 @@
+some data
Review comment:
Nope.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419709631
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419528025
##########
File path: .github/workflows/archery.yml
##########
@@ -51,10 +53,12 @@ jobs:
python-version: '3.7'
- name: Install
working-directory: dev/archery
- run: pip install pytest responses ruamel.yaml toolz jinja2 -e .
+ run: pip install pytest python-dotenv responses ruamel.yaml toolz jinja2 -e .
Review comment:
There is `-e .` at the and of the command. The additional dependencies listed here are required for testing and crossbow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419372644
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
Review comment:
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621287754
Revision: aff6c71b2c2f838489cf18a65d24dd4e68067230
Submitted crossbow builds: [ursa-labs/crossbow @ actions-175](https://github.com/ursa-labs/crossbow/branches/all?query=actions-175)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-175-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-175-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-175-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621883711
@github-actions crossbow submit test-debian-10-cpp test-debian-10-go-1.12 test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on issue #7021: Wrap docker-compose commands with archery [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-618494095
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621349417
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419413778
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
Review comment:
I'm not sure what do you mean, updated to be a bit more verbose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419538949
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
Review comment:
The command's interface tries to correspond to `docker-compose **run**` with the addition to build the required images.
```bash
docker-compose run <image> <cmd>
archery docker run <image> <cmd>
docker-compose run --env VAR_A=A -e VAR_B=B <image> <cmd>
archery docker run --env VAR_A=A -e VAR_B=B <image> <cmd>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419401740
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
Review comment:
Shorthand for `--env`. Using the full parameter name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621251535
Revision: 33c81628d4f4f9da45911798858b85f05a3646d3
Submitted crossbow builds: [ursa-labs/crossbow @ actions-173](https://github.com/ursa-labs/crossbow/branches/all?query=actions-173)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-173-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-173-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-173-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419998775
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
Review comment:
I don't understand why `--cache-leaf` means "force build". IMHO, either the option name or its description is off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419645732
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
Review comment:
Tried to clarify it.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419715294
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
Review comment:
Just added the `archery docker images` command. We can refine the CLI API later and even keep the old commands if we choose to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621696833
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420016943
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+from .compat import _ensure_path
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None,
+ params=None):
+ self.config_path = _ensure_path(config_path)
+ if dotenv_path:
+ self.dotenv_path = _ensure_path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.bin = default_bin(compose_bin, 'docker-compose')
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(str(self.dotenv_path))
+ if params is None:
+ self.params = {}
+ else:
+ self.params = {k: v for k, v in params.items() if k in self.dotenv}
+
+ # forward the process' environment variables
Review comment:
Try `UBUNTU=20.04 archery docker run ubuntu-cpp`. The `conda-cpp` image always depend on ubuntu 18.04 so the `UBUNTU` env variable is ignored.
Raising a warning about it would need to parse out the arguments used in that particular docker-compose service.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] removed a comment on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-618494095
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419416278
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
Review comment:
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419646876
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
Review comment:
Added explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-624178442
Hopefully we won't have any regressions, if we do I can handle those in follow-up PRs.
Thanks for the reviews!
Merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419413899
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419416691
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
Review comment:
The ones defined in cpp_build.sh. Added a reference for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621759490
@github-actions crossbow submit test-debian-10-cpp test-debian-10-go-1.12 test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419646703
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
Review comment:
Removed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621182535
Revision: 6675b729a69824eb256438c3864440cae20172ca
Submitted crossbow builds: [ursa-labs/crossbow @ actions-171](https://github.com/ursa-labs/crossbow/branches/all?query=actions-171)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-171-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-171-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-171-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420028511
##########
File path: dev/archery/archery/cli.py
##########
@@ -30,14 +31,16 @@
from .utils.codec import JsonEncoder
from .utils.lint import linter, python_numpydoc, LintValidationException
from .utils.logger import logger, ctx as log_ctx
-from .utils.source import ArrowSources
+from .utils.source import ArrowSources, InvalidArrowSource
from .utils.tmpdir import tmpdir
-from .bot import CommentBot, actions
# Set default logging to INFO in command line.
logging.basicConfig(level=logging.INFO)
+DEFAULT_ARROW_PATH = Path(__file__).parents[3].absolute()
Review comment:
Nope, removing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419645430
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420020562
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
Review comment:
I'm open to any other naming. `docker build` and `docker-compose build` use the `--no-cache` option to force rebuild the image, so I tried to stick with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420021841
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
+@click.option('--dry-run/--execute', default=False,
+ help='Display the docker-compose commands instead of executing '
+ 'them')
+@click.pass_obj
+def docker_compose_run(obj, image, command, env, build, cache, cache_leaf,
+ dry_run):
+ """Execute docker-compose builds.
+
+ To see the available builds run `archery docker list`.
+
+ Examples:
+
+ # execute a single build
+ archery docker run conda-python
+
+ # execute the builds but disable the image pulling
+ archery docker run --no-cache conda-python
+
+ # pass a docker-compose parameter, like the python version
+ PYTHON=3.8 archery docker run conda-python
+
+ # disable the cache only for the leaf image
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+ # entirely skip building the image
+ archery docker run --no-build conda-python
+
+ # pass runtime parameters via docker environment variables
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+ # starting an interactive bash session for debugging
+ archery docker run ubuntu-cpp bash
+ """
+ from .docker import UndefinedImage
+
+ compose = obj['compose']
+
+ if dry_run:
+ from types import MethodType
+
+ def _print_command(self, *args, **kwargs):
+ params = ['{}={}'.format(k, v) for k, v in self.params.items()]
+ command = ' '.join(params + ['docker-compose'] + list(args))
+ click.echo(command)
+
+ compose._execute = MethodType(_print_command, compose)
+
+ try:
+ if build:
+ compose.build(image, cache=cache, cache_leaf=cache_leaf)
+ compose.run(image, command=command)
+ except UndefinedImage as e:
+ raise click.ClickException(
+ "There is no service/image defined in docker-compose.yml with "
+ "name: {}".format(str(e))
+ )
+ except RuntimeError as e:
+ raise click.ClickException(str(e))
+
+
+@docker_compose.command('push')
+@click.argument('image')
+@click.option('--user', '-u', required=True, envvar='ARCHERY_DOCKER_USER',
+ help='Docker login user')
+@click.option('--password', '-p', required=True,
+ envvar='ARCHERY_DOCKER_PASSWORD',
+ help='Docker login user')
Review comment:
Updating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419400510
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
Review comment:
Described below, added a reference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420002457
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
Review comment:
Can't you copy/paste the copyright block from another .rst file instead? Using raw HTML for this is weird.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-623684032
> Some UX improvement I'd like to see:
>
> * I often fail to run `docker-compose` from the root of the sources leading to cryptic errors because it can't find the `.env`. Could you automatically fix this with a setcwd in archery? This is a nice UX feature for user who don't know this detail.
`archery docker run` now works from any subdirectory of arrow, even when it is installed in non-editable mode. The script handles the reading of dotenv defaults for better integration, so we always override those values.
> * It would be nice if upon failure, we'd get the list of environment variables changes affecting the docker build, e.g. PANDAS=master... printed somewhere such that we can easily reproduce.
Added.
> * Add a `archery docker list` command so we don't need to look at docker-compose.yml to get the list of available targets. Bonus point if you add a short description
Added and the bonus is deferred to https://issues.apache.org/jira/browse/ARROW-8697
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-622320430
@github-actions crossbow submit -g test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621200635
Revision: d6f60e9a47716224cdd1b2148b9efc754b0d3ba9
Submitted crossbow builds: [ursa-labs/crossbow @ actions-172](https://github.com/ursa-labs/crossbow/branches/all?query=actions-172)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-172-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-172-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-172-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419646070
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r418599653
##########
File path: .github/workflows/java.yml
##########
@@ -38,6 +38,8 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
Review comment:
No refactor for the debian java entry?
##########
File path: .github/workflows/python_cron.yml
##########
@@ -31,180 +31,71 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
jobs:
- debian:
- name: AMD64 Debian ${{ matrix.debian }} Python 3
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
- strategy:
- fail-fast: false
- matrix:
- debian: [10]
- env:
- DEBIAN: ${{ matrix.debian }}
- steps:
- - name: Checkout Arrow
- uses: actions/checkout@v2
- with:
- fetch-depth: 0
- - name: Fetch Submodules and Tags
- shell: bash
- run: ci/scripts/util_checkout.sh
- - name: Free Up Disk Space
- shell: bash
- run: ci/scripts/util_cleanup.sh
- - name: Docker Pull
- shell: bash
- run: |
- docker-compose pull --ignore-pull-failures debian-cpp
- docker-compose pull --ignore-pull-failures debian-python
- - name: Docker Build
- shell: bash
- run: |
- docker-compose build debian-cpp
- docker-compose build debian-python
- - name: Docker Run
- shell: bash
- run: docker-compose run debian-python
- - name: Docker Push
- if: success() && github.repository == 'apache/arrow'
- continue-on-error: true
- shell: bash
- run: |
- docker login -u ${{ secrets.DOCKERHUB_USER }} \
- -p ${{ secrets.DOCKERHUB_TOKEN }}
- docker-compose push debian-python
-
- ubuntu:
- name: AMD64 Ubuntu ${{ matrix.ubuntu }} Python 3
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
- strategy:
- fail-fast: false
- matrix:
- ubuntu: [18.04]
- env:
- UBUNTU: ${{ matrix.ubuntu }}
- steps:
- - name: Checkout Arrow
- uses: actions/checkout@v2
- with:
- fetch-depth: 0
- - name: Fetch Submodules and Tags
- shell: bash
- run: ci/scripts/util_checkout.sh
- - name: Free Up Disk Space
- shell: bash
- run: ci/scripts/util_cleanup.sh
- - name: Docker Pull
- shell: bash
- run: |
- docker-compose pull --ignore-pull-failures ubuntu-cpp
- docker-compose pull --ignore-pull-failures ubuntu-python
- - name: Docker Build
- shell: bash
- run: |
- docker-compose build ubuntu-cpp
- docker-compose build ubuntu-python
- - name: Docker Run
- shell: bash
- run: docker-compose run ubuntu-python
- - name: Docker Push
- if: success() && github.repository == 'apache/arrow'
- continue-on-error: true
- shell: bash
- run: |
- docker login -u ${{ secrets.DOCKERHUB_USER }} \
- -p ${{ secrets.DOCKERHUB_TOKEN }}
- docker-compose push ubuntu-python
-
- fedora:
- name: AMD64 Fedora ${{ matrix.fedora }} Python 3
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
- strategy:
- fail-fast: false
- matrix:
- fedora: [30]
- env:
- FEDORA: ${{ matrix.fedora }}
- steps:
- - name: Checkout Arrow
- uses: actions/checkout@v2
- with:
- fetch-depth: 0
- - name: Fetch Submodules and Tags
- shell: bash
- run: ci/scripts/util_checkout.sh
- - name: Free Up Disk Space
- shell: bash
- run: ci/scripts/util_cleanup.sh
- - name: Docker Pull
- shell: bash
- run: |
- docker-compose pull --ignore-pull-failures fedora-cpp
- docker-compose pull --ignore-pull-failures fedora-python
- - name: Docker Build
- shell: bash
- run: |
- docker-compose build fedora-cpp
- docker-compose build fedora-python
- - name: Docker Run
- shell: bash
- run: docker-compose run fedora-python
- - name: Docker Push
- if: success() && github.repository == 'apache/arrow'
- continue-on-error: true
- shell: bash
- run: |
- docker login -u ${{ secrets.DOCKERHUB_USER }} \
- -p ${{ secrets.DOCKERHUB_TOKEN }}
- docker-compose push fedora-python
-
- downstream:
- name: AMD64 Conda Python 3.7 ${{ matrix.title }}
+ docker:
+ name: ${{ matrix.title }}
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
strategy:
fail-fast: false
matrix:
name:
- - dask-latest
- - hdfs-2.9.2
- - turbodbc-latest
- - kartothek-latest
- - pandas-master
- - pandas-0.24
+ - debian-10-python-3
Review comment:
That's the type of diff I like to see.
##########
File path: .github/workflows/r.yml
##########
@@ -62,35 +65,24 @@ jobs:
with:
fetch-depth: 0
- name: Fetch Submodules and Tags
- shell: bash
run: ci/scripts/util_checkout.sh
- name: Free Up Disk Space
- shell: bash
run: ci/scripts/util_cleanup.sh
- - name: Docker Pull
- shell: bash
- run: |
- docker-compose pull --ignore-pull-failures ubuntu-cpp
- docker-compose pull --ignore-pull-failures ubuntu-r
- - name: Docker Build
- shell: bash
+ - name: Setup Python
+ uses: actions/setup-python@v1
+ with:
+ python-version: 3.8
+ - name: Setup Archery
+ run: pip install -e dev/archery[docker]
+ - name: Execute Docker Build
run: |
- docker-compose build ubuntu-cpp
- docker-compose build ubuntu-r
- - name: Docker Run
- shell: bash
- run: docker-compose run ubuntu-r
- - name: Dump install logs on failure
Review comment:
Dumping failures is useful for debugging.
##########
File path: docs/source/developers/documentation.rst
##########
@@ -92,10 +92,10 @@ you made.
Building with Docker
--------------------
-You can use Docker to build the documentation:
+You can use Archery to build the documentation within a docker container:
Review comment:
Also comment/link to installation.
##########
File path: docs/source/developers/integration.rst
##########
@@ -62,9 +62,9 @@ simplify this, we provide a Makefile.
.. code-block:: shell
# Build and run manually
- docker-compose build cpp
- docker-compose build python
- docker-compose run python
+ docker-compose build conda-cpp
+ docker-compose build conda-python
+ docker-compose run conda-python
- # Using the makefile with proper image dependency resolution
- make -f Makefile.docker python
+ # Using the archery with proper image dependency resolution
Review comment:
Add a comment/link on how to install.
##########
File path: dev/tasks/tasks.yml
##########
@@ -1914,286 +1744,165 @@ tasks:
not_cran: "TRUE"
test-ubuntu-18.04-r-3.6:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: azure
+ template: docker-tests/azure.linux.yml
params:
env:
UBUNTU: 18.04
R: 3.6
- build:
- - ubuntu-cpp
- - ubuntu-r
- run:
- - ubuntu-r
+ run: ubuntu-r
test-conda-r-3.6:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
R: 3.6
- build:
- - conda-cpp
- - conda-r
- run:
- - conda-r
+ run: conda-r
test-ubuntu-18.04-r-sanitizer:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: azure
+ template: docker-tests/azure.linux.yml
params:
- build:
- - ubuntu-r-sanitizer
- run:
- - ubuntu-r-sanitizer
+ run: ubuntu-r-sanitizer
test-debian-10-go-1.12:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: azure
+ template: docker-tests/azure.linux.yml
params:
env:
GO: 1.12
- build:
- - debian-go
- run:
- - debian-go
+ run: debian-go
test-ubuntu-18.04-docs:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: azure
+ template: docker-tests/azure.linux.yml
params:
env:
UBUNTU: 18.04
- build:
- - ubuntu-cpp
- - ubuntu-python
- - ubuntu-docs
- run:
- - ubuntu-docs
+ run: ubuntu-docs
############################## Integration tests ############################
test-conda-python-3.7-pandas-latest:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.7
PANDAS: latest
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-pandas
- run:
- - conda-python-pandas
+ run: conda-python-pandas
test-conda-python-3.8-pandas-latest:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.8
PANDAS: latest
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-pandas
- run:
- - conda-python-pandas
+ run: conda-python-pandas
test-conda-python-3.7-pandas-master:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.7
PANDAS: master
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-pandas
- run:
- - conda-python-pandas
+ run: --no-cache-leaf conda-python-pandas
test-conda-python-3.6-pandas-0.23:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.6
PANDAS: 0.23
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-pandas
- run:
- - conda-python-pandas
+ run: conda-python-pandas
test-conda-python-3.7-dask-latest:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.7
DASK: latest
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-dask
- run:
- - conda-python-dask
+ run: conda-python-dask
test-conda-python-3.8-dask-master:
- ci: circle
- platform: linux
- template: docker-tests/circle.linux.yml
+ ci: github
+ template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.8
DASK: latest
- build:
- - conda-cpp
- - conda-python
- nocache:
- - conda-python-dask
- run:
- - conda-python-dask
+ run: --no-cache-leaf conda-python-dask
Review comment:
Why `--no-cache-leaf` for this one, but not for `3.7`.
##########
File path: docker-compose.yml
##########
@@ -79,6 +79,52 @@ x-ccache: &ccache
CCACHE_MAXSIZE: 500M
CCACHE_DIR: /build/ccache
+x-hierarchy:
Review comment:
Since this is highly custom, add a paragraph documenting this, and some basic introductions on how to add a new target.
##########
File path: dev/archery/archery/tests/test_docker.py
##########
@@ -0,0 +1,168 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import mock
+from pathlib import Path
+
+import pytest
+
+from archery.docker import DockerCompose
+
+
+example_missing_service = """
+x-hierarchy:
+ - foo:
+ - sub-foo:
+ - sub-sub-foo
+ - another-sub-sub-foo
+ - bar:
+ - sub-bar
+ - baz
+
+services:
+ foo:
+ sub-sub-foo:
+ another-sub-sub-foo:
+ bar:
+ sub-bar:
+ baz:
+"""
+
+example_missing_node = """
+x-hierarchy:
+ - foo:
+ - sub-foo:
+ - sub-sub-foo
+ - another-sub-sub-foo
+ - bar
+ - baz
+
+services:
+ foo:
+ sub-foo:
+ sub-sub-foo:
+ another-sub-sub-foo:
+ bar:
+ sub-bar:
+ baz:
+"""
+
+example_ok = """
+x-hierarchy:
+ - foo:
+ - sub-foo:
+ - sub-sub-foo
+ - another-sub-sub-foo
+ - bar:
+ - sub-bar
+ - baz
+
+services:
+ foo:
+ sub-foo:
+ sub-sub-foo:
+ another-sub-sub-foo:
+ bar:
+ sub-bar:
+ baz:
+"""
+
+
+def create_config(tmpdir, yml_content):
+ config_path = tmpdir / 'docker-compose.yml'
+ with config_path.open('w') as fp:
+ fp.write(yml_content)
+ return DockerCompose(config_path)
+
+
+def test_config_validation(tmpdir):
+ compose = create_config(tmpdir, example_missing_service)
+ msg = "`sub-foo` is defined in `x-hierarchy` bot not in `services`"
+ with pytest.raises(ValueError, match=msg):
+ compose.validate()
+
+ compose = create_config(tmpdir, example_missing_node)
+ msg = "`sub-bar` is defined in `services` but not in `x-hierarchy`"
+ with pytest.raises(ValueError, match=msg):
+ compose.validate()
+
+ compose = create_config(tmpdir, example_ok)
+ compose.validate()
+
+
+def test_executed_docker_commands(tmpdir):
+ arrow_config = Path(__file__).parents[4] / 'docker-compose.yml'
Review comment:
Also use ArrowSource here.
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from pathlib import Path
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None):
+ self.config_path = Path(config_path)
+ if dotenv_path:
+ self.dotenv_path = Path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(self.dotenv_path)
+ self.bin = default_bin(compose_bin, "docker-compose")
+
+ def validate(self):
+ services = self.config['services'].keys()
+ nodes = self.nodes.keys()
+ errors = []
+
+ for name in nodes - services:
+ errors.append(
+ 'Service `{}` is defined in `x-hierarchy` bot not in '
+ '`services`'.format(name)
+ )
+ for name in services - nodes:
+ errors.append(
+ 'Service `{}` is defined in `services` but not in '
+ '`x-hierarchy`'.format(name)
+ )
+
+ # trigger docker-compose's own validation
+ result = self._execute('ps', check=False, stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+
+ if result.returncode != 0:
+ # strip the intro line of docker-compose errors
+ errors += result.stderr.decode().splitlines()[1:]
+
+ if errors:
+ msg = '\n'.join([' - {}'.format(msg) for msg in errors])
+ raise ValueError(
+ 'Found errors with docker-compose:\n{}'.format(msg)
+ )
+
+ def _compose_env(self):
+ return dict(os.environ, **self.dotenv)
Review comment:
The order effectively means that all build arguments are _always_ taken from the `.env` file and never from the process' environment because they're always set in the .env. That requires a unit test because this is a serious silent bug.
Confirmed:
```
$ UBUNTU=16.04 archery docker run ubuntu-cpp
Pulling ubuntu-cpp ... done
Building ubuntu-cpp
Step 1/9 : ARG arch=amd64
Step 2/9 : FROM ${arch}/ubuntu:18.04
---> 775349758637
Step 3/9 : SHELL ["/bin/bash", "-o", "pipefail", "-c"]
---> Running in 164f1875b0b7
Removing intermediate container 164f1875b0b7
---> 49d5bb057b74
Step 4/9 : ENV DEBIAN_FRONTEND=noninteractive
---> Running in e53c37c30d7c
Removing intermediate container e53c37c30d7c
---> a0409634a89f
Step 5/9 : ARG clang_tools
---> Running in b6eae4abe7fd
Removing intermediate container b6eae4abe7fd
---> 36b3f9d7c94b
Step 6/9 : ARG llvm
---> Running in abbaf989a65c
```
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from pathlib import Path
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None):
+ self.config_path = Path(config_path)
+ if dotenv_path:
+ self.dotenv_path = Path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(self.dotenv_path)
+ self.bin = default_bin(compose_bin, "docker-compose")
+
+ def validate(self):
+ services = self.config['services'].keys()
+ nodes = self.nodes.keys()
+ errors = []
+
+ for name in nodes - services:
+ errors.append(
+ 'Service `{}` is defined in `x-hierarchy` bot not in '
+ '`services`'.format(name)
+ )
+ for name in services - nodes:
+ errors.append(
+ 'Service `{}` is defined in `services` but not in '
+ '`x-hierarchy`'.format(name)
+ )
+
+ # trigger docker-compose's own validation
+ result = self._execute('ps', check=False, stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+
+ if result.returncode != 0:
+ # strip the intro line of docker-compose errors
+ errors += result.stderr.decode().splitlines()[1:]
+
+ if errors:
+ msg = '\n'.join([' - {}'.format(msg) for msg in errors])
+ raise ValueError(
+ 'Found errors with docker-compose:\n{}'.format(msg)
+ )
+
+ def _compose_env(self):
+ return dict(os.environ, **self.dotenv)
+
+ def _validate_image(self, name):
+ if name not in self.nodes:
+ raise UndefinedImage(name)
+
+ def _execute(self, *args, **kwargs):
+ # set default arguments for docker-compose
+ return super().run('--file', str(self.config_path), *args, **kwargs)
+
+ def build(self, image, cache=True, cache_leaf=True):
+ self._validate_image(image)
+ env = self._compose_env()
+
+ # build all parents
+ for parent in self.nodes[image]:
Review comment:
Small nit, when the tree has depth > 1, we say ancestors.
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from pathlib import Path
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None):
+ self.config_path = Path(config_path)
+ if dotenv_path:
+ self.dotenv_path = Path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(self.dotenv_path)
+ self.bin = default_bin(compose_bin, "docker-compose")
+
+ def validate(self):
+ services = self.config['services'].keys()
+ nodes = self.nodes.keys()
+ errors = []
+
+ for name in nodes - services:
+ errors.append(
+ 'Service `{}` is defined in `x-hierarchy` bot not in '
+ '`services`'.format(name)
+ )
+ for name in services - nodes:
+ errors.append(
+ 'Service `{}` is defined in `services` but not in '
+ '`x-hierarchy`'.format(name)
+ )
+
+ # trigger docker-compose's own validation
+ result = self._execute('ps', check=False, stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+
+ if result.returncode != 0:
+ # strip the intro line of docker-compose errors
+ errors += result.stderr.decode().splitlines()[1:]
+
+ if errors:
+ msg = '\n'.join([' - {}'.format(msg) for msg in errors])
+ raise ValueError(
+ 'Found errors with docker-compose:\n{}'.format(msg)
+ )
+
+ def _compose_env(self):
+ return dict(os.environ, **self.dotenv)
+
+ def _validate_image(self, name):
+ if name not in self.nodes:
+ raise UndefinedImage(name)
+
+ def _execute(self, *args, **kwargs):
+ # set default arguments for docker-compose
+ return super().run('--file', str(self.config_path), *args, **kwargs)
+
+ def build(self, image, cache=True, cache_leaf=True):
+ self._validate_image(image)
+ env = self._compose_env()
+
+ # build all parents
+ for parent in self.nodes[image]:
+ if cache:
+ self._execute('pull', '--ignore-pull-failures', parent,
Review comment:
Refactor into a small local function
```python
def build_target(target, cond, env):
if cond:
self._execute('pull', '--ignore-pull-failures', target, env=env)
self._execute('build', target, env=env)
else:
self._execute('build', '--no-cache', target, env=env)
```
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from pathlib import Path
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None):
+ self.config_path = Path(config_path)
+ if dotenv_path:
+ self.dotenv_path = Path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(self.dotenv_path)
+ self.bin = default_bin(compose_bin, "docker-compose")
+
+ def validate(self):
+ services = self.config['services'].keys()
+ nodes = self.nodes.keys()
+ errors = []
+
+ for name in nodes - services:
+ errors.append(
+ 'Service `{}` is defined in `x-hierarchy` bot not in '
+ '`services`'.format(name)
+ )
+ for name in services - nodes:
+ errors.append(
+ 'Service `{}` is defined in `services` but not in '
+ '`x-hierarchy`'.format(name)
+ )
+
+ # trigger docker-compose's own validation
+ result = self._execute('ps', check=False, stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+
+ if result.returncode != 0:
+ # strip the intro line of docker-compose errors
+ errors += result.stderr.decode().splitlines()[1:]
+
+ if errors:
+ msg = '\n'.join([' - {}'.format(msg) for msg in errors])
+ raise ValueError(
+ 'Found errors with docker-compose:\n{}'.format(msg)
+ )
+
+ def _compose_env(self):
+ return dict(os.environ, **self.dotenv)
+
+ def _validate_image(self, name):
+ if name not in self.nodes:
+ raise UndefinedImage(name)
+
+ def _execute(self, *args, **kwargs):
+ # set default arguments for docker-compose
+ return super().run('--file', str(self.config_path), *args, **kwargs)
+
+ def build(self, image, cache=True, cache_leaf=True):
+ self._validate_image(image)
+ env = self._compose_env()
+
+ # build all parents
+ for parent in self.nodes[image]:
+ if cache:
+ self._execute('pull', '--ignore-pull-failures', parent,
+ env=env)
+ self._execute('build', parent, env=env)
+ else:
+ self._execute('build', '--no-cache', parent, env=env)
+
+ # build the image at last
+ if cache and cache_leaf:
+ self._execute('pull', '--ignore-pull-failures', image, env=env)
+ self._execute('build', image, env=env)
+ else:
+ self._execute('build', '--no-cache', image, env=env)
+
+ def run(self, image, command=None, env=None):
+ self._validate_image(image)
+ args = []
+ if env is not None:
+ for k, v in env.items():
+ args.extend(['-e', '{}={}'.format(k, v)])
+ args.append(image)
+ if command is not None:
+ args.append(command)
+ self._execute('run', '--rm', *args, env=self._compose_env())
+
+ def push(self, image, user, password):
+ try:
+ Docker().run('login', '-u', user, '-p', password)
Review comment:
Use `--password-stdin`, otherwise they'll leak in the process argv via ps/top/etc and you won't need the next exception re-raising.
##########
File path: dev/archery/archery/cli.py
##########
@@ -643,5 +647,78 @@ def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option('--config', '-c', type=click.Path(exists=True),
Review comment:
Use `--src` like the other commands. See [archery/utils/source.py,](https://github.com/apache/arrow/blob/master/dev/archery/archery/utils/source.py) you may want to add a `docker_compose` property.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621697556
Revision: c01d30dc0e423e7dd201044f580b1434d670d4f4
Submitted crossbow builds: [ursa-labs/crossbow @ actions-180](https://github.com/ursa-labs/crossbow/branches/all?query=actions-180)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-180-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-180-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-180-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621173345
Revision: 91ecded74fc4c2096626005b1127a5811488555f
Submitted crossbow builds: [ursa-labs/crossbow @ actions-170](https://github.com/ursa-labs/crossbow/branches/all?query=actions-170)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-170-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-170-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-170-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r418596433
##########
File path: .github/workflows/archery.yml
##########
@@ -51,10 +53,12 @@ jobs:
python-version: '3.7'
- name: Install
working-directory: dev/archery
- run: pip install pytest responses ruamel.yaml toolz jinja2 -e .
+ run: pip install pytest python-dotenv responses ruamel.yaml toolz jinja2 -e .
Review comment:
Why not use pip install -e to ensure that requirements are up to date?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420036720
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
Review comment:
Right, updating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419711119
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from pathlib import Path
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None):
+ self.config_path = Path(config_path)
+ if dotenv_path:
+ self.dotenv_path = Path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(self.dotenv_path)
+ self.bin = default_bin(compose_bin, "docker-compose")
+
+ def validate(self):
+ services = self.config['services'].keys()
+ nodes = self.nodes.keys()
+ errors = []
+
+ for name in nodes - services:
+ errors.append(
+ 'Service `{}` is defined in `x-hierarchy` bot not in '
+ '`services`'.format(name)
+ )
+ for name in services - nodes:
+ errors.append(
+ 'Service `{}` is defined in `services` but not in '
+ '`x-hierarchy`'.format(name)
+ )
+
+ # trigger docker-compose's own validation
+ result = self._execute('ps', check=False, stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+
+ if result.returncode != 0:
+ # strip the intro line of docker-compose errors
+ errors += result.stderr.decode().splitlines()[1:]
+
+ if errors:
+ msg = '\n'.join([' - {}'.format(msg) for msg in errors])
+ raise ValueError(
+ 'Found errors with docker-compose:\n{}'.format(msg)
+ )
+
+ def _compose_env(self):
+ return dict(os.environ, **self.dotenv)
+
+ def _validate_image(self, name):
+ if name not in self.nodes:
+ raise UndefinedImage(name)
+
+ def _execute(self, *args, **kwargs):
+ # set default arguments for docker-compose
+ return super().run('--file', str(self.config_path), *args, **kwargs)
+
+ def build(self, image, cache=True, cache_leaf=True):
+ self._validate_image(image)
+ env = self._compose_env()
+
+ # build all parents
+ for parent in self.nodes[image]:
+ if cache:
+ self._execute('pull', '--ignore-pull-failures', parent,
+ env=env)
+ self._execute('build', parent, env=env)
+ else:
+ self._execute('build', '--no-cache', parent, env=env)
+
+ # build the image at last
+ if cache and cache_leaf:
+ self._execute('pull', '--ignore-pull-failures', image, env=env)
+ self._execute('build', image, env=env)
+ else:
+ self._execute('build', '--no-cache', image, env=env)
+
+ def run(self, image, command=None, env=None):
+ self._validate_image(image)
+ args = []
+ if env is not None:
+ for k, v in env.items():
+ args.extend(['-e', '{}={}'.format(k, v)])
+ args.append(image)
+ if command is not None:
+ args.append(command)
+ self._execute('run', '--rm', *args, env=self._compose_env())
+
+ def push(self, image, user, password):
+ try:
+ Docker().run('login', '-u', user, '-p', password)
Review comment:
Created follow-up https://issues.apache.org/jira/browse/ARROW-8698
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621266712
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419993794
##########
File path: .github/workflows/archery.yml
##########
@@ -48,13 +50,15 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v1
with:
- python-version: '3.7'
- - name: Install
+ python-version: '3.5'
Review comment:
Are you sure? That version is starting to get old...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621181816
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-623695509
@github-actions crossbow submit -g test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-622320787
Revision: f3fe79f4e89811a0e24b240d1b306315cdce95ee
Submitted crossbow builds: [ursa-labs/crossbow @ actions-192](https://github.com/ursa-labs/crossbow/branches/all?query=actions-192)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-cpp-valgrind)|
|test-conda-python-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.6)|
|test-conda-python-3.6-pandas-0.23|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.6-pandas-0.23)|
|test-conda-python-3.7|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7)|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-pandas-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-pandas-master)|
|test-conda-python-3.7-spark-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-spark-master)|
|test-conda-python-3.7-turbodbc-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-turbodbc-latest)|
|test-conda-python-3.7-turbodbc-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.7-turbodbc-master)|
|test-conda-python-3.8|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.8)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.8-dask-master)|
|test-conda-python-3.8-jpype|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.8-jpype)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-python-3.8-pandas-latest)|
|test-conda-r-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-conda-r-3.6)|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-debian-10-cpp)|
|test-debian-10-go-1.12|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-debian-10-go-1.12)|
|test-debian-10-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-debian-10-python-3)|
|test-debian-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-debian-c-glib)|
|test-debian-ruby|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-debian-ruby)|
|test-fedora-30-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-fedora-30-cpp)|
|test-fedora-30-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-fedora-30-python-3)|
|test-r-linux-as-cran|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-192-github-test-r-linux-as-cran)|
|test-r-rhub-ubuntu-gcc-release|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rhub-ubuntu-gcc-release)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rstudio-r-base-3.6-centos6)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-ubuntu-16.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-16.04-cpp)|
|test-ubuntu-18.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-cmake32|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-18.04-cpp-cmake32)|
|test-ubuntu-18.04-cpp-release|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-docs|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-ubuntu-18.04-docs)|
|test-ubuntu-18.04-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-ubuntu-18.04-python-3)|
|test-ubuntu-18.04-r-3.6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-ubuntu-18.04-r-3.6)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-192-circle-test-ubuntu-c-glib)|
|test-ubuntu-ruby|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-192-azure-test-ubuntu-ruby)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420019510
##########
File path: .github/workflows/cpp_cron.yml
##########
@@ -36,170 +36,69 @@ env:
jobs:
- debian:
- name: AMD64 Debian ${{ matrix.debian }} C++
+ docker:
+ name: ${{ matrix.title }}
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') && github.repository == 'apache/arrow' }}
strategy:
fail-fast: false
matrix:
- debian: [10]
+ name:
+ - debian-10-cpp
+ - fedora-30-cpp
+ - ubuntu-16.04-cpp
+ - ubuntu-18.04-cpp
+ - ubuntu-18.04-cpp-cmake32
+ include:
+ - name: debian-10-cpp
+ image: debian-cpp
+ title: AMD64 Debian 10 C++
+ debian: 10
+ - name: fedora-30-cpp
+ image: fedora-cpp
+ title: AMD64 Fedora 30 C++
+ fedora: 30
+ - name: ubuntu-16.04-cpp
+ image: ubuntu-cpp
+ title: AMD64 Ubuntu 16.04 C++
+ ubuntu: 16.04
+ - name: ubuntu-18.04-cpp
+ image: ubuntu-cpp
+ title: AMD64 Ubuntu 18.04 C++
+ ubuntu: 18.04
+ - name: ubuntu-18.04-cpp-cmake32
+ image: ubuntu-cpp-cmake32
+ title: AMD64 Ubuntu 18.04 C++ CMake 3.2
+ ubuntu: 18.04
env:
- DEBIAN: ${{ matrix.debian }}
+ # the defaults here should correspond to the values in .env
Review comment:
The `docker-compose.yml` doesn't contain default values for these environment variables, they are grabbed from the dotenv file.
If we don't provide a default value for the env variable in the GHA yml then it'll set it to empty. Since we need to explicitly list the docker volumes in the docker-compose.yml:
```
amd64-ubuntu-14.04-cache:
amd64-ubuntu-16.04-cache:
amd64-ubuntu-18.04-cache:
amd64-ubuntu-20.04-cache:
```
So an empty UBUNTU variable would look for `amd64-ubuntu--cache` which is not defined, so docker-compose bails out.
In my upcoming GHA cache PR I change the type of the docker volumes which may let us to remove the explicit volume definitions, so perhaps we can remove the default env values from the workflow ymls as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419809722
##########
File path: dev/archery/archery/cli.py
##########
@@ -30,14 +31,16 @@
from .utils.codec import JsonEncoder
from .utils.lint import linter, python_numpydoc, LintValidationException
from .utils.logger import logger, ctx as log_ctx
-from .utils.source import ArrowSources
+from .utils.source import ArrowSources, InvalidArrowSource
from .utils.tmpdir import tmpdir
-from .bot import CommentBot, actions
# Set default logging to INFO in command line.
logging.basicConfig(level=logging.INFO)
+DEFAULT_ARROW_PATH = Path(__file__).parents[3].absolute()
Review comment:
Is this used?
##########
File path: dev/archery/archery/docker.py
##########
@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import subprocess
+
+from dotenv import dotenv_values
+from ruamel.yaml import YAML
+
+from .utils.command import Command, default_bin
+from .compat import _ensure_path
+
+
+def flatten(node, parents=None):
+ parents = list(parents or [])
+ if isinstance(node, str):
+ yield (node, parents)
+ elif isinstance(node, list):
+ for value in node:
+ yield from flatten(value, parents=parents)
+ elif isinstance(node, dict):
+ for key, value in node.items():
+ yield (key, parents)
+ yield from flatten(value, parents=parents + [key])
+ else:
+ raise TypeError(node)
+
+
+class UndefinedImage(Exception):
+ pass
+
+
+class Docker(Command):
+
+ def __init__(self, docker_bin=None):
+ self.bin = default_bin(docker_bin, "docker")
+
+
+class DockerCompose(Command):
+
+ def __init__(self, config_path, dotenv_path=None, compose_bin=None,
+ params=None):
+ self.config_path = _ensure_path(config_path)
+ if dotenv_path:
+ self.dotenv_path = _ensure_path(dotenv_path)
+ else:
+ self.dotenv_path = self.config_path.parent / '.env'
+
+ yaml = YAML()
+ with self.config_path.open() as fp:
+ self.config = yaml.load(fp)
+
+ self.bin = default_bin(compose_bin, 'docker-compose')
+ self.nodes = dict(flatten(self.config['x-hierarchy']))
+ self.dotenv = dotenv_values(str(self.dotenv_path))
+ if params is None:
+ self.params = {}
+ else:
+ self.params = {k: v for k, v in params.items() if k in self.dotenv}
+
+ # forward the process' environment variables
Review comment:
The calling process' environment should override the dotenv, it's the only way to override a configuration without touching the .env file.
```
$ UBUNTU=20.04 archery docker run conda-cpp
Building conda-cpp
Step 1/16 : ARG arch=amd64
Step 2/16 : FROM ${arch}/ubuntu:18.04
---> c3c304cb4f22
Step 3/16 : ARG arch=amd64
---> Running in 13411a23c148
```
Something is wrong with the unit test too if it didn't catch this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420023826
##########
File path: dev/archery/archery/testing.py
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from contextlib import contextmanager
+import os
+from unittest import mock
+import re
+
+
+class PartialEnv(dict):
+
+ def __eq__(self, other):
+ return self.items() <= other.items()
Review comment:
I use it to check only the relevant subset of environment variables in the mocked `subprocess.run(..., env=env)` call, see assertion [here](https://github.com/apache/arrow/pull/7021/files#diff-518bc71ef52fefbb9cfd67d60eeb9a7fR312).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420024247
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
Review comment:
Updating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621199563
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621285608
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420019964
##########
File path: .github/workflows/python.yml
##########
@@ -34,103 +34,75 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
jobs:
- ubuntu:
- name: AMD64 Ubuntu 16.04 Python 3.5
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ docker:
+ name: ${{ matrix.title }}
+ runs-on: ubuntu-latest
+ if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ strategy:
+ fail-fast: false
+ matrix:
+ name:
+ - ubuntu-16.04-python-3
+ - conda-python-3.6-nopandas
+ - conda-python-3.8-pandas-latest
+ include:
+ - name: ubuntu-16.04-python-3
+ image: ubuntu-python
+ # this image always builds with python 3.5
+ title: AMD64 Ubuntu 16.04 Python 3.5
+ ubuntu: 16.04
+ - name: conda-python-3.8-nopandas
+ image: conda-python
+ title: AMD64 Conda Python 3.8 Without Pandas
+ python: 3.8
+ - name: conda-python-3.6-pandas-0.23
+ image: conda-python-pandas
+ title: AMD64 Conda Python 3.8 Pandas 0.23
+ python: 3.6
Review comment:
Nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420029596
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,224 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+``archery`` requires ``python>=3.5``. It is recommended to install archery in
+``editable`` mode with the ``-e`` flag to automatically update the intallation
+by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+For the available commands and options invoke the installed archery commands
+with the ``--help`` flag:
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+
+Examples
+~~~~~~~~
+
+**List the available images:**
+
+.. code:: bash
+
+ archery docker images
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**Show the docker-compose commands instead of executing them:**
+
+.. code:: bash
+
+ archery docker run --dry-run conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+In case of the example below the command builds the
+``conda-cpp > conda-python > conda-python-pandas`` branch of the image tree
+where the leaf image is ``conda-python-pandas``.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doesn't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+The layer caching mechanism of docker-compose is less reliable than docker's
+depending on the version, ``cache_from`` build entry and the used backend
+(docker-py, docker-cli, docker-cli and buildkit). This can lead to different
+layer hashes - even when executing the same build command repeatedly -
+eventually causing cache misses full image rebuilds.
+
+If the image has been already built but the cache doesn't work properly, it can
+be useful to skip the build phases:
Review comment:
Tried to illustrate with an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621947873
Revision: 0d768d6948627f165819cbb4542fca1503a340b1
Submitted crossbow builds: [ursa-labs/crossbow @ actions-184](https://github.com/ursa-labs/crossbow/branches/all?query=actions-184)
|Task|Status|
|----|------|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-184-circle-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621356894
Revision: 719a93477453133902369cbad5a979a9eb8c4b04
Submitted crossbow builds: [ursa-labs/crossbow @ actions-176](https://github.com/ursa-labs/crossbow/branches/all?query=actions-176)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-176-azure-test-conda-python-3.7)|
|test-conda-python-3.7-pandas-latest|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-176-circle-test-conda-python-3.7-pandas-latest)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-176-github-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621250583
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419646543
##########
File path: docs/source/developers/integration.rst
##########
@@ -56,15 +56,19 @@ build mount is used for caching and sharing state between staged images.
You can build and run a service by using the `build` and `run` docker-compose
sub-command, e.g. `docker-compose build python && docker-compose run python`.
We do not publish the build images, you need to build them manually. This
-method requires the user to build the images in reverse dependency order. To
-simplify this, we provide a Makefile.
+method requires the user to build the images in reverse dependency order.
.. code-block:: shell
# Build and run manually
- docker-compose build cpp
- docker-compose build python
- docker-compose run python
+ docker-compose build conda-cpp
+ docker-compose build conda-python
+ docker-compose run conda-python
- # Using the makefile with proper image dependency resolution
- make -f Makefile.docker python
+To simplify this, Archery provides a command for it:
+
+.. code-block:: shell
+
+ archery docker run conda-python
+
+See the `Running Docker Builds`_ for more details.
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419995918
##########
File path: .github/workflows/python.yml
##########
@@ -34,103 +34,75 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
jobs:
- ubuntu:
- name: AMD64 Ubuntu 16.04 Python 3.5
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ docker:
+ name: ${{ matrix.title }}
+ runs-on: ubuntu-latest
+ if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ strategy:
+ fail-fast: false
+ matrix:
+ name:
+ - ubuntu-16.04-python-3
+ - conda-python-3.6-nopandas
+ - conda-python-3.8-pandas-latest
+ include:
+ - name: ubuntu-16.04-python-3
+ image: ubuntu-python
+ # this image always builds with python 3.5
+ title: AMD64 Ubuntu 16.04 Python 3.5
+ ubuntu: 16.04
+ - name: conda-python-3.8-nopandas
+ image: conda-python
+ title: AMD64 Conda Python 3.8 Without Pandas
+ python: 3.8
+ - name: conda-python-3.6-pandas-0.23
+ image: conda-python-pandas
+ title: AMD64 Conda Python 3.8 Pandas 0.23
+ python: 3.6
Review comment:
Title says "3.8"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r420021398
##########
File path: dev/archery/archery/cli.py
##########
@@ -650,11 +651,130 @@ def integration(with_all=False, random_seed=12345, **args):
@click.option('--crossbow-token', '-ct', envvar='CROSSBOW_GITHUB_TOKEN',
help='OAuth token for pushing to the crossow repository')
def trigger_bot(event_name, event_payload, arrow_token, crossbow_token):
+ from .bot import CommentBot, actions
+
event_payload = json.loads(event_payload.read())
bot = CommentBot(name='github-actions', handler=actions, token=arrow_token)
bot.handle(event_name, event_payload)
+@archery.group('docker')
+@click.option("--src", metavar="<arrow_src>", default=None,
+ callback=validate_arrow_sources,
+ help="Specify Arrow source directory.")
+@click.pass_obj
+def docker_compose(obj, src):
+ """Interact with docker-compose based builds."""
+ from .docker import DockerCompose
+
+ config_path = src.path / 'docker-compose.yml'
+ if not config_path.exists():
+ raise click.ClickException(
+ "Docker compose configuration cannot be found in directory {}, "
+ "try to pass the arrow source directory explicitly.".format(src)
+ )
+
+ # take the docker-compose parameters like PYTHON, PANDAS, UBUNTU from the
+ # environment variables to keep the usage similar to docker-compose
+ obj['compose'] = DockerCompose(config_path, params=os.environ)
+ obj['compose'].validate()
+
+
+@docker_compose.command('run')
+@click.argument('image')
+@click.argument('command', required=False, default=None)
+@click.option('--env', '-e', multiple=True,
+ help='Set environment variable within the container')
+@click.option('--build/--no-build', default=True,
+ help='Force build the image and its parents')
+@click.option('--cache/--no-cache', default=True,
+ help='Try to pull the image and its parents')
+@click.option('--cache-leaf/--no-cache-leaf', default=True,
+ help='Force build the image and its parents')
Review comment:
I mean the description doesn't correspond to the option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621760055
Revision: d84d408146463c6418d2d2c8901894daf6b816c3
Submitted crossbow builds: [ursa-labs/crossbow @ actions-182](https://github.com/ursa-labs/crossbow/branches/all?query=actions-182)
|Task|Status|
|----|------|
|test-conda-python-3.7|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-182-azure-test-conda-python-3.7)|
|test-debian-10-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-182-github-test-debian-10-cpp)|
|test-debian-10-go-1.12|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-182-circle-test-debian-10-go-1.12)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-623696198
Revision: a62b83c6a5f64894e3ebb9e31a50e942a881f799
Submitted crossbow builds: [ursa-labs/crossbow @ actions-202](https://github.com/ursa-labs/crossbow/branches/all?query=actions-202)
|Task|Status|
|----|------|
|test-conda-cpp|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-cpp-valgrind)|
|test-conda-python-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.6)|
|test-conda-python-3.6-pandas-0.23|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.6-pandas-0.23)|
|test-conda-python-3.7|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7)|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-pandas-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-pandas-master)|
|test-conda-python-3.7-spark-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-spark-master)|
|test-conda-python-3.7-turbodbc-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-turbodbc-latest)|
|test-conda-python-3.7-turbodbc-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.7-turbodbc-master)|
|test-conda-python-3.8|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.8)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.8-dask-master)|
|test-conda-python-3.8-jpype|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.8-jpype)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-python-3.8-pandas-latest)|
|test-conda-r-3.6|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-conda-r-3.6)|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-debian-10-cpp)|
|test-debian-10-go-1.12|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-debian-10-go-1.12)|
|test-debian-10-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-debian-10-python-3)|
|test-debian-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-debian-c-glib)|
|test-debian-ruby|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-debian-ruby)|
|test-fedora-30-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-fedora-30-cpp)|
|test-fedora-30-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-fedora-30-python-3)|
|test-r-linux-as-cran|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-202-github-test-r-linux-as-cran)|
|test-r-rhub-ubuntu-gcc-release|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rhub-ubuntu-gcc-release)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rstudio-r-base-3.6-centos6)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-ubuntu-16.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-16.04-cpp)|
|test-ubuntu-18.04-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-cmake32|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-18.04-cpp-cmake32)|
|test-ubuntu-18.04-cpp-release|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-docs|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-ubuntu-18.04-docs)|
|test-ubuntu-18.04-python-3|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-ubuntu-18.04-python-3)|
|test-ubuntu-18.04-r-3.6|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-ubuntu-18.04-r-3.6)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-c-glib|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-202-circle-test-ubuntu-c-glib)|
|test-ubuntu-ruby|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-202-azure-test-ubuntu-ruby)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs removed a comment on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs removed a comment on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621199563
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621172630
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419995997
##########
File path: .github/workflows/python.yml
##########
@@ -34,103 +34,75 @@ on:
env:
DOCKER_BUILDKIT: 0
COMPOSE_DOCKER_CLI_BUILD: 1
+ ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_TOKEN }}
+ ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_USER }}
jobs:
- ubuntu:
- name: AMD64 Ubuntu 16.04 Python 3.5
- runs-on: ubuntu-latest
- if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ docker:
+ name: ${{ matrix.title }}
+ runs-on: ubuntu-latest
+ if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ strategy:
+ fail-fast: false
+ matrix:
+ name:
+ - ubuntu-16.04-python-3
+ - conda-python-3.6-nopandas
+ - conda-python-3.8-pandas-latest
+ include:
+ - name: ubuntu-16.04-python-3
+ image: ubuntu-python
+ # this image always builds with python 3.5
+ title: AMD64 Ubuntu 16.04 Python 3.5
+ ubuntu: 16.04
+ - name: conda-python-3.8-nopandas
+ image: conda-python
+ title: AMD64 Conda Python 3.8 Without Pandas
+ python: 3.8
+ - name: conda-python-3.6-pandas-0.23
+ image: conda-python-pandas
+ title: AMD64 Conda Python 3.8 Pandas 0.23
+ python: 3.6
+ pandas: 0.23
+ - name: conda-python-3.6-pandas-latest
+ image: conda-python-pandas
+ title: AMD64 Conda Python 3.8 Pandas latest
+ python: 3.6
Review comment:
Title says "3.8"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-620234647
> it's pretty straightforward to use docker-compose to reproduce these builds locally
I see both sides of this -- with some builds requiring 3 or more `docker-compose build` invocations and relying on various environment variables / parameters, there's a lot of room for frustration and user error (speaking from experience). I'm also pretty anti-copypasta in build configurations -- having all the build logic tightly coupled to GHA seems to go against some of the spirit of our prior decoupling from Travis CI, Appveyor.
Of course, everything has to be well-documented and able to be serviced successfully by most Arrow maintainers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621958397
Revision: 0b532b06da2ffab64d34d4420790eee3e4f64ca3
Submitted crossbow builds: [ursa-labs/crossbow @ actions-185](https://github.com/ursa-labs/crossbow/branches/all?query=actions-185)
|Task|Status|
|----|------|
|test-debian-10-cpp|[](https://circleci.com/gh/ursa-labs/crossbow/tree/actions-185-circle-test-debian-10-cpp)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419365852
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
There is a tiny intro under the benchmarking guide. I'm deferring it to a follow-up https://issues.apache.org/jira/browse/ARROW-8688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419518977
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
Review comment:
Does it though? `run` here seems to entail `build && run`.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
An immediate followup, right? I'm not sure that I agree with deferring but I'll allow it as long as it's the next thing you do. It looks like you already added some of the relevant information below, not sure how much more you need.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
Review comment:
Sounds like YAGNI to me, anyone else have an opinion?
##########
File path: docs/source/example1.dat
##########
@@ -0,0 +1 @@
+some data
Review comment:
Was this supposed to be checked in?
##########
File path: docs/source/developers/integration.rst
##########
@@ -56,15 +56,19 @@ build mount is used for caching and sharing state between staged images.
You can build and run a service by using the `build` and `run` docker-compose
sub-command, e.g. `docker-compose build python && docker-compose run python`.
We do not publish the build images, you need to build them manually. This
-method requires the user to build the images in reverse dependency order. To
-simplify this, we provide a Makefile.
+method requires the user to build the images in reverse dependency order.
.. code-block:: shell
# Build and run manually
- docker-compose build cpp
- docker-compose build python
- docker-compose run python
+ docker-compose build conda-cpp
+ docker-compose build conda-python
+ docker-compose run conda-python
- # Using the makefile with proper image dependency resolution
- make -f Makefile.docker python
+To simplify this, Archery provides a command for it:
+
+.. code-block:: shell
+
+ archery docker run conda-python
+
+See the `Running Docker Builds`_ for more details.
Review comment:
```suggestion
See `Running Docker Builds`_ for more details.
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
Review comment:
Wish: a `--dry-run` flag that just prints the commands that would be run. Would be helpful for those who are more familiar with `docker-compose` and for those who need to edit docker-compose.yml
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
Review comment:
This still doesn't tell me what a leaf image is.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
Review comment:
```suggestion
``archery`` requires ``python>=3.5``. It is recommended to install archery in ``editable``
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
+
+**Run the image with custom command:**
+
+Custom docker commands may be passed as the second argument. The following
Review comment:
second argument? Looks like the fourth to me.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
Review comment:
why would I do this?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
Review comment:
How? (be explicit, don't make me guess by inspecting the example)
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
Review comment:
available C++... what? build flags?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
Review comment:
```suggestion
Note that it doesn't pull the conda-python-pandas image and disable the cache
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
+
+**Run the image with custom command:**
+
+Custom docker commands may be passed as the second argument. The following
+example starts an interactive ``bash`` session in the container - useful for
+debugging the build interactively:
+
+.. code:: bash
+
+ archery docker run ubuntu-cpp bash
+
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
Review comment:
```suggestion
This reduces duplication and streamlines maintenance, but
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r419646333
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
+
+**Run the image with custom command:**
+
+Custom docker commands may be passed as the second argument. The following
Review comment:
Of the `archery docker run` command.
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,197 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommended is
+to use the archery tool:
+
+Installation
+~~~~~~~~~~~~
+
+Requires ``python>=3.5``. It is recommended to install archery in ``editable``
+mode to automatically update the intallation by pulling the arrow repository.
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Inspect the available commands and options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: bash
+
+ archery docker --help
+ archery docker run --help
+
+``archery docker run`` tries to provide a similar interface to
+``docker-compose run``, but with builtin support for building hierarchical
+images.
+
+Examples
+~~~~~~~~
+
+**Execute a build:**
+
+.. code:: bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code:: bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+**To disable the image pulling:**
+
+.. code:: bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code:: bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+**To disable the cache only for the leaf image:**
+
+Useful to force building the development version of a dependency.
+The leaf image is ``conda-python-pandas`` in the example.
+
+.. code:: bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python-pandas
+
+Which translates to:
+
+.. code:: bash
+
+ export PANDAS=master
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose build --no-cache conda-python-pandas
+ docker-compose run --rm conda-python-pandas
+
+Note that it doens't pull the conda-python-pandas image and disable the cache
+when building it.
+
+``PANDAS`` is a `build parameter <Docker Build Parameters>`_, see the
+defaults in the .env file.
+
+**To entirely skip building the image:**
+
+.. code:: bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass docker environment variables to
+the container during its execution:
+
+.. code:: bash
+
+ archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+See the available C++ in the ``ci/scripts/cpp_build.sh`` script.
+
+**Run the image with custom command:**
+
+Custom docker commands may be passed as the second argument. The following
+example starts an interactive ``bash`` session in the container - useful for
+debugging the build interactively:
+
+.. code:: bash
+
+ archery docker run ubuntu-cpp bash
+
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7021: Wrap docker-compose commands with archery
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#issuecomment-621199652
@github-actions crossbow submit test-conda-python-3.7-pandas-latest test-debian-10-cpp test-conda-python-3.7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org