You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/05/01 16:56:29 UTC
[GitHub] [arrow] nealrichardson commented on a change in pull request #7021: ARROW-8628: [Dev] Wrap docker-compose commands with archery
nealrichardson commented on a change in pull request #7021:
URL: https://github.com/apache/arrow/pull/7021#discussion_r418622281
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
Can this link to the main Archery docs? (*Are there Archery docs?*)
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
Review comment:
```suggestion
There are multiple ways to execute the docker based builds. The recommended is
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
Review comment:
Can I run these docker-compose commands myself?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
Review comment:
Would I ever call `archery docker` and not immediately put `run` next? If not, why not just `archery run conda-python`?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
Review comment:
Among the questions I'd have for the archery docs specifically right here:
* What is archery?
* What system requirements does it have? (like, python version?)
* What do I do if the simple instructions don't work?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
Review comment:
What's a leaf image?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
Review comment:
Is there `archery docker --help` or something that tell me all of the flags like this?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
Review comment:
This needs more explanation, particularly to distinguish build-time parameters from environment variables
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
+
+A good example is cpp_build.sh build script which forwards environment
+variables as CMake options - so the same scripts can be invoked in various
+configurations without the necessity of chaning it. For examples see how the
Review comment:
chaning?
##########
File path: docs/source/developers/integration.rst
##########
@@ -62,9 +62,9 @@ simplify this, we provide a Makefile.
.. code-block:: shell
# Build and run manually
- docker-compose build cpp
- docker-compose build python
- docker-compose run python
+ docker-compose build conda-cpp
+ docker-compose build conda-python
+ docker-compose run conda-python
- # Using the makefile with proper image dependency resolution
- make -f Makefile.docker python
+ # Using the archery with proper image dependency resolution
Review comment:
```suggestion
# Using archery with proper image dependency resolution
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
Review comment:
That's what `-e` means?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
Review comment:
For C++, can I pass all cmake flags that way?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
+
+A good example is cpp_build.sh build script which forwards environment
+variables as CMake options - so the same scripts can be invoked in various
+configurations without the necessity of chaning it. For examples see how the
+environment variables are passed in the docker-compose.yml's C++ images.
+
+Adding New Images
+~~~~~~~~~~~~~~~~~
+
+See more in the docker-compose.yml
Review comment:
I don't understand
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
Review comment:
```suggestion
possible to be responsible for only a subset of tasks.
```
What subset? Why minimal?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
Review comment:
Ok, these are the build parameters. Can you link to this section where you mention that above?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
Review comment:
```suggestion
The parametrization is done through environment variables with useful defaults to
```
What parametrization?
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
Review comment:
```suggestion
keep the build configurations declarative. Note that these parameters are
```
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
Review comment:
... by doing what? (Please be explicit so I don't have to diff the examples to figure out what to do)
##########
File path: docs/source/developers/docker.rst
##########
@@ -0,0 +1,143 @@
+.. raw:: html
+
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+Running Docker Builds
+=====================
+
+Most of our Linux based continuous integration tasks are decoupled from public
+CI services using docker and docker-compose. Keeping the CI configuration
+minimal makes local reproducibility possible.
+
+Usage
+-----
+
+There are multiple ways to execute the docker based builds. The recommented is
+to use the archery tool:
+
+Installation:
+
+.. code:: bash
+
+ pip install -e dev/archery[docker]
+
+Execute a build:
+
+.. code::bash
+
+ archery docker run conda-python
+
+Archery calls the following docker-compose commands:
+
+.. code::bash
+
+ docker-compose pull --ignore-pull-failures conda-cpp
+ docker-compose build conda-cpp
+ docker-compose pull --ignore-pull-failures conda-python
+ docker-compose build conda-python
+ docker-compose run --rm conda-python
+
+To disable the image pulling:
+
+.. code::bash
+
+ archery docker run --no-cache conda-python
+
+Which translates to:
+
+.. code::bash
+
+ docker-compose build --no-cache conda-cpp
+ docker-compose build --no-cache conda-python
+ docker-compose run --rm conda-python
+
+To disable the cache only for the leaf image - useful to force building the
+development version of a dependency:
+
+.. code::bash
+
+ PANDAS=master archery docker run --no-cache-leaf conda-python
+
+``PANDAS`` is a build time parameter, see the defaults in the .env file.
+
+To entirely skip building the image:
+
+.. code::bash
+
+ archery docker run --no-build conda-python
+
+In order to alter the runtime parameters pass environment variables to the
+container in execution time:
+
+.. code::bash
+
+ archery docker run -e CMAKE_BUILD_TYPE=release ubuntu-cpp
+
+Starting an interactive bash session for debugging:
+
+.. code::bash
+
+ archery docker run ubuntu-cpp bash
+
+Development
+-----------
+
+The docker-compose configuration is tuned towards reusable development
+containers using hierarchical images. For example multiple language bindings
+are dependent on the C++ implementation, so instead of redefining the
+C++ environment multiple Dockerfiles, we can reuse the exact same base C++
+image when building Glib, Ruby, R and Python bindings.
+This helps reducing duplications and preventing a series of maintenance, but
+makes the docker-compose configuration more complicated.
+
+Docker Build Parameters
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The build time parameters are pushed down to the [dockerfiles] to make the
+image building more flexible. These are usually called docker build args.
+The docker-compose.yml uses these build args extensively for:
+
+- defining the docker registry used for caching
+- platform architectures
+- operation systems and versions
+- defining various versions if dependencies
+
+The default values are stored in the top level .env file. For examples see the
+docker-compose.yml
+
+Build Scripts
+~~~~~~~~~~~~~
+
+The scripts maintainted under ci/scripts directory should be kept as minimal as
+possible to be responsible only a subset of tasks.
+
+The parametrization is done through environment variables with sane defaults to
+keep the build configurations declerative. Note that these parameters are
+different from the ones described previously, these are affecting the runtime
+behaviour of the builds scripts within the containers.
Review comment:
```suggestion
behavior of the builds scripts within the containers.
```
Also this sentence is not clear to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org