You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/11 09:51:24 UTC
[GitHub] [arrow] raulcd opened a new pull request, #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
raulcd opened a new pull request, #12854:
URL: https://github.com/apache/arrow/pull/12854
This PR adds new jobs to the Python CI worflow in order to exercise the Python minimal build examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd closed pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd closed pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
URL: https://github.com/apache/arrow/pull/12854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1098278427
Can we make these nightly builds instead? We already have a lot of queueing in Github Actions PR jobs currently and we should try to avoid adding more builds, IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849352296
##########
python/examples/minimal_build/build_conda.sh:
##########
@@ -28,8 +28,6 @@ CPP_BUILD_DIR=$HOME/arrow-cpp-build
ARROW_ROOT=/arrow
PYTHON=3.10
-git clone --depth=100 https://github.com/apache/arrow.git /arrow
-
#----------------------------------------------------------------------
Review Comment:
Is this script used only in the minimal build docker images? If yes then could we use `ci/scripts/install_conda.sh` instead in the dockerfile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1095110109
On this workflow run we can see how a test failed on the minimal build and the jobs failed:
https://github.com/raulcd/arrow/actions/runs/2148907327
I am reverting the test failure to have the minimal builds jobs pass again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849354148
##########
python/examples/minimal_build/docker-compose.yml:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+version: '3.5'
+
+services:
+ minimal-fedora-conda:
+ build:
+ context: .
+ dockerfile: Dockerfile.fedora
+ volumes:
Review Comment:
Can you use yml anchors here, like [in the main docker compose file](https://github.com/apache/arrow/blob/master/docker-compose.yml#L245)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1094836144
https://issues.apache.org/jira/browse/ARROW-15893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1096495906
@kszucs if you can take a look on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849346743
##########
.github/workflows/python.yml:
##########
@@ -112,6 +112,43 @@ jobs:
continue-on-error: true
run: archery docker push ${{ matrix.image }}
+ build-example:
+ name: ${{ matrix.title }}
+ runs-on: ubuntu-latest
+ if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ timeout-minutes: 60
+ strategy:
+ fail-fast: false
+ matrix:
+ name:
+ - minimal-fedora-conda
+ - minimal-fedora-venv
+ - minimal-ubuntu-conda
+ - minimal-ubuntu-venv
+ include:
+ - name: minimal-fedora-conda
+ image: minimal-fedora-conda
+ title: Fedora Conda Minimal Build
Review Comment:
Could you please use `<arch> <os> <version> <build-name>` title to better align with the rest of the build titles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849879561
##########
python/examples/minimal_build/README.md:
##########
@@ -34,10 +34,21 @@ docker build -t arrow_fedora_minimal -f Dockerfile.fedora .
Then build PyArrow with conda or pip, respectively:
```
# With pip
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_venv.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_venv.sh
# With conda
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_conda.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_conda.sh
+```
+
+Alternatively you can use [Docker Compose][docker-compose] to build and run using:
Review Comment:
I agree with the idea.
It seems that we can use `docker-compose` with `podman` by `dnf install podman-plugins` and `systemctl enable --now podman.socket`. So we can document `docker-compose` interface and how to use `docker-compose` with `podman`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849346743
##########
.github/workflows/python.yml:
##########
@@ -112,6 +112,43 @@ jobs:
continue-on-error: true
run: archery docker push ${{ matrix.image }}
+ build-example:
+ name: ${{ matrix.title }}
+ runs-on: ubuntu-latest
+ if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+ timeout-minutes: 60
+ strategy:
+ fail-fast: false
+ matrix:
+ name:
+ - minimal-fedora-conda
+ - minimal-fedora-venv
+ - minimal-ubuntu-conda
+ - minimal-ubuntu-venv
+ include:
+ - name: minimal-fedora-conda
+ image: minimal-fedora-conda
+ title: Fedora Conda Minimal Build
Review Comment:
Could you please use `<arch> <os> <version> <build-name>` title to better align with the rest of the build titles?
In this case something like: "AMD64 Fedora <version> Python Minimal Conda Build"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r847327663
##########
python/examples/minimal_build/README.md:
##########
@@ -34,10 +34,21 @@ docker build -t arrow_fedora_minimal -f Dockerfile.fedora .
Then build PyArrow with conda or pip, respectively:
```
# With pip
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_venv.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_venv.sh
# With conda
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_conda.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_conda.sh
+```
+
+Alternatively you can use [Docker Compose][docker-compose] to build and run using:
Review Comment:
For the reviewer, do you think it would be better to only document the Docker compose interface as we do on the CPP minimal build instead of having `docker`, `docker-compose` and `podman`? See:
https://github.com/apache/arrow/blob/master/cpp/examples/minimal_build/README.md#running-the-example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849348343
##########
python/examples/minimal_build/README.md:
##########
@@ -34,10 +34,21 @@ docker build -t arrow_fedora_minimal -f Dockerfile.fedora .
Then build PyArrow with conda or pip, respectively:
```
# With pip
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_venv.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_venv.sh
# With conda
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_conda.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_conda.sh
+```
+
+Alternatively you can use [Docker Compose][docker-compose] to build and run using:
Review Comment:
I don't have a strong opinion here, but I agree that docker-compose and podman should be enough especially since the volume paths getting more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849348343
##########
python/examples/minimal_build/README.md:
##########
@@ -34,10 +34,21 @@ docker build -t arrow_fedora_minimal -f Dockerfile.fedora .
Then build PyArrow with conda or pip, respectively:
```
# With pip
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_venv.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_venv.sh
# With conda
-docker run --rm -t -i -v $PWD:/io arrow_fedora_minimal /io/build_conda.sh
+docker run --rm -t -i -v $PWD:/io -v $PWD/../../..:/arrow arrow_fedora_minimal /io/build_conda.sh
+```
+
+Alternatively you can use [Docker Compose][docker-compose] to build and run using:
Review Comment:
I don't have a strong opinion here, but I agree that docker-compose and podman should be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
kszucs commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849512829
##########
python/examples/minimal_build/build_conda.sh:
##########
@@ -28,8 +28,6 @@ CPP_BUILD_DIR=$HOME/arrow-cpp-build
ARROW_ROOT=/arrow
PYTHON=3.10
-git clone --depth=100 https://github.com/apache/arrow.git /arrow
-
#----------------------------------------------------------------------
Review Comment:
Sure, feel free to defer it to a follow-up ticket.
Installing conda can take significant time which is hard to iterate with. This time can be reduced by installing conda during docker-build in the relevant dockerfile. We could also switch to use the official conda image instead but still installing the conda dependencies from the dockerfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1102625802
Closing this temporally and moving the ticket to 9.0.0. I'll create a new PR with new nightly test builds instead of adding them to our python builds as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1124910804
This is continued in https://github.com/apache/arrow/pull/13113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1096495905
@kszucs if you can take a look on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
raulcd commented on code in PR #12854:
URL: https://github.com/apache/arrow/pull/12854#discussion_r849503021
##########
python/examples/minimal_build/build_conda.sh:
##########
@@ -28,8 +28,6 @@ CPP_BUILD_DIR=$HOME/arrow-cpp-build
ARROW_ROOT=/arrow
PYTHON=3.10
-git clone --depth=100 https://github.com/apache/arrow.git /arrow
-
#----------------------------------------------------------------------
Review Comment:
Yes, this is used only on the minimal build examples for conda. I am not entirely sure about using the `ci/scripts/install_conda.sh`.
The current script does not only install conda but contains all the steps for the minimal build in order to help debug basic build environments, based on the `README.md`:
```
This directory shows how to bootstrap a local build from source on Linux with
an eye toward maximum portability across different Linux distributions. This
may help for contributors debugging build issues caused by their local
environments.
```
In my opinion splitting the `install/build` steps on two different scripts, the install on `ci/scripts/install_conda.sh` and the build on `python/examples/minimal_build/build_conda.sh` might be confusing for the purpose of debugging minimal builds.
I also understand that having two "install_conda" scripts is also not ideal.
Is your idea to have these two steps split on two different scripts?
If that's the case I would propose to include it on a different ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on pull request #12854: ARROW-15893: [Python] Exercise Python minimal build examples on CI
Posted by GitBox <gi...@apache.org>.
amol- commented on PR #12854:
URL: https://github.com/apache/arrow/pull/12854#issuecomment-1097887585
@kou would you mind taking a look when you have the time to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org