You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2023/12/14 20:57:51 UTC
[PR] chore(ci): Update docker images [arrow-nanoarrow]
paleolimbot opened a new pull request, #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428633708
##########
ci/docker/centos7.dockerfile:
##########
@@ -34,13 +29,24 @@ RUN curl -L https://github.com/apache/arrow/archive/refs/tags/apache-arrow-9.0.0
cmake3 ../arrow-apache-arrow-9.0.0/cpp \
-DARROW_JEMALLOC=OFF \
-DARROW_SIMD_LEVEL=NONE \
+ -DARROW_WITH_ZLIB=ON \
-DCMAKE_INSTALL_PREFIX=../arrow && \
cmake3 --build . && \
make install
+RUN python3 -m venv /venv
+RUN source /venv/bin/activate && \
+ pip install --upgrade pip && \
+ pip install build Cython pytest pytest-cython numpy
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
# For R. Note that arrow is not installed (takes too long).
RUN mkdir ~/.R && echo "MAKEFLAGS = -j$(nproc)" > ~/.R/Makevars
-RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr"), repos = "https://cloud.r-project.org")'
+RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr", "bit64"), repos = "https://cloud.r-project.org")'
RUN rm -f ~/.R/Makevars
Review Comment:
(It also should be said that the native build of these images happens once a week and take 10 minutes...the QEMU ones take forever, though)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428534144
##########
ci/docker/centos7.dockerfile:
##########
@@ -20,12 +20,7 @@ ARG NANOARROW_ARCH
FROM --platform=linux/${NANOARROW_ARCH} centos:7
RUN yum install -y epel-release
-RUN yum install -y git gnupg curl R gcc-c++ cmake3 python3-devel
-# On centos7/arm64, cythonize/build is required and fails with cython >= 3
-RUN pip3 install build "Cython < 3.0.0" numpy pytest
-
-RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+RUN yum install -y git gnupg curl R gcc-c++ gcc-gfortran cmake3
# For Arrow C++. Use 9.0.0 because this version works fine with the default gcc
Review Comment:
iirc we also publish an .rpm for arrow c++? Why not use that? (can't wait for centos 7 to reach eol...)
##########
ci/docker/centos7.dockerfile:
##########
@@ -34,13 +29,24 @@ RUN curl -L https://github.com/apache/arrow/archive/refs/tags/apache-arrow-9.0.0
cmake3 ../arrow-apache-arrow-9.0.0/cpp \
-DARROW_JEMALLOC=OFF \
-DARROW_SIMD_LEVEL=NONE \
+ -DARROW_WITH_ZLIB=ON \
-DCMAKE_INSTALL_PREFIX=../arrow && \
cmake3 --build . && \
make install
+RUN python3 -m venv /venv
+RUN source /venv/bin/activate && \
+ pip install --upgrade pip && \
+ pip install build Cython pytest pytest-cython numpy
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
# For R. Note that arrow is not installed (takes too long).
RUN mkdir ~/.R && echo "MAKEFLAGS = -j$(nproc)" > ~/.R/Makevars
-RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr"), repos = "https://cloud.r-project.org")'
+RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr", "bit64"), repos = "https://cloud.r-project.org")'
RUN rm -f ~/.R/Makevars
Review Comment:
This is part of every docker file with the minor difference of installing arrow or not, should that not be moved into a script? + use ppm to install arrow for speed?
##########
ci/docker/ubuntu.dockerfile:
##########
@@ -34,14 +32,17 @@ RUN apt-get install -y -V ca-certificates lsb-release wget && \
apt-get install -y -V libarrow-dev
# For documentation build + Python build
-# Note: sphinx can be unpinned when the interaction between sphinx and breathe
-# has been sorted: https://github.com/sphinx-doc/sphinx/issues/11605
-RUN pip3 install pydata-sphinx-theme "sphinx<7.2.0" breathe build Cython numpy pytest pyarrow
+RUN python3 -m venv --upgrade-deps /venv
+RUN echo "source /venv/bin/activate && pip install pydata-sphinx-theme sphinx breathe build Cython numpy pytest pytest-cython pytest-cov pyarrow" | bash
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN locale-gen en_US.UTF-8 && update-locale en_US.UTF-8
# For R. Note that we install arrow here so that the integration tests for R run
Review Comment:
I don't see arrow R being installed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428532881
##########
ci/docker/ubuntu.dockerfile:
##########
@@ -34,14 +32,17 @@ RUN apt-get install -y -V ca-certificates lsb-release wget && \
apt-get install -y -V libarrow-dev
# For documentation build + Python build
-# Note: sphinx can be unpinned when the interaction between sphinx and breathe
-# has been sorted: https://github.com/sphinx-doc/sphinx/issues/11605
-RUN pip3 install pydata-sphinx-theme "sphinx<7.2.0" breathe build Cython numpy pytest pyarrow
+RUN python3 -m venv --upgrade-deps /venv
+RUN echo "source /venv/bin/activate && pip install pydata-sphinx-theme sphinx breathe build Cython numpy pytest pytest-cython pytest-cov pyarrow" | bash
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN locale-gen en_US.UTF-8 && update-locale en_US.UTF-8
# For R. Note that we install arrow here so that the integration tests for R run
Review Comment:
I don't see arrow R being installed?
##########
ci/docker/ubuntu.dockerfile:
##########
@@ -34,14 +32,17 @@ RUN apt-get install -y -V ca-certificates lsb-release wget && \
apt-get install -y -V libarrow-dev
# For documentation build + Python build
-# Note: sphinx can be unpinned when the interaction between sphinx and breathe
-# has been sorted: https://github.com/sphinx-doc/sphinx/issues/11605
-RUN pip3 install pydata-sphinx-theme "sphinx<7.2.0" breathe build Cython numpy pytest pyarrow
+RUN python3 -m venv --upgrade-deps /venv
+RUN echo "source /venv/bin/activate && pip install pydata-sphinx-theme sphinx breathe build Cython numpy pytest pytest-cython pytest-cov pyarrow" | bash
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN locale-gen en_US.UTF-8 && update-locale en_US.UTF-8
# For R. Note that we install arrow here so that the integration tests for R run
Review Comment:
I don't see arrow R being installed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428633315
##########
ci/docker/centos7.dockerfile:
##########
@@ -34,13 +29,24 @@ RUN curl -L https://github.com/apache/arrow/archive/refs/tags/apache-arrow-9.0.0
cmake3 ../arrow-apache-arrow-9.0.0/cpp \
-DARROW_JEMALLOC=OFF \
-DARROW_SIMD_LEVEL=NONE \
+ -DARROW_WITH_ZLIB=ON \
-DCMAKE_INSTALL_PREFIX=../arrow && \
cmake3 --build . && \
make install
+RUN python3 -m venv /venv
+RUN source /venv/bin/activate && \
+ pip install --upgrade pip && \
+ pip install build Cython pytest pytest-cython numpy
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
# For R. Note that arrow is not installed (takes too long).
RUN mkdir ~/.R && echo "MAKEFLAGS = -j$(nproc)" > ~/.R/Makevars
-RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr"), repos = "https://cloud.r-project.org")'
+RUN R -e 'install.packages(c("blob", "hms", "tibble", "rlang", "testthat", "tibble", "vctrs", "withr", "bit64"), repos = "https://cloud.r-project.org")'
RUN rm -f ~/.R/Makevars
Review Comment:
Definitely! I'm going to punt that to the next time I have to update these, though, since I need to do it for Python too. I think pak can help, as can `Config/Needs/XXX`, but I haven't quite gotten around to that level of automation yet.
I should probably be using Posit Package Manager for the other dependencies, too (although it would only help on x86 I think).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428630067
##########
ci/docker/ubuntu.dockerfile:
##########
@@ -34,14 +32,17 @@ RUN apt-get install -y -V ca-certificates lsb-release wget && \
apt-get install -y -V libarrow-dev
# For documentation build + Python build
-# Note: sphinx can be unpinned when the interaction between sphinx and breathe
-# has been sorted: https://github.com/sphinx-doc/sphinx/issues/11605
-RUN pip3 install pydata-sphinx-theme "sphinx<7.2.0" breathe build Cython numpy pytest pyarrow
+RUN python3 -m venv --upgrade-deps /venv
+RUN echo "source /venv/bin/activate && pip install pydata-sphinx-theme sphinx breathe build Cython numpy pytest pytest-cython pytest-cov pyarrow" | bash
+ENV NANOARROW_PYTHON_VENV "/venv"
+
+# Locale required for R CMD check
+RUN locale-gen en_US.UTF-8 && update-locale en_US.UTF-8
# For R. Note that we install arrow here so that the integration tests for R run
Review Comment:
I moved the comment (it happens a few lines later)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#discussion_r1428630772
##########
ci/docker/centos7.dockerfile:
##########
@@ -20,12 +20,7 @@ ARG NANOARROW_ARCH
FROM --platform=linux/${NANOARROW_ARCH} centos:7
RUN yum install -y epel-release
-RUN yum install -y git gnupg curl R gcc-c++ cmake3 python3-devel
-# On centos7/arm64, cythonize/build is required and fails with cython >= 3
-RUN pip3 install build "Cython < 3.0.0" numpy pytest
-
-RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+RUN yum install -y git gnupg curl R gcc-c++ gcc-gfortran cmake3
# For Arrow C++. Use 9.0.0 because this version works fine with the default gcc
Review Comment:
It looks like the last time this was built for arm64 was Arrow 3.0.0 (!!!). (This gets used on ARM almost entirely because it's how I debug these locally and I'm on M1 Mac). It's just a minimal build so it's not too bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Update docker images [arrow-nanoarrow]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #339:
URL: https://github.com/apache/arrow-nanoarrow/pull/339#issuecomment-1856584124
## [Codecov](https://app.codecov.io/gh/apache/arrow-nanoarrow/pull/339?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`ef92588`)](https://app.codecov.io/gh/apache/arrow-nanoarrow/commit/ef925885166be36bfdbacb35aabe23e0be9f7c47?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 88.04% compared to head [(`ae98afd`)](https://app.codecov.io/gh/apache/arrow-nanoarrow/pull/339?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 88.03%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #339 +/- ##
==========================================
- Coverage 88.04% 88.03% -0.01%
==========================================
Files 72 68 -4
Lines 11370 11269 -101
==========================================
- Hits 10011 9921 -90
+ Misses 1359 1348 -11
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/arrow-nanoarrow/pull/339?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org