You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "lisancao (via GitHub)" <gi...@apache.org> on 2026/10/06 21:13:33 UTC
[PR] Install `pyarrow` and `pandas` in `r-python.template` [spark-docker]
lisancao opened a new pull request, #138:
URL: https://github.com/apache/spark-docker/pull/138
### What changes were proposed in this pull request?
This PR installs PyArrow and pandas in the Python images by adding one line to the `HAVE_PY` block of `r-python.template`:
```dockerfile
PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --no-cache-dir "pyarrow>=18.0.0" "pandas>=2.2.0,<3"; \
```
Published Dockerfiles are unchanged; this applies to the next generated versions (4.2.1, 4.3.0).
Closes #137 (a SPARK JIRA will follow once my JIRA account is approved).
### Why are the changes needed?
Since 4.2.0, regular Python UDFs are Arrow-optimized by default (SPARK-54555). The python3 images ship without PyArrow or pandas, so a plain `@udf` fails on the unmodified image with `ModuleNotFoundError: No module named 'pyarrow'`, and `@pandas_udf` fails with `No module named 'pandas'`.
The floors are PySpark 4.2's own requirements. `PIP_BREAK_SYSTEM_PACKAGES=1` keeps the line working on the Ubuntu 26.04 base proposed in #133, where a plain system pip install is refused (PEP 668); pip 22.0 on jammy ignores it. `<3` because PySpark currently warns that pandas 3.x isn't fully supported, and pip picks 3.x on Python 3.14.
### Does this PR introduce _any_ user-facing change?
Yes. Python images built from this template can run default Python UDFs and pandas UDFs without a custom image. The image grows by about 320 MB uncompressed (1.32 GB to 1.64 GB for the 4.2.0 java21 python3 image in a local build).
### How was this patch tested?
Rendered the template for 4.2.0, built `scala2.13-java21-python3-ubuntu` locally, started a Spark Connect server from the stock and patched images, and ran three UDFs from `pyspark-client==4.2.0` on Python 3.10:
| UDF | stock image | patched image |
|---|---|---|
| `@udf` (default) | `ModuleNotFoundError: No module named 'pyarrow'` | OK |
| `@udf(useArrow=False)` | OK | OK |
| `@pandas_udf` | `ModuleNotFoundError: No module named 'pandas'` | OK |
Also ran the install line on `eclipse-temurin:21-jammy` (Python 3.10) and `eclipse-temurin:21-resolute` (Python 3.14); both got pyarrow 25.0.1 and pandas 2.3.3. Without `PIP_BREAK_SYSTEM_PACKAGES`, resolute refuses the install.
This contribution is my original work, and I license it to the project under the project's open source license.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Install `pyarrow` and `pandas` in `r-python.template` [spark-docker]
Posted by "lisancao (via GitHub)" <gi...@apache.org>.
lisancao closed pull request #138: Install `pyarrow` and `pandas` in `r-python.template`
URL: https://github.com/apache/spark-docker/pull/138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Install `pyarrow` and `pandas` in `r-python.template` [spark-docker]
Posted by "lisancao (via GitHub)" <gi...@apache.org>.
lisancao commented on PR #138:
URL: https://github.com/apache/spark-docker/pull/138#issuecomment-6047388428
Thanks @dongjoon-hyun, so sorry. I didn't realize these images mirror the K8s Dockerfiles in the main repo and were not the right place for the issue/PR.
Since the main repo's Python binding also ships without PyArrow, I can propose this change against the main repo and close the discussion here. Thank you for the correction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Install `pyarrow` and `pandas` in `r-python.template` [spark-docker]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #138:
URL: https://github.com/apache/spark-docker/pull/138#issuecomment-6047401610
If your proposal is accepted in the main repo, you can re-open this too, @lisancao .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Install `pyarrow` and `pandas` in `r-python.template` [spark-docker]
Posted by "lisancao (via GitHub)" <gi...@apache.org>.
lisancao commented on PR #138:
URL: https://github.com/apache/spark-docker/pull/138#issuecomment-6047379679
Thanks @dongjoon-hyun, so sorry. I didn't realize these images mirror the K8s Dockerfiles in the main repo and were not the right place for the issue/PR.
Since the main repo's Python binding also ships without PyArrow, I can propose this change against the main repo and close the discussion here. Thank you for the correction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org