You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "tuhaihe (via GitHub)" <gi...@apache.org> on 2025/11/12 06:34:48 UTC
[PR] Fix: Python build dependencies installation [cloudberry]
tuhaihe opened a new pull request, #1434:
URL: https://github.com/apache/cloudberry/pull/1434
Improve wheel and cython dependency management in gpMgmt/bin/Makefile to handle Ubuntu 24.04's PEP 668 restrictions while maintaining compatibility with Rocky Linux and older Ubuntu versions.
Changes:
- Split wheel and cython dependency checks into separate commands
- Add fallback to --break-system-packages flag for Ubuntu 24.04+
- Only install dependencies if not already present in the system
- Maintain backward compatibility with existing build environments
This resolves build failures on Ubuntu 24.04 where pip install --user is restricted by default, while preserving the existing behavior on Rocky Linux 8/9 and Ubuntu 20.04/22.04 systems.
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #1433
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe merged PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3520278906
We also need to update the installation docs to install the required dependencies before building Cloudberry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3853557130
> Hi @leborchuk could you please help take a look at this PR? Thanks!
Ah, a very unusual code. Why would anyone even need to install wheel in the Makefile? For me is Ok to break system packages here, but I should understand why we need install it here, why just not to exit with error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#discussion_r2726603476
##########
gpMgmt/bin/Makefile:
##########
@@ -111,8 +111,19 @@ download-python-deps:
else \
echo "PyGreSQL-$(PYGRESQL_VERSION).tar.gz already exists, skipping download"; \
fi
- # Install wheel and cython for PyYAML building
- pip3 install --user wheel "cython<3.0.0"
+ # Install wheel and cython for PyYAML building (only if not exists)
+ @if python3 -c "import wheel" >/dev/null 2>&1; then \
+ echo "wheel already exists, skipping installation"; \
+ else \
+ echo "Installing wheel..."; \
+ pip3 install --user wheel 2>/dev/null || pip3 install --user --break-system-packages wheel; \
Review Comment:
Hi @gfphoenix78, thanks for raising this — the concern is valid.
We plan to update the docs ([[cloudberry-site#337](https://github.com/apache/cloudberry-site/pull/337)](https://github.com/apache/cloudberry-site/pull/337)) to recommend installing the required Python dependencies **before** building, so in most cases this step will be skipped entirely.
In the Makefile, `pip3 install --user --break-system-packages` is intentionally used **only as a fallback**, and only when the standard `--user` installation is blocked on Ubuntu 24.04 due to PEP 668. The dependencies involved here (`wheel`, `cython`) are **build-time only** and are installed into the user site, not the system Python environment.
That said, we agree this is not ideal long-term. A cleaner approach would be to isolate build-time Python dependencies (e.g. via a dedicated venv), which we can consider as a follow-up improvement. This change focuses on unblocking the Ubuntu 24.04 build while keeping backward compatibility with existing environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3520916661
Could you explain why we need flag `--break-system-packages`? Where are these py packages installed, system library path for python? Is it possible to install them under `$GPHOME/lib`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on code in PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#discussion_r2711430096
##########
gpMgmt/bin/Makefile:
##########
@@ -111,8 +111,19 @@ download-python-deps:
else \
echo "PyGreSQL-$(PYGRESQL_VERSION).tar.gz already exists, skipping download"; \
fi
- # Install wheel and cython for PyYAML building
- pip3 install --user wheel "cython<3.0.0"
+ # Install wheel and cython for PyYAML building (only if not exists)
+ @if python3 -c "import wheel" >/dev/null 2>&1; then \
+ echo "wheel already exists, skipping installation"; \
+ else \
+ echo "Installing wheel..."; \
+ pip3 install --user wheel 2>/dev/null || pip3 install --user --break-system-packages wheel; \
Review Comment:
`--break-system-packages` may hide possible problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3857821233
Thanks for the review and feedback!
I agree that installing Python dependencies implicitly in the build system is not ideal in the long term, especially with PEP 668 and distro packaging policies.
For this PR, I’d like to keep the current behavior for compatibility reasons (it is consistent with the existing build logic and avoids breaking current users and CI environments). This change mainly addresses the Ubuntu 24.04 build failure while preserving backward compatibility.
As a follow-up, I plan to propose a cleaner approach:
- Remove implicit pip installation from the build process.
- Recommend users to install Python dependencies explicitly via `pip3 install -r python-dependencies.txt` (or via venv / distro packages).
- Update the build documentation accordingly and make the build fail fast if required Python modules are missing.
This would align better with reproducible builds and packaging best practices, but it should be done in a phased manner to avoid disrupting existing workflows.
How about this? @leborchuk @gfphoenix78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on code in PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#discussion_r2711655952
##########
gpMgmt/bin/Makefile:
##########
@@ -111,8 +111,19 @@ download-python-deps:
else \
echo "PyGreSQL-$(PYGRESQL_VERSION).tar.gz already exists, skipping download"; \
fi
- # Install wheel and cython for PyYAML building
- pip3 install --user wheel "cython<3.0.0"
+ # Install wheel and cython for PyYAML building (only if not exists)
+ @if python3 -c "import wheel" >/dev/null 2>&1; then \
+ echo "wheel already exists, skipping installation"; \
+ else \
+ echo "Installing wheel..."; \
+ pip3 install --user wheel 2>/dev/null || pip3 install --user --break-system-packages wheel; \
Review Comment:
Is any possible approach recommended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3851198362
Hi @leborchuk could you please help take a look at this PR? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3858376758
> Thanks for the review and feedback!
>
> I agree that installing Python dependencies implicitly in the build system is not ideal in the long term, especially with PEP 668 and distro packaging policies.
>
> For this PR, I’d like to keep the current behavior for compatibility reasons (it is consistent with the existing build logic and avoids breaking current users and CI environments). This change mainly addresses the Ubuntu 24.04 build failure while preserving backward compatibility.
>
> As a follow-up, I plan to propose a cleaner approach:
>
> * Remove implicit pip installation from the build process.
> * Recommend users to install Python dependencies explicitly via `pip3 install -r python-dependencies.txt` (or via venv / distro packages).
> * Update the build documentation accordingly and make the build fail fast if required Python modules are missing.
>
> This would align better with reproducible builds and packaging best practices, but it should be done in a phased manner to avoid disrupting existing workflows.
>
> How about this? @leborchuk @gfphoenix78
Yes, that's normal. Here we try to solve the issue that code does not work in Ubuntu 24.04.
Of course, the whole approach should be reconsidered, we should just instruct script user how to prepare environment or get the prepared environment for him. But do not change the environment yourself. It'd be better to change it in another PR. Do not confuse doing a big refactoring with supporting Ubuntu 24.04 altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Fix: Python build dependencies installation [cloudberry]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #1434:
URL: https://github.com/apache/cloudberry/pull/1434#issuecomment-3525135689
> Could you explain why we need flag `--break-system-packages`? Where are these py packages installed, system library path for python? Is it possible to install them under `$GPHOME/lib`?
Hi @gfphoenix78 Thanks for the review!
The `--break-system-packages` flag is needed for Ubuntu 24.04 and newer, which enforce [PEP 668](https://peps.python.org/pep-0668/)
restrictions. This policy prevents `pip install --user` from modifying system-managed Python environments. Without this flag, pip will refuse to install packages like `wheel` or `cython`, causing build failures.
Regarding the installation location: when using `--user`, the packages are installed under the user’s local site directory (e.g., ~/.local/lib/python3.x/site-packages), not under the system library path. This avoids conflicts with system Python.
Installing them into `$GPHOME/lib` would require customizing `PYTHONPATH` and ensuring consistent Python environments across all build and runtime contexts, which might introduce complexity and compatibility issues. The current approach minimizes intrusion into the system and maintains backward compatibility with other platforms like Rocky Linux and older Ubuntu versions.
In short, `--break-system-packages` is only used as a fallback on Ubuntu 24.04+, while other systems continue using the existing safe paths.
We will also need to update the build docs to guide the users to install the needed packages from the software source. If these packages are detected as installed, then will skip the installation during the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org