You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "autophagy (via GitHub)" <gi...@apache.org> on 2025/01/23 15:11:44 UTC
[PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
autophagy opened a new pull request, #26065:
URL: https://github.com/apache/flink/pull/26065
## What is the purpose of the change
Given the issues raised in https://issues.apache.org/jira/browse/LEGAL-688, we've decided to move away from using conda. `conda` is being used in Flink in the following ways:
- Setting up PyFlink developer environments.
- Setting up a range of environments with different Python versions (3.8, 3.9, 3.10, 3.11) to both validate the Flink apis over those versions, and to build Python wheels for those versions.
- Setting up a Python environment for use in building the binary releases.
- Setting up a Python environment or the PyFlink, PyFlink Yarn and PyFlink Kubernetes end-to-end tests.
To summarise, conda is largely being used for the creation of Python virtual environments for both developers and CI. This PR replaces conda with [uv](https://docs.astral.sh/uv/), a fast Python package and virtual environment manager. In the cases where conda was being used to manage non-python packages (for example, installing the `zip` binary in the PyFlink yarn tests) this has instead been changed to use the package manager native to the environment (`apt`, in this particular case).
## Brief change log
- Replaced `conda` with `uv` for managing the creation of Python environments and installing Python packages for PyFlink in both testing, wheel building and developer contexts.
- Added `tox-uv` so that `tox` can manage the Python environments it uses via `uv`.
## Verifying this change
This change is already covered by existing tests, such as:
- PyFlink end-to-end tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #26065:
URL: https://github.com/apache/flink/pull/26065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on PR #26065:
URL: https://github.com/apache/flink/pull/26065#issuecomment-2616161221
@dawidwys I will take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #26065:
URL: https://github.com/apache/flink/pull/26065#discussion_r1951160748
##########
flink-python/README.md:
##########
@@ -43,30 +43,30 @@ PyFlink depends on the following libraries to execute the above script:
2. setuptools (>=37.0.0)
3. pip (>=20.3)
-### Running Test Cases
+### Running Test Cases
-Currently, we use conda and tox to verify the compatibility of the Flink Python API for multiple versions of Python and will integrate some useful plugins with tox, such as flake8.
+Currently, we use tox and to verify the compatibility of the Flink Python API for multiple versions of Python and will integrate some useful plugins with tox, such as flake8.
We can enter the directory where this README.md file is located and run test cases by executing
```
./dev/lint-python.sh
```
-To use your system conda environment, you can set `FLINK_CONDA_HOME` variable:
+To use your system uv environment, you can set `FLINK_UV_HOME` variable:
```shell
-export FLINK_CONDA_HOME=$(dirname $(dirname $CONDA_EXE))
+export FLINK_UV_HOME=$(dirname $(dirname $(which uv)))
```
Create a virtual environment:
```shell
-conda create -n pyflink_38 python=3.8
+uv venv pyflink_38 python=3.8
Review Comment:
I tried this, but that fails with
```
error: unexpected argument 'python=3.8' found
```
So I Google'd and find out that it should be:
```
uv venv --python=3.8 pyflink_38
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26065:
URL: https://github.com/apache/flink/pull/26065#issuecomment-2610441079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa3f31c9ec8490e338995066c5437eada7fe827",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfa3f31c9ec8490e338995066c5437eada7fe827",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa3f31c9ec8490e338995066c5437eada7fe827 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36900][python] Migrate from conda to uv for managing Python environments for PyFlink [flink]
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on PR #26065:
URL: https://github.com/apache/flink/pull/26065#issuecomment-2612662959
@autophagy Have you seen the CI failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org