You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/02/17 13:59:04 UTC
[PR] Migrate Python usage to uv workspace [datafusion]
adriangb opened a new pull request, #20414:
URL: https://github.com/apache/datafusion/pull/20414
## Summary
- Replace three independent `requirements.txt` files with a uv workspace (`benchmarks`, `dev`, `docs` projects)
- Single `uv.lock` lockfile for reproducible dependency resolution
- Simplify `bench.sh` by removing all ad-hoc venv/pip logic in favor of `uv run`
## Test plan
- [ ] `uv sync` resolves all deps from repo root
- [ ] `uv run --project benchmarks python3 benchmarks/compare.py` works
- [ ] `uv run --project docs sphinx-build docs/source docs/build` builds docs
- [ ] Run a benchmark from `bench.sh` that uses Python (e.g., h2o data gen or compare flow)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#discussion_r2819706516
##########
.github/workflows/docs_pr.yaml:
##########
@@ -44,16 +44,10 @@ jobs:
with:
submodules: true
fetch-depth: 1
- - name: Setup Python
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- with:
- python-version: "3.12"
+ - name: Setup uv
+ uses: astral-sh/setup-uv@f0ec1fc3b38f5e7cd731bb6ce540c5af426746bb # v6.1.0
Review Comment:
No, but it is generally encouraged to lock to commits in GHA: a commit provides an immutable reference to "safe" code. A tag is mutable. If a malicious actor gains control of an action repository they can upload a new v6 and infect everyone. If everyone is pinned to the commit they can't force the malicious code into everyone's CI unless you opt in by updating the hash. TLDR because there are no lockfiles for CI and because CI is a critical vector for supply chain attacks it's best to pin to a hash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#discussion_r2833303783
##########
docs/source/user-guide/example-usage.md:
##########
@@ -29,7 +29,7 @@ Find latest available Datafusion version on [DataFusion's
crates.io] page. Add the dependency to your `Cargo.toml` file:
```toml
-datafusion = "52.0.0"
+datafusion = "52.1.0"
Review Comment:
@timsaucer this is a drive by that the working version update script did 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#discussion_r2818270003
##########
.github/workflows/docs.yaml:
##########
@@ -40,17 +40,11 @@ jobs:
ref: asf-site
path: asf-site
- - name: Setup Python
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- with:
- python-version: "3.12"
+ - name: Setup uv
+ uses: astral-sh/setup-uv@f0ec1fc3b38f5e7cd731bb6ce540c5af426746bb # v6.1.0
Review Comment:
Same as below, do we need to specific a specific sha here?
##########
pyproject.toml:
##########
@@ -0,0 +1,2 @@
+[tool.uv.workspace]
+members = ["benchmarks", "dev", "docs"]
Review Comment:
Is there a reason to use three `pyproject.toml` files instead of just one here at the root and three dependency groups?
##########
.github/workflows/docs_pr.yaml:
##########
@@ -44,16 +44,10 @@ jobs:
with:
submodules: true
fetch-depth: 1
- - name: Setup Python
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- with:
- python-version: "3.12"
+ - name: Setup uv
+ uses: astral-sh/setup-uv@f0ec1fc3b38f5e7cd731bb6ce540c5af426746bb # v6.1.0
Review Comment:
Is the specific commit a requirement? I think `astral-sh/setup-uv@v6` is pretty stable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#discussion_r2819707382
##########
pyproject.toml:
##########
@@ -0,0 +1,2 @@
+[tool.uv.workspace]
+members = ["benchmarks", "dev", "docs"]
Review Comment:
Dunno, maybe personal preference, I feel it better mirrors the cargo workspace structure and since we use cargo workspaces here I felt it fit better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#issuecomment-3934863737
> > From the PR description it isn't clear to me if you've run all of those commands to verify they work as expected. Maybe just update the description if they've all been manually verified.
>
> This is the only thing holding me up from hitting approve
Ah sorry missed that. And it was a good call, there were some minor issues (falsa not properly declaring it's deps)...
I've now run all of the commands manually and can confirm they work (as far as I can tell they do what they should be and I get no dep errors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #20414:
URL: https://github.com/apache/datafusion/pull/20414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#issuecomment-3927288548
@timsaucer let me know if my responses clear up the issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#issuecomment-3927330462
> From the PR description it isn't clear to me if you've run all of those commands to verify they work as expected. Maybe just update the description if they've all been manually verified.
This is the only thing holding me up from hitting approve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Migrate Python usage to uv workspace [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20414:
URL: https://github.com/apache/datafusion/pull/20414#issuecomment-3935894591
Thank you, Tim! I’ll send this to merge and if any issues pop up or there are ergonomic improvement I will handle as a follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org