You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/05 07:57:28 UTC
(texera) 01/01: chore(deps): remove unused Python dependencies (#6103)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6103-92b7fc9168d79379a4871b9c9fd80fcb9df75041
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 44ec60d25e63c337424bea8be569ac19b4df3d92
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Sun Jul 5 00:57:06 2026 -0700
chore(deps): remove unused Python dependencies (#6103)
### What changes were proposed in this PR?
Removes three Python packages that nothing uses and moves `ruff` to the
dev-only requirements file:
| Package | File | Why it can go |
| --- | --- | --- |
| `Deprecated==1.2.14` | requirements.txt | Last import removed in
#2847; no installed package requires it |
| `protobuf==7.34.1` | requirements.txt | No `google.protobuf` import
anywhere; betterproto 2.0 runtime does not depend on it (only its
optional extras do) |
| `pybase64==1.3.2` | operator-requirements.txt | Added in #2167, but
the operator templates have always used the stdlib `base64` (see
`ImageUtility.scala` in that PR) — zero references to `pybase64` in the
entire git history |
| `ruff==0.14.7` | requirements.txt → dev-requirements.txt | Lint tool,
not a runtime dependency; keeping it in requirements.txt wrongly puts it
in LICENSE-binary-python |
Follow-up changes required by the move/removals:
- `.github/workflows/build.yml`: the "Lint with Ruff" step now runs
after "Install dev dependencies", since ruff comes from
dev-requirements.txt.
- `amber/LICENSE-binary-python`: dropped the four matching entries.
`wrapt` stays — it is still a hard dependency of `aiobotocore`, not just
of `Deprecated`.
- Added `# Not imported directly: ...` comments to the transitive pins
(`setuptools`, `python-dateutil`, `s3fs`/`aiobotocore`,
`SQLAlchemy`/`pg8000`, `scikit-image`) so they don't look unused in
future audits.
### Any related issues, documentation, discussions?
Closes #6101
### How was this PR tested?
Uninstalled the three packages from the local venv (`pip uninstall -y
protobuf Deprecated pybase64`), then ran the existing test suite and the
same lint commands CI runs:
```
cd amber && pytest -m "not integration" -q
# 551 passed, 1 deselected in 64s
cd amber && ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python
# All checks passed! 190 files already formatted
```
No new tests: dependency-manifest-only change; CI's
`check_binary_deps.py` license gate verifies the LICENSE-binary-python
entries match the installed set.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-fable-5)
Co-authored-by: Claude Fable 5 <no...@anthropic.com>
---
.github/workflows/build.yml | 8 +++++---
amber/LICENSE-binary-python | 4 ----
amber/dev-requirements.txt | 4 ++++
amber/operator-requirements.txt | 3 ++-
amber/requirements.txt | 9 ++++++---
5 files changed, 17 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 449556bab4..115fe1bd10 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -800,9 +800,6 @@ jobs:
run: psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
env:
PGPASSWORD: postgres
- - name: Lint with Ruff
- run: |
- cd amber && ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python
- name: Install dev dependencies
# Test-only deps live in amber/dev-requirements.txt and are
# installed after the LICENSE-binary snapshot above so they never
@@ -811,6 +808,11 @@ jobs:
run: |
python -m pip install uv
if [ -f amber/dev-requirements.txt ]; then uv pip install --system -r amber/dev-requirements.txt; fi
+ - name: Lint with Ruff
+ # ruff comes from dev-requirements.txt, so this must run after the
+ # dev-dependency install above.
+ run: |
+ cd amber && ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python
- name: Install protoc
# Version pinned in bin/protoc-version.txt.
run: |
diff --git a/amber/LICENSE-binary-python b/amber/LICENSE-binary-python
index afd33d9547..a30ff7976a 100644
--- a/amber/LICENSE-binary-python
+++ b/amber/LICENSE-binary-python
@@ -253,7 +253,6 @@ Python packages:
- betterproto==2.0.0b7
- cachetools==6.2.6
- charset-normalizer==3.4.7
- - deprecated==1.2.14
- filelock==3.29.4
- fonttools==4.63.0
- fs==2.4.16
@@ -277,7 +276,6 @@ Python packages:
- pyyaml==6.0.3
- readerwriterlock==1.0.9
- rich==14.3.4
- - ruff==0.14.7
- scramp==1.4.9
- six==1.17.0
- sqlalchemy==2.0.37
@@ -312,7 +310,6 @@ Python packages:
- numpy==2.1.0
- pandas==2.2.3
- pg8000==1.31.5
- - protobuf==7.34.1
- psutil==5.9.0
- s3fs==2025.9.0
- scikit-image==0.25.2
@@ -332,7 +329,6 @@ Python packages:
- imageio==2.37.3
- praw==7.6.1
- prawcore==2.4.0
- - pybase64==1.3.2
- pygments==2.20.0
- update-checker==1.0.0
- wrapt==1.17.3
diff --git a/amber/dev-requirements.txt b/amber/dev-requirements.txt
index 3ac549283d..82e98b7cb4 100644
--- a/amber/dev-requirements.txt
+++ b/amber/dev-requirements.txt
@@ -29,6 +29,10 @@ pytest-timeout==2.4.0
# Codecov's Phase 1 upload.
pytest-cov==7.1.0
+# Linter/formatter for amber's Python sources; run by CI's "Lint with
+# Ruff" step and bin/fix-format.sh.
+ruff==0.14.7
+
# protoc plugin for bin/python-proto-gen.sh; runtime needs only the
# base `betterproto` in requirements.txt.
betterproto[compiler]==2.0.0b7
diff --git a/amber/operator-requirements.txt b/amber/operator-requirements.txt
index 99de2e11a3..b8f7d1f057 100644
--- a/amber/operator-requirements.txt
+++ b/amber/operator-requirements.txt
@@ -19,7 +19,6 @@ wordcloud==1.9.3
plotly==5.24.1
praw==7.6.1
pillow==12.2.0
-pybase64==1.3.2
# Pin torch to the CPU wheel on Linux x86_64 to avoid the NVIDIA CUDA deps.
--extra-index-url https://download.pytorch.org/whl/cpu
@@ -28,4 +27,6 @@ torch==2.12.0 ; platform_system != "Linux" or platform_machine != "x86_64"
scikit-learn==1.5.0
transformers==5.3.0
+# Not imported by any operator template: plotly's create_ternary_contour
+# (ternary contour operator, #4193) requires scikit-image at runtime.
scikit-image==0.25.2
diff --git a/amber/requirements.txt b/amber/requirements.txt
index caf9b6bedc..2067770ffa 100644
--- a/amber/requirements.txt
+++ b/amber/requirements.txt
@@ -15,30 +15,33 @@
# specific language governing permissions and limitations
# under the License.
+# Not imported directly: `fs` needs pkg_resources at import time (#4199).
setuptools==80.10.2
numpy==2.1.0
pandas==2.2.3
-ruff==0.14.7
loguru==0.7.0
pyarrow==23.0.1
+# Not imported directly: runtime dependency of betterproto and pg8000.
python-dateutil==2.8.2
-protobuf==7.34.1
betterproto==2.0.0b7
pampy==0.3.0
overrides==7.4.0
typing_extensions==4.14.1
-Deprecated==1.2.14
fs==2.4.16
bidict==0.22.0
cached_property==1.5.2
psutil==5.9.0
tzlocal==2.1
+# Not imported directly: s3fs (with aiobotocore) backs pyiceberg's
+# fsspec-based S3 FileIO (#4272).
s3fs==2025.9.0
aiobotocore==2.25.1
botocore==1.40.53
pyiceberg==0.11.1
readerwriterlock==1.0.9
tenacity==8.5.0
+# Not imported directly: pyiceberg's SqlCatalog needs SQLAlchemy with the
+# pg8000 driver (postgresql+pg8000:// connection string in iceberg_utils.py).
SQLAlchemy==2.0.37
pg8000==1.31.5
pympler==1.1