You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "TravisMai (via GitHub)" <gi...@apache.org> on 2026/10/20 06:05:28 UTC
[PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
TravisMai opened a new pull request, #3492:
URL: https://github.com/apache/sedona/pull/3492
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
## Is this PR related to a ticket?
- Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3080
## What changes were proposed in this PR?
1. Hook priorities for parallel execution: Independent, read-only linters are grouped into priority aliases named after the language/domain they check (lint-python, lint-yaml, lint-shell, lint-actions, lint-makefile, lint-json, lint-toml, lint-xml, lint-docs, lint-rst, lint-generic). All of these resolve to the same numeric priority, so prek runs them concurrently instead of one at a time — e.g. bandit, codespell, gitleaks, shellcheck, yamllint, markdownlint, and actionlint all run in parallel now.
Hooks that rewrite file contents (formatters, license insertion, tab fixers, etc.) are kept at explicit, strictly increasing integer priorities that preserve today's execution order, rather than relying on prek's implicit position-based priority. This avoids correctness issues: several mutating hooks touch overlapping files (e.g. insert-license and clang-format both rewrite .c files), and prek only guarantees correct results when hooks sharing a priority don't mutate the same files.
2. Built-in Fast Hooks: Bumped minimum_prek_version to 0.4.11 (the version that introduced reusable priority aliases). No config changes were needed for the Built-in Fast Hooks feature itself: hook IDs that already match one of prek's native Rust implementations (check-yaml, check-json, trailing-whitespace, end-of-file-fixer, detect-private-key, etc.) are automatically run via prek's fast path once the minimum version requirement is met.
Also updated docs/setup/compile.md to document the new priority scheme.
## How was this patch tested?
- prek validate-config .pre-commit-config.yaml passes.
- prek run --all-files passes all 70 hooks with no failures, in the same effective order as before the change (mutating hooks sequential, independent linters now running concurrently — confirmed via increased CPU utilization during the run).
## Did this PR include necessary documentation updates?
- Yes, I have updated the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4330947892
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,46 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep increasing integer priorities instead, since
+# some of them touch the same files and prek only parallelizes safely
+# when hooks sharing a priority don't mutate the same files. Those
+# integers are spaced by 10 (0, 10, 20, ...) so a new hook can be slotted
+# in later without renumbering every hook after it.
+priorities:
+ lint-actions: 1000
+ lint-docs: 1000
+ lint-generic: 1000
+ lint-json: 1000
+ lint-makefile: 1000
+ lint-python: 1000
+ lint-rst: 1000
+ lint-shell: 1000
+ lint-toml: 1000
+ lint-xml: 1000
+ lint-yaml: 1000
Review Comment:
Fixed in fb87d538ff2cd6dc19226a74d1c4a5d55d5632d9: lint-actions now anchors the shared value (&lint_priority 1000) and every other lint-* alias references it (*lint_priority), so there's a single source of truth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4330947892
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,46 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep increasing integer priorities instead, since
+# some of them touch the same files and prek only parallelizes safely
+# when hooks sharing a priority don't mutate the same files. Those
+# integers are spaced by 10 (0, 10, 20, ...) so a new hook can be slotted
+# in later without renumbering every hook after it.
+priorities:
+ lint-actions: 1000
+ lint-docs: 1000
+ lint-generic: 1000
+ lint-json: 1000
+ lint-makefile: 1000
+ lint-python: 1000
+ lint-rst: 1000
+ lint-shell: 1000
+ lint-toml: 1000
+ lint-xml: 1000
+ lint-yaml: 1000
Review Comment:
Fixed in d08b9ee7ab3f3e9cbc43ec63f4a899ad9bef3e4b: lint-actions now anchors the shared value (&lint_priority 1000) and every other lint-* alias references it (*lint_priority), so there's a single source of truth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4323184460
##########
.pre-commit-config.yaml:
##########
@@ -377,107 +433,137 @@ repos:
args: [--config=.github/linters/zizmor.yml]
files: ^\.github/workflows/.*$
types: [yaml]
+ priority: lint-actions
- repo: https://github.com/pre-commit/pygrep-hooks
rev: 3a6eb0fadf60b3cccfd80bad9dbb6fae7e47b316 # frozen: v1.10.0
hooks:
- id: python-check-mock-methods
name: run check for not-real mock methods
description: Prevent common mistakes of assert mck.not_called(), assert mck.called_once_with(...) and mck.assert_called
+ priority: lint-python
- id: python-no-eval
name: run check for eval()
description: A quick check for the eval() built-in function
exclude: ^python/tests/flink/test_flink_registration\.py$
+ priority: lint-python
- id: python-no-log-warn
name: run check for use logger.warning(
description: A quick check for the deprecated .warn() method of python loggers
+ priority: lint-python
- id: python-use-type-annotations
name: run check for type annotations not comments
description: Enforce that python3.6+ type annotations are used instead of type comments
+ priority: lint-python
- id: rst-backticks
name: run rst-backticks
description: detect common mistake of using single backticks when writing rst
+ priority: lint-rst
- id: rst-directive-colons
name: run rst-directive-colons
description: detect mistake of rst directive not ending with double colon or space before the double colon
+ priority: lint-rst
- id: rst-inline-touching-normal
name: run rst-inline-touching-normal
description: detect mistake of inline code touching normal text in rst
+ priority: lint-rst
- id: text-unicode-replacement-char
name: run check for no unicode replacement char
description: Forbid files which have a UTF-8 Unicode replacement character
+ priority: lint-generic
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-ast
name: run check-ast
description: check Python files for syntax errors
+ priority: lint-python
- id: check-builtin-literals
name: run check-builtin-literals
description: check Python files for proper use of built-in literals
+ priority: lint-python
- id: check-case-conflict
name: run check-case-conflict
description: check for case conflicts in file names
+ priority: lint-generic
- id: check-executables-have-shebangs
name: run check-executables-have-shebangs
description: check that executable scripts have shebang lines
+ priority: lint-generic
- id: check-illegal-windows-names
name: run check-illegal-windows-names
description: check for Windows-illegal file names
+ priority: lint-generic
- id: check-json
name: run check-json
description: check JSON files for syntax errors
exclude: ^docs/usecases/data/testPolygon\.json$|^spark/common/src/test/resources/.*\.json$
+ priority: lint-json
- id: check-merge-conflict
name: run check-merge-conflict
description: check for merge conflict markers
+ priority: lint-generic
- id: check-shebang-scripts-are-executable
name: run check-shebang-scripts-are-executable
description: check that scripts with shebangs are executable
+ priority: lint-generic
- id: check-toml
name: run check-toml
description: check TOML files for syntax errors
+ priority: lint-toml
- id: check-vcs-permalinks
name: run check-vcs-permalinks
description: ensures that links to vcs websites are permalinks
+ priority: lint-generic
- id: check-xml
name: run check-xml
description: attempts to load all xml files to verify syntax
+ priority: lint-xml
- id: check-yaml
name: run check-yaml
description: attempts to load all yaml files to verify syntax
exclude: ^mkdocs\.yml$
+ priority: lint-yaml
- id: debug-statements
name: run debug-statements
description: check for debugger imports and py37+ `breakpoint()` calls in python source.
+ priority: lint-python
- id: destroyed-symlinks
name: run destroyed-symlinks
description: detects symlinks which are changed to regular files with a content of a path which that symlink was pointing to
+ priority: lint-generic
- id: detect-aws-credentials
name: run detect-aws-credentials
description: checks for the existence of AWS secrets that you have set up with the AWS CLI
args: [--allow-missing-credentials]
+ priority: lint-generic
- id: detect-private-key
name: run detect-private-key
description: checks for the existence of private keys
+ priority: lint-generic
- id: end-of-file-fixer
name: run end-of-file-fixer
description: makes sure files end in a newline and only a newline
exclude: \.svg$|^docs/image|^spark/common/src/test/resources
+ priority: 310
Review Comment:
"same effective order as before" was inaccurate. Mutators now run as one sequential block, followed by all lint-* hooks running concurrently, which changes the previous file-position interleaving. Verified this doesn't change any hook's pass/fail outcome. Corrected the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4322311151
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,44 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
Review Comment:
Fixed in 824819ddd223fdd7edfa7ed62639459ab5d0e450. Bumped the dev dependency in pyproject.toml from prek>=0.4.5 to prek>=0.4.11 to match minimum_prek_version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4323182423
##########
.pre-commit-config.yaml:
##########
@@ -377,107 +433,137 @@ repos:
args: [--config=.github/linters/zizmor.yml]
files: ^\.github/workflows/.*$
types: [yaml]
+ priority: lint-actions
- repo: https://github.com/pre-commit/pygrep-hooks
rev: 3a6eb0fadf60b3cccfd80bad9dbb6fae7e47b316 # frozen: v1.10.0
hooks:
- id: python-check-mock-methods
name: run check for not-real mock methods
description: Prevent common mistakes of assert mck.not_called(), assert mck.called_once_with(...) and mck.assert_called
+ priority: lint-python
- id: python-no-eval
name: run check for eval()
description: A quick check for the eval() built-in function
exclude: ^python/tests/flink/test_flink_registration\.py$
+ priority: lint-python
- id: python-no-log-warn
name: run check for use logger.warning(
description: A quick check for the deprecated .warn() method of python loggers
+ priority: lint-python
- id: python-use-type-annotations
name: run check for type annotations not comments
description: Enforce that python3.6+ type annotations are used instead of type comments
+ priority: lint-python
- id: rst-backticks
name: run rst-backticks
description: detect common mistake of using single backticks when writing rst
+ priority: lint-rst
- id: rst-directive-colons
name: run rst-directive-colons
description: detect mistake of rst directive not ending with double colon or space before the double colon
+ priority: lint-rst
- id: rst-inline-touching-normal
name: run rst-inline-touching-normal
description: detect mistake of inline code touching normal text in rst
+ priority: lint-rst
- id: text-unicode-replacement-char
name: run check for no unicode replacement char
description: Forbid files which have a UTF-8 Unicode replacement character
+ priority: lint-generic
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-ast
name: run check-ast
description: check Python files for syntax errors
+ priority: lint-python
- id: check-builtin-literals
name: run check-builtin-literals
description: check Python files for proper use of built-in literals
+ priority: lint-python
- id: check-case-conflict
name: run check-case-conflict
description: check for case conflicts in file names
+ priority: lint-generic
- id: check-executables-have-shebangs
name: run check-executables-have-shebangs
description: check that executable scripts have shebang lines
+ priority: lint-generic
- id: check-illegal-windows-names
name: run check-illegal-windows-names
description: check for Windows-illegal file names
+ priority: lint-generic
- id: check-json
name: run check-json
description: check JSON files for syntax errors
exclude: ^docs/usecases/data/testPolygon\.json$|^spark/common/src/test/resources/.*\.json$
+ priority: lint-json
Review Comment:
"same effective order as before" was inaccurate. Mutators now run as one sequential block, followed by all lint-* hooks running concurrently, which changes the previous file-position interleaving. Verified this doesn't change any hook's pass/fail outcome. Corrected the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4330961470
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,46 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep increasing integer priorities instead, since
+# some of them touch the same files and prek only parallelizes safely
+# when hooks sharing a priority don't mutate the same files. Those
+# integers are spaced by 10 (0, 10, 20, ...) so a new hook can be slotted
+# in later without renumbering every hook after it.
Review Comment:
Fixed in fb87d538ff2cd6dc19226a74d1c4a5d55d5632d9: clarified that priority 0 is the meta hooks' shared value, and the mutating-hook sequence actually starts at 10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4322220011
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,44 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
Review Comment:
**[P2] Raise the development dependency alongside the minimum version**
The config now requires `prek>=0.4.11`, but the root `pyproject.toml` still specifies `prek>=0.4.5`. Contributors whose existing local `uv.lock` selects 0.4.5–0.4.10 retain that version after `uv sync --all-groups`, so the documented check workflow fails before running hooks. Please update the development dependency to `prek>=0.4.11` in the same change.
Verified locally in an isolated directory using the exact project dependencies: after seeding the lockfile with `uv lock --upgrade-package prek==0.4.5`, `uv sync --all-groups` retained 0.4.5. `uv run prek validate-config` accepted the base config but rejected this PR’s config with “Required minimum prek version `0.4.11` is greater than current version `0.4.5`”. The new config validates successfully with prek 0.4.11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4330952783
##########
docs/setup/compile.md:
##########
@@ -182,6 +182,8 @@ make docsbuild
This guide explains how Apache Sedona manages local code quality checks, formatting, and linting. To ensure a fast and consistent developer experience, we combine `uv` (a lightning-fast Python package and environment manager) with `prek` (our internal pre-commit wrapper framework) and expose everything via a simple, unified `Makefile`.
+`.pre-commit-config.yaml` assigns the independent, read-only linters a `priority` named after the language or domain they check (e.g. `lint-python`, `lint-yaml`, `lint-shell`, `lint-actions`, `lint-makefile`, `lint-rst`). All of these aliases resolve to the same priority value, so `prek` runs them concurrently instead of one at a time — linting different languages in parallel, as described in [issue #3080](https://github.com/apache/sedona/issues/3080). Hooks that rewrite file contents (formatters, license insertion, tab fixers, etc.) instead get explicit, strictly increasing numeric `priority` values that preserve today's sequential execution order, since several of them touch overlapping files and `prek` only guarantees correct results when hooks sharing a priority don't mutate the same files. For hook IDs that match one of `prek`'s [Built-in Fast Hooks](https://prek.j178.dev/builtin/) (such as `check-yaml`, `check-json`, `trailing-whitespace`, and `end-of-file-fixer`), `prek` au
tomatically swaps in its native Rust implementation instead of spinning up a Python environment, no config changes required.
Review Comment:
Fixed in d08b9ee7ab3f3e9cbc43ec63f4a899ad9bef3e4b: split into a short intro plus three bullets (parallel lint aliases / sequential mutators / Built-in Fast Hooks). markdownlint passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "jbampton (via GitHub)" <gi...@apache.org>.
jbampton commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4322384032
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,44 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep strictly increasing integer priorities instead,
+# since some of them touch the same files and prek only parallelizes
+# safely when hooks sharing a priority don't mutate the same files.
+priorities:
+ lint-actions: 100
+ lint-docs: 100
+ lint-generic: 100
+ lint-json: 100
+ lint-makefile: 100
+ lint-python: 100
+ lint-rst: 100
+ lint-shell: 100
+ lint-toml: 100
+ lint-xml: 100
+ lint-yaml: 100
repos:
- repo: meta
hooks:
- id: identity
name: run identity check
description: a simple hook which prints all arguments passed to it, useful for debugging
+ priority: 0
- id: check-hooks-apply
name: run check hooks apply
description: check that all the hooks apply to the repository
+ priority: 1
Review Comment:
Can both of these hooks have the same priority ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4326220023
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,46 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep increasing integer priorities instead, since
+# some of them touch the same files and prek only parallelizes safely
+# when hooks sharing a priority don't mutate the same files. Those
+# integers are spaced by 10 (0, 10, 20, ...) so a new hook can be slotted
+# in later without renumbering every hook after it.
+priorities:
+ lint-actions: 1000
+ lint-docs: 1000
+ lint-generic: 1000
+ lint-json: 1000
+ lint-makefile: 1000
+ lint-python: 1000
+ lint-rst: 1000
+ lint-shell: 1000
+ lint-toml: 1000
+ lint-xml: 1000
+ lint-yaml: 1000
Review Comment:
All `lint-*` aliases map to the same numeric value, which is easy to accidentally diverge over time. To keep a single source of truth while preserving the descriptive names, consider using YAML anchors/aliases (e.g., anchor `1000` once, then reference it for each key) so changing the shared lint priority only needs one edit.
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,46 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep increasing integer priorities instead, since
+# some of them touch the same files and prek only parallelizes safely
+# when hooks sharing a priority don't mutate the same files. Those
+# integers are spaced by 10 (0, 10, 20, ...) so a new hook can be slotted
+# in later without renumbering every hook after it.
Review Comment:
The wording here can be read as implying a single strictly increasing sequence starting at `0`, but `0` is also used for multiple non-mutating/meta hooks. Consider clarifying the comment to distinguish the mutating-hook numeric sequence (e.g., starting at `10` and spaced by 10) from any shared priorities used by read-only/meta hooks.
##########
docs/setup/compile.md:
##########
@@ -182,6 +182,8 @@ make docsbuild
This guide explains how Apache Sedona manages local code quality checks, formatting, and linting. To ensure a fast and consistent developer experience, we combine `uv` (a lightning-fast Python package and environment manager) with `prek` (our internal pre-commit wrapper framework) and expose everything via a simple, unified `Makefile`.
+`.pre-commit-config.yaml` assigns the independent, read-only linters a `priority` named after the language or domain they check (e.g. `lint-python`, `lint-yaml`, `lint-shell`, `lint-actions`, `lint-makefile`, `lint-rst`). All of these aliases resolve to the same priority value, so `prek` runs them concurrently instead of one at a time — linting different languages in parallel, as described in [issue #3080](https://github.com/apache/sedona/issues/3080). Hooks that rewrite file contents (formatters, license insertion, tab fixers, etc.) instead get explicit, strictly increasing numeric `priority` values that preserve today's sequential execution order, since several of them touch overlapping files and `prek` only guarantees correct results when hooks sharing a priority don't mutate the same files. For hook IDs that match one of `prek`'s [Built-in Fast Hooks](https://prek.j178.dev/builtin/) (such as `check-yaml`, `check-json`, `trailing-whitespace`, and `end-of-file-fixer`), `prek` au
tomatically swaps in its native Rust implementation instead of spinning up a Python environment, no config changes required.
Review Comment:
This is a very long single line/paragraph, which makes the docs harder to read and maintain (and can also trip common markdown style rules). Consider splitting it into multiple wrapped lines and/or a short bullet list (e.g., one bullet for parallel lint aliases, one for sequential mutators, one for Built-in Fast Hooks) while keeping the same wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #3492:
URL: https://github.com/apache/sedona/pull/3492#issuecomment-6517780461
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4330952783
##########
docs/setup/compile.md:
##########
@@ -182,6 +182,8 @@ make docsbuild
This guide explains how Apache Sedona manages local code quality checks, formatting, and linting. To ensure a fast and consistent developer experience, we combine `uv` (a lightning-fast Python package and environment manager) with `prek` (our internal pre-commit wrapper framework) and expose everything via a simple, unified `Makefile`.
+`.pre-commit-config.yaml` assigns the independent, read-only linters a `priority` named after the language or domain they check (e.g. `lint-python`, `lint-yaml`, `lint-shell`, `lint-actions`, `lint-makefile`, `lint-rst`). All of these aliases resolve to the same priority value, so `prek` runs them concurrently instead of one at a time — linting different languages in parallel, as described in [issue #3080](https://github.com/apache/sedona/issues/3080). Hooks that rewrite file contents (formatters, license insertion, tab fixers, etc.) instead get explicit, strictly increasing numeric `priority` values that preserve today's sequential execution order, since several of them touch overlapping files and `prek` only guarantees correct results when hooks sharing a priority don't mutate the same files. For hook IDs that match one of `prek`'s [Built-in Fast Hooks](https://prek.j178.dev/builtin/) (such as `check-yaml`, `check-json`, `trailing-whitespace`, and `end-of-file-fixer`), `prek` au
tomatically swaps in its native Rust implementation instead of spinning up a Python environment, no config changes required.
Review Comment:
Fixed in fb87d538ff2cd6dc19226a74d1c4a5d55d5632d9: split into a short intro plus three bullets (parallel lint aliases / sequential mutators / Built-in Fast Hooks). markdownlint passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4322600542
##########
.pre-commit-config.yaml:
##########
@@ -21,23 +21,44 @@ default_stages: [pre-commit, pre-push]
default_language_version:
python: python3
node: 24.16.0
-minimum_prek_version: '0.4.0'
+# 0.4.11 added reusable `priority` aliases, used below.
+minimum_prek_version: '0.4.11'
+# lint-* aliases share one priority so independent, read-only checks run
+# concurrently (see https://github.com/apache/sedona/issues/3080). Hooks
+# that rewrite files keep strictly increasing integer priorities instead,
+# since some of them touch the same files and prek only parallelizes
+# safely when hooks sharing a priority don't mutate the same files.
+priorities:
+ lint-actions: 100
+ lint-docs: 100
+ lint-generic: 100
+ lint-json: 100
+ lint-makefile: 100
+ lint-python: 100
+ lint-rst: 100
+ lint-shell: 100
+ lint-toml: 100
+ lint-xml: 100
+ lint-yaml: 100
repos:
- repo: meta
hooks:
- id: identity
name: run identity check
description: a simple hook which prints all arguments passed to it, useful for debugging
+ priority: 0
- id: check-hooks-apply
name: run check hooks apply
description: check that all the hooks apply to the repository
+ priority: 1
Review Comment:
Fixed in d08b9ee7ab3f3e9cbc43ec63f4a899ad9bef3e4b. identity and check-hooks-apply are both read-only, so they now share priority 0. Also spaced all the mutator priorities by 10 (0, 10, 20, ...) instead of packing them sequentially, so new hooks can be slotted in later without renumbering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3492:
URL: https://github.com/apache/sedona/pull/3492#discussion_r4322828146
##########
.pre-commit-config.yaml:
##########
@@ -377,107 +433,137 @@ repos:
args: [--config=.github/linters/zizmor.yml]
files: ^\.github/workflows/.*$
types: [yaml]
+ priority: lint-actions
- repo: https://github.com/pre-commit/pygrep-hooks
rev: 3a6eb0fadf60b3cccfd80bad9dbb6fae7e47b316 # frozen: v1.10.0
hooks:
- id: python-check-mock-methods
name: run check for not-real mock methods
description: Prevent common mistakes of assert mck.not_called(), assert mck.called_once_with(...) and mck.assert_called
+ priority: lint-python
- id: python-no-eval
name: run check for eval()
description: A quick check for the eval() built-in function
exclude: ^python/tests/flink/test_flink_registration\.py$
+ priority: lint-python
- id: python-no-log-warn
name: run check for use logger.warning(
description: A quick check for the deprecated .warn() method of python loggers
+ priority: lint-python
- id: python-use-type-annotations
name: run check for type annotations not comments
description: Enforce that python3.6+ type annotations are used instead of type comments
+ priority: lint-python
- id: rst-backticks
name: run rst-backticks
description: detect common mistake of using single backticks when writing rst
+ priority: lint-rst
- id: rst-directive-colons
name: run rst-directive-colons
description: detect mistake of rst directive not ending with double colon or space before the double colon
+ priority: lint-rst
- id: rst-inline-touching-normal
name: run rst-inline-touching-normal
description: detect mistake of inline code touching normal text in rst
+ priority: lint-rst
- id: text-unicode-replacement-char
name: run check for no unicode replacement char
description: Forbid files which have a UTF-8 Unicode replacement character
+ priority: lint-generic
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-ast
name: run check-ast
description: check Python files for syntax errors
+ priority: lint-python
- id: check-builtin-literals
name: run check-builtin-literals
description: check Python files for proper use of built-in literals
+ priority: lint-python
- id: check-case-conflict
name: run check-case-conflict
description: check for case conflicts in file names
+ priority: lint-generic
- id: check-executables-have-shebangs
name: run check-executables-have-shebangs
description: check that executable scripts have shebang lines
+ priority: lint-generic
- id: check-illegal-windows-names
name: run check-illegal-windows-names
description: check for Windows-illegal file names
+ priority: lint-generic
- id: check-json
name: run check-json
description: check JSON files for syntax errors
exclude: ^docs/usecases/data/testPolygon\.json$|^spark/common/src/test/resources/.*\.json$
+ priority: lint-json
Review Comment:
The PR description claims “the same effective order as before,” but assigning `check-json` (and other read-only checks in this repo) to `lint-*` (1000) while `end-of-file-fixer` is `310` changes the relative order compared to the existing in-repo hook ordering (previously `check-json` ran before `end-of-file-fixer`). If preserving the previous effective order is a requirement, consider giving these read-only hooks priorities that keep them positioned where they were before (e.g., a shared `lint-*` value that sits before `end-of-file-fixer` within the sequence), or update the PR description/docs to explicitly note that linters now run after mutators.
##########
.pre-commit-config.yaml:
##########
@@ -377,107 +433,137 @@ repos:
args: [--config=.github/linters/zizmor.yml]
files: ^\.github/workflows/.*$
types: [yaml]
+ priority: lint-actions
- repo: https://github.com/pre-commit/pygrep-hooks
rev: 3a6eb0fadf60b3cccfd80bad9dbb6fae7e47b316 # frozen: v1.10.0
hooks:
- id: python-check-mock-methods
name: run check for not-real mock methods
description: Prevent common mistakes of assert mck.not_called(), assert mck.called_once_with(...) and mck.assert_called
+ priority: lint-python
- id: python-no-eval
name: run check for eval()
description: A quick check for the eval() built-in function
exclude: ^python/tests/flink/test_flink_registration\.py$
+ priority: lint-python
- id: python-no-log-warn
name: run check for use logger.warning(
description: A quick check for the deprecated .warn() method of python loggers
+ priority: lint-python
- id: python-use-type-annotations
name: run check for type annotations not comments
description: Enforce that python3.6+ type annotations are used instead of type comments
+ priority: lint-python
- id: rst-backticks
name: run rst-backticks
description: detect common mistake of using single backticks when writing rst
+ priority: lint-rst
- id: rst-directive-colons
name: run rst-directive-colons
description: detect mistake of rst directive not ending with double colon or space before the double colon
+ priority: lint-rst
- id: rst-inline-touching-normal
name: run rst-inline-touching-normal
description: detect mistake of inline code touching normal text in rst
+ priority: lint-rst
- id: text-unicode-replacement-char
name: run check for no unicode replacement char
description: Forbid files which have a UTF-8 Unicode replacement character
+ priority: lint-generic
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-ast
name: run check-ast
description: check Python files for syntax errors
+ priority: lint-python
- id: check-builtin-literals
name: run check-builtin-literals
description: check Python files for proper use of built-in literals
+ priority: lint-python
- id: check-case-conflict
name: run check-case-conflict
description: check for case conflicts in file names
+ priority: lint-generic
- id: check-executables-have-shebangs
name: run check-executables-have-shebangs
description: check that executable scripts have shebang lines
+ priority: lint-generic
- id: check-illegal-windows-names
name: run check-illegal-windows-names
description: check for Windows-illegal file names
+ priority: lint-generic
- id: check-json
name: run check-json
description: check JSON files for syntax errors
exclude: ^docs/usecases/data/testPolygon\.json$|^spark/common/src/test/resources/.*\.json$
+ priority: lint-json
- id: check-merge-conflict
name: run check-merge-conflict
description: check for merge conflict markers
+ priority: lint-generic
- id: check-shebang-scripts-are-executable
name: run check-shebang-scripts-are-executable
description: check that scripts with shebangs are executable
+ priority: lint-generic
- id: check-toml
name: run check-toml
description: check TOML files for syntax errors
+ priority: lint-toml
- id: check-vcs-permalinks
name: run check-vcs-permalinks
description: ensures that links to vcs websites are permalinks
+ priority: lint-generic
- id: check-xml
name: run check-xml
description: attempts to load all xml files to verify syntax
+ priority: lint-xml
- id: check-yaml
name: run check-yaml
description: attempts to load all yaml files to verify syntax
exclude: ^mkdocs\.yml$
+ priority: lint-yaml
- id: debug-statements
name: run debug-statements
description: check for debugger imports and py37+ `breakpoint()` calls in python source.
+ priority: lint-python
- id: destroyed-symlinks
name: run destroyed-symlinks
description: detects symlinks which are changed to regular files with a content of a path which that symlink was pointing to
+ priority: lint-generic
- id: detect-aws-credentials
name: run detect-aws-credentials
description: checks for the existence of AWS secrets that you have set up with the AWS CLI
args: [--allow-missing-credentials]
+ priority: lint-generic
- id: detect-private-key
name: run detect-private-key
description: checks for the existence of private keys
+ priority: lint-generic
- id: end-of-file-fixer
name: run end-of-file-fixer
description: makes sure files end in a newline and only a newline
exclude: \.svg$|^docs/image|^spark/common/src/test/resources
+ priority: 310
Review Comment:
The PR description claims “the same effective order as before,” but assigning `check-json` (and other read-only checks in this repo) to `lint-*` (1000) while `end-of-file-fixer` is `310` changes the relative order compared to the existing in-repo hook ordering (previously `check-json` ran before `end-of-file-fixer`). If preserving the previous effective order is a requirement, consider giving these read-only hooks priorities that keep them positioned where they were before (e.g., a shared `lint-*` value that sits before `end-of-file-fixer` within the sequence), or update the PR description/docs to explicitly note that linters now run after mutators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "TravisMai (via GitHub)" <gi...@apache.org>.
TravisMai commented on PR #3492:
URL: https://github.com/apache/sedona/pull/3492#issuecomment-6330605910
Hi @jbampton @jiayuasu, I've addressed the previous review comments and pushed the updates, including the latest Copilot suggestions. Would appreciate it if you could take another look when you have time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-3080] Add prek hook priorities for parallel execution and use built-in fast hooks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #3492:
URL: https://github.com/apache/sedona/pull/3492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org