You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gengliangwang (via GitHub)" <gi...@apache.org> on 2026/07/08 18:08:35 UTC
[PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
gengliangwang opened a new pull request, #57131:
URL: https://github.com/apache/spark/pull/57131
### What changes were proposed in this pull request?
This PR adds a JIRA-vs-PR verification step to `dev/merge_spark_pr.py`. After the committer enters the PR number (right after the existing Epic/Umbrella check, before the "Proceed with merging?" confirmation), the script now, for each SPARK JIRA id linked from the PR title:
- prints the PR title next to the ticket's summary, status, and issue type, so a mismatch is easy to spot;
- prints a red warning on the status line when the ticket is already `Resolved`/`Closed` (a fresh merge should target an open ticket);
- requires an explicit `y/N` confirmation. Answering anything other than `y` aborts the merge and asks the committer to fix the PR title.
Implementation notes:
- A new pure helper `format_jira_verification(...)` renders the block and is covered by inline doctests. Coloring the Resolved/Closed warning is gated behind a `use_color` flag (off in doctests so the expected output stays plain text, on at the real call site).
- The existing Epic/Umbrella loop already fetches each linked issue; it now also collects them into `linked_issues`, so the verification step reuses those objects and performs no extra JIRA round-trips.
- A small `red(...)` helper was factored out of `print_error` (no behavior change) and reused for the warning.
No JIRA id is newly required by this change: `[MINOR]`/`[TRIVIAL]` PRs still merge with no SPARK id (the loop is a no-op for them). The existing `Title.parse` requirement is unchanged.
### Why are the changes needed?
`merge_spark_pr.py` already prints each linked JIRA's summary, but nothing forces the committer to confirm the ticket actually matches the PR. A PR can therefore be merged against the wrong ticket - for example when a GitHub PR number in the body (`Closes #NNNNN`) is mistaken for a SPARK JIRA id - which resolves an unrelated, possibly already-Resolved ticket on merge. This step surfaces both signals (title/summary mismatch and an already-resolved ticket) and requires a human confirmation before the merge proceeds.
### Does this PR introduce _any_ user-facing change?
No. This only affects the committer-facing merge tooling.
### How was this patch tested?
- `python3 -m doctest dev/merge_spark_pr.py` - all doctests pass (including the two new `format_jira_verification` cases).
- Manually rendered the block with `use_color=True` and confirmed the Resolved/Closed warning is wrapped in the same red escape as `print_error`.
- Verified all added lines are within the 100-char limit and contain no non-ASCII characters.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57131:
URL: https://github.com/apache/spark/pull/57131#issuecomment-4917968181
Thank you, @gengliangwang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in `merge_spark_pr.py` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #57131: [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in `merge_spark_pr.py`
URL: https://github.com/apache/spark/pull/57131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57131:
URL: https://github.com/apache/spark/pull/57131#discussion_r3547566715
##########
dev/merge_spark_pr.py:
##########
@@ -686,6 +690,150 @@ def cherry_pick(pr_num, merge_hash, default_branch, branch_names, target_ref, al
return [_do_cherry_pick(pr_num, merge_hash, pick_ref)]
+# Common words carry no signal when comparing a PR title to a JIRA summary, so they are
+# dropped before scoring. Kept deliberately small: over-aggressive stopword removal makes
+# unrelated titles look similar. Component tags ([SQL], [CORE], ...) are stripped separately.
+_SIMILARITY_STOPWORDS = frozenset(
+ {
+ "a",
+ "an",
+ "the",
+ "to",
+ "for",
+ "of",
+ "in",
+ "on",
+ "and",
+ "or",
+ "with",
+ "is",
+ "are",
+ "be",
+ "when",
+ "should",
+ "make",
+ "add",
+ "fix",
+ "fixes",
+ "support",
+ "enable",
+ "use",
+ "using",
+ }
+)
+_SIMILARITY_WORD_RE = re.compile(r"[a-z0-9]+")
+# Below this Jaccard word-overlap score, warn that the PR title and JIRA summary look
+# unrelated. Tuned to flag clear mismatches (an unrelated ticket scores ~0) while leaving
+# paraphrases and reworded summaries (which still share key nouns) above the line.
+SIMILARITY_WARN_THRESHOLD = 0.2
+
+
+def title_similarity(pr_title, summary):
+ """Jaccard word-overlap between a PR title and a JIRA summary, in [0.0, 1.0].
+
+ Component/version tags ([SQL], [4.x], ...) and the leading SPARK id are stripped,
+ words are lowercased, and common stopwords are dropped, so the score reflects the
+ substantive words the two share. 1.0 means identical word sets; 0.0 means none in
+ common (or an empty side).
+
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Compute a stable checksum")
+ 1.0
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Refactor the logging backend")
+ 0.0
+ >>> round(title_similarity("[SPARK-1] Ceil and floor overflow", "Handle floor overflow"), 2)
+ 0.5
+
+ A [FOLLOWUP] tag is stripped like any other bracket tag, so a follow-up whose
+ title still matches the ticket scores the same as the untagged title:
Review Comment:
Typically, the follow-up PR title's score is not the same with the original PR title (and the JIRA issue title).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in `merge_spark_pr.py` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #57131:
URL: https://github.com/apache/spark/pull/57131#issuecomment-4931636935
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/16dbbe2c8a5482c3590c7f9738c5d4ecc4c69b55
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57131:
URL: https://github.com/apache/spark/pull/57131#discussion_r3547562313
##########
dev/merge_spark_pr.py:
##########
@@ -686,6 +690,150 @@ def cherry_pick(pr_num, merge_hash, default_branch, branch_names, target_ref, al
return [_do_cherry_pick(pr_num, merge_hash, pick_ref)]
+# Common words carry no signal when comparing a PR title to a JIRA summary, so they are
+# dropped before scoring. Kept deliberately small: over-aggressive stopword removal makes
+# unrelated titles look similar. Component tags ([SQL], [CORE], ...) are stripped separately.
+_SIMILARITY_STOPWORDS = frozenset(
+ {
+ "a",
+ "an",
+ "the",
+ "to",
+ "for",
+ "of",
+ "in",
+ "on",
+ "and",
+ "or",
+ "with",
+ "is",
+ "are",
+ "be",
+ "when",
+ "should",
+ "make",
+ "add",
+ "fix",
+ "fixes",
+ "support",
+ "enable",
+ "use",
+ "using",
+ }
+)
+_SIMILARITY_WORD_RE = re.compile(r"[a-z0-9]+")
+# Below this Jaccard word-overlap score, warn that the PR title and JIRA summary look
+# unrelated. Tuned to flag clear mismatches (an unrelated ticket scores ~0) while leaving
+# paraphrases and reworded summaries (which still share key nouns) above the line.
+SIMILARITY_WARN_THRESHOLD = 0.2
+
+
+def title_similarity(pr_title, summary):
+ """Jaccard word-overlap between a PR title and a JIRA summary, in [0.0, 1.0].
+
+ Component/version tags ([SQL], [4.x], ...) and the leading SPARK id are stripped,
+ words are lowercased, and common stopwords are dropped, so the score reflects the
+ substantive words the two share. 1.0 means identical word sets; 0.0 means none in
+ common (or an empty side).
+
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Compute a stable checksum")
+ 1.0
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Refactor the logging backend")
+ 0.0
+ >>> round(title_similarity("[SPARK-1] Ceil and floor overflow", "Handle floor overflow"), 2)
+ 0.5
+
+ A [FOLLOWUP] tag is stripped like any other bracket tag, so a follow-up whose
+ title still matches the ticket scores the same as the untagged title:
Review Comment:
Is this true? Usually, the follow-up title is supposed to describe its code which is usually missed in the original PR. Like the following. The following is an example in `branch-3.5`.
```
$ git log --oneline | grep SPARK-56463
77117c7b32a [SPARK-56463][PYTHON][FOLLOWUP][3.5] Fix wrong kwargs
092f02df06e [SPARK-56463][PYTHON] Disallow unpickling UDT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in `merge_spark_pr.py` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #57131:
URL: https://github.com/apache/spark/pull/57131#issuecomment-4931633601
@dongjoon-hyun thanks for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57131:
URL: https://github.com/apache/spark/pull/57131#issuecomment-4919945989
Thank you for updating, @gengliangwang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #57131:
URL: https://github.com/apache/spark/pull/57131#issuecomment-4919629616
Thanks @dongjoon-hyun! Good call on both — it works for both patterns:
- **`[FOLLOWUP]`**: the JIRA id is extracted from the title regardless of the tag, so verification runs on the referenced ticket. A follow-up usually reuses the parent's (now-`Resolved`) ticket, so the Resolved/Closed warning tends to fire, and a reworded follow-up title can also trip the low-similarity warning — both advisory `y/N`, not hard blocks.
- **`Revert "..."`**: the title is kept verbatim and the id is still extracted from it, so it points at the original ticket and shows the warnings — the intended nudge to double-check.
I've added `title_similarity` doctests for both patterns (973733bff4e): the `[FOLLOWUP]` tag is stripped like any bracket tag (still scores 1.0 on a match), and the `Revert "..."` prefix stays verbatim (the extra word lowers the score but a genuine match still scores well above the threshold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57131:
URL: https://github.com/apache/spark/pull/57131#discussion_r3547566715
##########
dev/merge_spark_pr.py:
##########
@@ -686,6 +690,150 @@ def cherry_pick(pr_num, merge_hash, default_branch, branch_names, target_ref, al
return [_do_cherry_pick(pr_num, merge_hash, pick_ref)]
+# Common words carry no signal when comparing a PR title to a JIRA summary, so they are
+# dropped before scoring. Kept deliberately small: over-aggressive stopword removal makes
+# unrelated titles look similar. Component tags ([SQL], [CORE], ...) are stripped separately.
+_SIMILARITY_STOPWORDS = frozenset(
+ {
+ "a",
+ "an",
+ "the",
+ "to",
+ "for",
+ "of",
+ "in",
+ "on",
+ "and",
+ "or",
+ "with",
+ "is",
+ "are",
+ "be",
+ "when",
+ "should",
+ "make",
+ "add",
+ "fix",
+ "fixes",
+ "support",
+ "enable",
+ "use",
+ "using",
+ }
+)
+_SIMILARITY_WORD_RE = re.compile(r"[a-z0-9]+")
+# Below this Jaccard word-overlap score, warn that the PR title and JIRA summary look
+# unrelated. Tuned to flag clear mismatches (an unrelated ticket scores ~0) while leaving
+# paraphrases and reworded summaries (which still share key nouns) above the line.
+SIMILARITY_WARN_THRESHOLD = 0.2
+
+
+def title_similarity(pr_title, summary):
+ """Jaccard word-overlap between a PR title and a JIRA summary, in [0.0, 1.0].
+
+ Component/version tags ([SQL], [4.x], ...) and the leading SPARK id are stripped,
+ words are lowercased, and common stopwords are dropped, so the score reflects the
+ substantive words the two share. 1.0 means identical word sets; 0.0 means none in
+ common (or an empty side).
+
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Compute a stable checksum")
+ 1.0
+ >>> title_similarity("[SPARK-1][SQL] Compute stable checksum", "Refactor the logging backend")
+ 0.0
+ >>> round(title_similarity("[SPARK-1] Ceil and floor overflow", "Handle floor overflow"), 2)
+ 0.5
+
+ A [FOLLOWUP] tag is stripped like any other bracket tag, so a follow-up whose
+ title still matches the ticket scores the same as the untagged title:
Review Comment:
Typically, the follow-up PR title's score is not the same with the JIRA issue title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org