You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/06/09 11:55:12 UTC
[PR] [INFRA] Reconcile primary JIRA components with the PR title in merge_spark_pr.py [spark]
zhengruifeng opened a new pull request, #56400:
URL: https://github.com/apache/spark/pull/56400
### What changes were proposed in this pull request?
This PR adds a component-reconciliation step to `dev/merge_spark_pr.py`. When a PR is merged, the script already normalizes the component tags in the PR title (e.g. `[SQL]`, `[CORE]`). This change compares the *primary* components implied by the title against the primary components on the linked JIRA ticket and, on a mismatch, prompts the committer to:
- **overwrite** the JIRA ticket's primary components with the PR title's,
- **append** the PR title's primary components to the ticket, or
- **keep** the JIRA ticket unchanged (the default).
Only primary components participate in the comparison. Non-primary title tags (e.g. `[TEST]`) and non-primary JIRA components (e.g. `Optimizer`) are ignored when deciding whether the sets differ, and are preserved by both update paths. The JIRA issue summary printed during a merge now also lists the ticket's components.
Supporting helpers were added: `Component.find_by_jira_name` (reverse lookup from a JIRA component name to its registry entry) and a `primary_only` flag on `jira_components_from_title_tags`.
### Why are the changes needed?
The PR title and the JIRA ticket can drift out of sync on which components a change touches. Today the merge tool resolves the ticket without checking, so a committer has to notice and fix component mismatches by hand. Surfacing the difference at merge time, with a safe default of leaving JIRA untouched, makes it easy to keep the two consistent without forcing any change.
### Does this PR introduce _any_ user-facing change?
No. `dev/merge_spark_pr.py` is a committer-only tool.
### How was this patch tested?
The script runs its doctests on startup via `doctest.testmod()`. New doctests were added for `jira_components_from_title_tags` (including the `primary_only` path) and `Component.find_by_jira_name`; the full suite passes (68 examples). Formatting was verified with `black 26.3.1` (the repo's pinned version) against the root `pyproject.toml`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56400:
URL: https://github.com/apache/spark/pull/56400#discussion_r3444192252
##########
dev/merge_spark_pr.py:
##########
@@ -652,13 +652,90 @@ def print_jira_issue_summary(issue):
assignee = assignee.displayName
assignee = "Assignee\t%s\n" % assignee
status = "Status\t\t%s\n" % issue.fields.status.name
+ components = "Components\t%s\n" % [x.name for x in issue.fields.components]
url = "Url\t\t%s/%s\n" % (JIRA_BASE, issue.key)
target_versions = "Affected\t%s\n" % [x.name for x in issue.fields.versions]
fix_versions = ""
if len(issue.fields.fixVersions) > 0:
fix_versions = "Fixed\t\t%s\n" % [x.name for x in issue.fields.fixVersions]
print("=== JIRA %s ===" % issue.key)
- print("%s%s%s%s%s%s" % (summary, assignee, status, url, target_versions, fix_versions))
+ print(
+ "%s%s%s%s%s%s%s"
+ % (summary, assignee, status, components, url, target_versions, fix_versions)
+ )
+
+
+def jira_components_from_title_tags(tags):
+ """Canonical SPARK JIRA component names implied by PR-title component tags.
+
+ Each tag is resolved through the component registry; a tag that maps to a
+ JIRA component contributes that component's canonical name, whether primary
+ or not (e.g. [SQL] -> "SQL", [TEST] -> "Tests"). Tags that are not JIRA
+ components (status markers like [FOLLOWUP]/[MINOR], version tags like [4.X],
+ or unknown tags) contribute nothing. Aliases normalize to the canonical
+ name. The result preserves input order and is de-duplicated.
+
+ >>> jira_components_from_title_tags(["SQL", "CORE"])
+ ['SQL', 'Spark Core']
+ >>> jira_components_from_title_tags(["PYSPARK", "DOCS"])
+ ['PySpark', 'Documentation']
+ >>> jira_components_from_title_tags(["SQL", "TEST"])
+ ['SQL', 'Tests']
+ >>> jira_components_from_title_tags(["SQL", "FOLLOWUP", "4.X", "BOGUS"])
+ ['SQL']
+ >>> jira_components_from_title_tags(["SQL", "SQL"])
+ ['SQL']
+ """
+ names = []
+ for tag in tags:
+ c = Component.find(tag)
+ if c is not None and c.jira_name:
+ names.append(c.jira_name)
+ return list(dict.fromkeys(names))
+
+
+def reconcile_jira_components(issue, title_components):
+ """Prompt to sync JIRA components when they differ from the PR title.
+
+ ``title_components`` is the list of normalized PR-title component tags (e.g.
+ ["SQL", "TEST"]). Every tag that maps to a JIRA component -- primary or not,
+ e.g. [SQL] -> "SQL" and [TEST] -> "Tests" -- is reconciled; tags with no JIRA
+ component ([MINOR], [FOLLOWUP], version tags, unknown tags) are dropped. The
+ mapped names are compared, as a set, against the issue's current components.
+ On a mismatch, offer to overwrite JIRA with the PR title's components, append
+ them to the existing ones, or keep JIRA unchanged (the default).
+ """
+ title_jira_components = jira_components_from_title_tags(title_components)
+ if not title_jira_components:
+ return
+ current = [c.name for c in issue.fields.components]
+ if set(current) == set(title_jira_components):
+ return
+
+ print()
+ print("=" * 80)
+ print("PR title components differ from JIRA %s:" % issue.key)
+ print(" PR title: %s" % ", ".join(title_jira_components))
+ print(" JIRA: %s" % (", ".join(current) if current else "(none)"))
+ print("=" * 80)
+ choice = get_input(
+ "[o]verwrite JIRA with PR title / [a]ppend PR title to JIRA / [k]eep JIRA as is "
+ "(default: keep): ",
+ {"o": ["o", "overwrite"], "a": ["a", "append"], "k": ["k", "keep", ""]},
+ )
+ if choice == "k":
+ print("Keeping JIRA %s components unchanged." % issue.key)
+ return
+ if choice == "o":
+ new_names = list(title_jira_components)
+ else: # "a": append the PR title's components, keeping the existing ones first.
Review Comment:
Nit: if "a" is really what we're looking for here, then it may be better to just use explicit `elif choice == "a"` to document intent more tightly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56400:
URL: https://github.com/apache/spark/pull/56400#issuecomment-4840787164
thanks all, merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56400:
URL: https://github.com/apache/spark/pull/56400#discussion_r3496595384
##########
dev/merge_spark_pr.py:
##########
@@ -652,13 +652,90 @@ def print_jira_issue_summary(issue):
assignee = assignee.displayName
assignee = "Assignee\t%s\n" % assignee
status = "Status\t\t%s\n" % issue.fields.status.name
+ components = "Components\t%s\n" % [x.name for x in issue.fields.components]
url = "Url\t\t%s/%s\n" % (JIRA_BASE, issue.key)
target_versions = "Affected\t%s\n" % [x.name for x in issue.fields.versions]
fix_versions = ""
if len(issue.fields.fixVersions) > 0:
fix_versions = "Fixed\t\t%s\n" % [x.name for x in issue.fields.fixVersions]
print("=== JIRA %s ===" % issue.key)
- print("%s%s%s%s%s%s" % (summary, assignee, status, url, target_versions, fix_versions))
+ print(
+ "%s%s%s%s%s%s%s"
+ % (summary, assignee, status, components, url, target_versions, fix_versions)
+ )
+
+
+def jira_components_from_title_tags(tags):
+ """Canonical SPARK JIRA component names implied by PR-title component tags.
+
+ Each tag is resolved through the component registry; a tag that maps to a
+ JIRA component contributes that component's canonical name, whether primary
+ or not (e.g. [SQL] -> "SQL", [TEST] -> "Tests"). Tags that are not JIRA
+ components (status markers like [FOLLOWUP]/[MINOR], version tags like [4.X],
+ or unknown tags) contribute nothing. Aliases normalize to the canonical
+ name. The result preserves input order and is de-duplicated.
+
+ >>> jira_components_from_title_tags(["SQL", "CORE"])
+ ['SQL', 'Spark Core']
+ >>> jira_components_from_title_tags(["PYSPARK", "DOCS"])
+ ['PySpark', 'Documentation']
+ >>> jira_components_from_title_tags(["SQL", "TEST"])
+ ['SQL', 'Tests']
+ >>> jira_components_from_title_tags(["SQL", "FOLLOWUP", "4.X", "BOGUS"])
+ ['SQL']
+ >>> jira_components_from_title_tags(["SQL", "SQL"])
+ ['SQL']
+ """
+ names = []
+ for tag in tags:
+ c = Component.find(tag)
Review Comment:
Good catch -- `[PYTHON]` is the canonical tag. The registry already maps `PYTHON -> "PySpark"` (with `PYSPARK` as an alias) and keeps `PS -> "Pandas API on Spark"` as a separate component, so behavior was correct; only this doctest example used the non-preferred `PYSPARK` alias. Updated it to use `PYTHON`.
##########
dev/merge_spark_pr.py:
##########
@@ -652,13 +652,90 @@ def print_jira_issue_summary(issue):
assignee = assignee.displayName
assignee = "Assignee\t%s\n" % assignee
status = "Status\t\t%s\n" % issue.fields.status.name
+ components = "Components\t%s\n" % [x.name for x in issue.fields.components]
url = "Url\t\t%s/%s\n" % (JIRA_BASE, issue.key)
target_versions = "Affected\t%s\n" % [x.name for x in issue.fields.versions]
fix_versions = ""
if len(issue.fields.fixVersions) > 0:
fix_versions = "Fixed\t\t%s\n" % [x.name for x in issue.fields.fixVersions]
print("=== JIRA %s ===" % issue.key)
- print("%s%s%s%s%s%s" % (summary, assignee, status, url, target_versions, fix_versions))
+ print(
+ "%s%s%s%s%s%s%s"
+ % (summary, assignee, status, components, url, target_versions, fix_versions)
+ )
+
+
+def jira_components_from_title_tags(tags):
+ """Canonical SPARK JIRA component names implied by PR-title component tags.
+
+ Each tag is resolved through the component registry; a tag that maps to a
+ JIRA component contributes that component's canonical name, whether primary
+ or not (e.g. [SQL] -> "SQL", [TEST] -> "Tests"). Tags that are not JIRA
+ components (status markers like [FOLLOWUP]/[MINOR], version tags like [4.X],
+ or unknown tags) contribute nothing. Aliases normalize to the canonical
+ name. The result preserves input order and is de-duplicated.
+
+ >>> jira_components_from_title_tags(["SQL", "CORE"])
+ ['SQL', 'Spark Core']
+ >>> jira_components_from_title_tags(["PYSPARK", "DOCS"])
+ ['PySpark', 'Documentation']
+ >>> jira_components_from_title_tags(["SQL", "TEST"])
+ ['SQL', 'Tests']
+ >>> jira_components_from_title_tags(["SQL", "FOLLOWUP", "4.X", "BOGUS"])
+ ['SQL']
+ >>> jira_components_from_title_tags(["SQL", "SQL"])
+ ['SQL']
+ """
+ names = []
+ for tag in tags:
+ c = Component.find(tag)
+ if c is not None and c.jira_name:
+ names.append(c.jira_name)
+ return list(dict.fromkeys(names))
+
+
+def reconcile_jira_components(issue, title_components):
+ """Prompt to sync JIRA components when they differ from the PR title.
+
+ ``title_components`` is the list of normalized PR-title component tags (e.g.
+ ["SQL", "TEST"]). Every tag that maps to a JIRA component -- primary or not,
+ e.g. [SQL] -> "SQL" and [TEST] -> "Tests" -- is reconciled; tags with no JIRA
+ component ([MINOR], [FOLLOWUP], version tags, unknown tags) are dropped. The
+ mapped names are compared, as a set, against the issue's current components.
+ On a mismatch, offer to overwrite JIRA with the PR title's components, append
+ them to the existing ones, or keep JIRA unchanged (the default).
+ """
+ title_jira_components = jira_components_from_title_tags(title_components)
+ if not title_jira_components:
+ return
+ current = [c.name for c in issue.fields.components]
+ if set(current) == set(title_jira_components):
+ return
+
+ print()
+ print("=" * 80)
+ print("PR title components differ from JIRA %s:" % issue.key)
+ print(" PR title: %s" % ", ".join(title_jira_components))
+ print(" JIRA: %s" % (", ".join(current) if current else "(none)"))
+ print("=" * 80)
+ choice = get_input(
+ "[o]verwrite JIRA with PR title / [a]ppend PR title to JIRA / [k]eep JIRA as is "
+ "(default: keep): ",
+ {"o": ["o", "overwrite"], "a": ["a", "append"], "k": ["k", "keep", ""]},
+ )
+ if choice == "k":
+ print("Keeping JIRA %s components unchanged." % issue.key)
+ return
+ if choice == "o":
+ new_names = list(title_jira_components)
+ else: # "a": append the PR title's components, keeping the existing ones first.
Review Comment:
Done -- switched the catch-all `else` to an explicit `elif choice == "a"`. Safe since `get_input` only ever returns `"o"`, `"a"`, or `"k"`, and `"k"` returns early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56400:
URL: https://github.com/apache/spark/pull/56400#issuecomment-4689397257
I used this script to merge https://github.com/apache/spark/pull/56450 whose jira https://issues.apache.org/jira/browse/SPARK-57388 was incorrectly set to 'Spark Core', and then we can correct it by typing 'o'verwrite :
```
...
Check if the JIRA information is as expected (y/N): y
JIRA is unassigned, choose assignee
[0] Ruifeng Zheng (Reporter)
Enter number of user, or userid, to assign to (blank to leave unassigned): 0
================================================================================
PR title components differ from JIRA SPARK-57388:
PR title: Project Infra
JIRA: Spark Core
================================================================================
[o]verwrite JIRA with PR title / [a]ppend PR title to JIRA / [k]eep JIRA as is (default: keep): o
Updated JIRA SPARK-57388 components to: Project Infra
Enter comma-separated fix version(s) [4.3.0]:
=== JIRA SPARK-57388 ===
Summary Pin downstream actions/checkout to a single resolved SHA in maven_test.yml and python_hosted_runner_test.yml
Assignee Ruifeng Zheng
Status Resolved
Components ['Project Infra']
Url https://issues.apache.org/jira/browse/SPARK-57388
Affected ['4.3.0']
Fixed ['4.3.0']
Successfully resolved SPARK-57388 with fixVersions=['4.3.0']!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #56400: [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script
URL: https://github.com/apache/spark/pull/56400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57341][INFRA] Reconcile JIRA components with the PR title in merge script [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #56400:
URL: https://github.com/apache/spark/pull/56400#discussion_r3431991514
##########
dev/merge_spark_pr.py:
##########
@@ -652,13 +652,90 @@ def print_jira_issue_summary(issue):
assignee = assignee.displayName
assignee = "Assignee\t%s\n" % assignee
status = "Status\t\t%s\n" % issue.fields.status.name
+ components = "Components\t%s\n" % [x.name for x in issue.fields.components]
url = "Url\t\t%s/%s\n" % (JIRA_BASE, issue.key)
target_versions = "Affected\t%s\n" % [x.name for x in issue.fields.versions]
fix_versions = ""
if len(issue.fields.fixVersions) > 0:
fix_versions = "Fixed\t\t%s\n" % [x.name for x in issue.fields.fixVersions]
print("=== JIRA %s ===" % issue.key)
- print("%s%s%s%s%s%s" % (summary, assignee, status, url, target_versions, fix_versions))
+ print(
+ "%s%s%s%s%s%s%s"
+ % (summary, assignee, status, components, url, target_versions, fix_versions)
+ )
+
+
+def jira_components_from_title_tags(tags):
+ """Canonical SPARK JIRA component names implied by PR-title component tags.
+
+ Each tag is resolved through the component registry; a tag that maps to a
+ JIRA component contributes that component's canonical name, whether primary
+ or not (e.g. [SQL] -> "SQL", [TEST] -> "Tests"). Tags that are not JIRA
+ components (status markers like [FOLLOWUP]/[MINOR], version tags like [4.X],
+ or unknown tags) contribute nothing. Aliases normalize to the canonical
+ name. The result preserves input order and is de-duplicated.
+
+ >>> jira_components_from_title_tags(["SQL", "CORE"])
+ ['SQL', 'Spark Core']
+ >>> jira_components_from_title_tags(["PYSPARK", "DOCS"])
+ ['PySpark', 'Documentation']
+ >>> jira_components_from_title_tags(["SQL", "TEST"])
+ ['SQL', 'Tests']
+ >>> jira_components_from_title_tags(["SQL", "FOLLOWUP", "4.X", "BOGUS"])
+ ['SQL']
+ >>> jira_components_from_title_tags(["SQL", "SQL"])
+ ['SQL']
+ """
+ names = []
+ for tag in tags:
+ c = Component.find(tag)
Review Comment:
Hm .. PySpark has to be "PYTHON" actually. and we have PS that is pandas API on Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org