You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/05/21 00:44:50 UTC
[PR] [INFRA] Require [COMPONENT] tag in PR title at merge time [spark]
zhengruifeng opened a new pull request, #56026:
URL: https://github.com/apache/spark/pull/56026
### What changes were proposed in this pull request?
Add a check in `dev/merge_spark_pr.py` that prompts the committer for one or more `[COMPONENT]` tags when the PR title lacks them, then inserts the tags into the title after the `[SPARK-XXX]` ref(s).
The prompt lists common Spark components with brief descriptions:
```
PR title is missing the [COMPONENT] tag.
Common components:
[CORE] - Spark Core
[SQL] - Spark SQL
[SS] - Structured Streaming
[STREAMING] - DStream API
[CONNECT] - Spark Connect
[PYTHON] - PySpark
[PS] - pandas API on Spark
[ML] - Spark ML (DataFrame-based)
[MLLIB] - MLlib (RDD-based)
[R] - SparkR
[GRAPHX] - GraphX
[UI] - Spark Web UI
[K8S] - Kubernetes
[YARN] - Hadoop YARN
[BUILD] - build system
[INFRA] - project infrastructure
[DOCS] - documentation changes
[TESTS] - test-only changes
Current title: [SPARK-56853] Improve PATH Tests
Enter comma-separated component(s) to insert into the title (e.g. CORE,SQL):
```
Status markers (`[MINOR]`, `[FOLLOWUP]`, `[FOLLOW-UP]`, `[TRIVIAL]`) do not satisfy the component requirement. Revert PRs, which reuse the original commit's title verbatim, are skipped. `[WIP]` is intentionally not on the marker list — a WIP PR should be aborted at the earlier WIP warning, not merged.
### Why are the changes needed?
Some PRs are merged without a `[COMPONENT]` tag in the title (e.g. apache/spark#55866 was merged as `[SPARK-56853] Improve PATH Tests`). This loses module attribution in the changelog and breaks the convention `[SPARK-XXXXX][COMPONENT] Title` that Spark uses.
### Does this PR introduce _any_ user-facing change?
No. The change only affects the committer-facing interactive merge tool.
### How was this patch tested?
Added doctests for `title_missing_component` and `insert_components_into_title`. All doctests pass via `python3 -m doctest dev/merge_spark_pr.py`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3285281963
##########
dev/merge_spark_pr.py:
##########
Review Comment:
the committer will have to type in `[INFRA]` with `[PROJECT INFRA]`, but it seems `[PROJECT INFRA]` is not used in the past 9 years, the last PR using it is https://github.com/apache/spark/pull/17873
maybe we can introduce a alias mapper later, which map `[PROJECT INFRA]` to `[INFRA]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56026:
URL: https://github.com/apache/spark/pull/56026#issuecomment-4518499897
### `prompt_for_components` coverage on real-world titles
Among the parseable titles from the previous sample (983 commits, 172 open PRs), how many would lack a primary tag and trigger the prompt at merge time:
| Source | Parseable | Need prompt | Rate |
|---|---|---|---|
| Commits | 983 | 78 | ~8% |
| Open PRs | 172 | 14 | ~8% |
Breakdown of *why* the prompt would fire:
| Category | Commits | PRs |
|---|---|---|
| No tags at all (just `[SPARK-NNNNN] ...`) | 26 | 8 |
| Only `[PS]` (alone or with `[TEST]` / `[FOLLOWUP]`) | 38 | 3 |
| Only `[TEST]` (alone or with `[FOLLOWUP]`) | 9 | 0 |
| Only `[FOLLOWUP]` | 2 | 0 |
| Only `[UDF]` | 0 | 2 |
| Other (`[WEBIUI]`, `[SHS]`, `[EXAMPLE]`, `[REPL]`) | 3 | 1 |
| **Total** | **78** | **14** |
Observations:
- `[PS]` (Pandas API on Spark) accounts for ~half of the prompts. Promoting it to primary would cut the prompt rate roughly in half.
- `[WEBIUI]` looks like a typo of `[WEBUI]`, which is already an alias of the primary `UI` tag.
- The "no tags at all" group (26 commits, 8 PRs) is the prompt working as intended — titles like `[SPARK-NNNNN] ...` with no component tag get caught.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3285281963
##########
dev/merge_spark_pr.py:
##########
Review Comment:
the script will automatically map it to `[INFRA]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3289110372
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
Review Comment:
Stale example: `[PS]` was promoted to `primary=True` in the latest commit (`Make PS primary; ...`). Replace with a still-non-primary tag so the docstring stays accurate.
```suggestion
requirement. Non-primary JIRA components (e.g. [TEST], [SHUFFLE], [DEPLOY])
```
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
Review Comment:
Dead commented-out registry entry — the alias is now declared on the `R` row above (`Component("R", ("SPARKR",), ...)`), so this comment line can just be deleted.
```suggestion
```
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
Review Comment:
`_BRACKET_TAG_RE` excludes internal spaces (`[A-Za-z0-9._-]+`), so every alias/canonical with an embedded space is unreachable from `Title.parse` — only resolvable through the `prompt_for_components` interactive path. That affects:
- canonical: `"BLOCK MANAGER"` (only the underscore alias is parser-reachable)
- aliases: `"SPARK CORE"`, `"PROJECT INFRA"`, `"WEB UI"`, `"SPARK SHELL"`, `"JAVA API"`, `"FOLLOW UP"`
Practical impact: in your reply to @LuciferYang you said `[PROJECT INFRA]` in a title would automatically map to `[INFRA]`, but the parser actually leaves it in the body text. I verified by importing the module:
```
>>> Title.parse("[SPARK-1234][PROJECT INFRA] Fix")
leading=['SPARK-1234'], components=[], text='[PROJECT INFRA] Fix'
>>> str(_)
'[SPARK-1234] [PROJECT INFRA] Fix'
```
The prompt fires (no primary), and the final title becomes `[SPARK-1234][INFRA] [PROJECT INFRA] Fix`.
Your dry-run on 1000 commits + 200 PRs shows this form doesn't occur in practice, so it's a theoretical regression only. Two options if you want to close it cleanly: drop the with-space variants from the registry (since they're parser-unreachable they only serve the interactive path), or extend the regex to accept internal whitespace so the canonical form is consistent everywhere. Up to you — happy to leave as-is if you prefer.
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
+_SPARK_ID_RE = re.compile(r"^SPARK-\d+$", re.IGNORECASE)
+_VERSION_TAG_RE = re.compile(r"^\d+\.(\d+|X)$")
+_LEADING_TAGS = frozenset({"MINOR", "TRIVIAL"})
+
+
+class Title:
Review Comment:
Same coverage gap I flagged last round: there is no doctest for the empty-body case (`Title.parse("[SPARK-1234]")`). It parses fine (`leading=['SPARK-1234'], components=[], text=''`) and round-trips through `__str__` correctly, but a doctest would pin that behavior — and exercise the `if not self.text: return parts` branch in `__str__`, which is otherwise uncovered.
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
+_SPARK_ID_RE = re.compile(r"^SPARK-\d+$", re.IGNORECASE)
+_VERSION_TAG_RE = re.compile(r"^\d+\.(\d+|X)$")
+_LEADING_TAGS = frozenset({"MINOR", "TRIVIAL"})
+
+
+class Title:
+ """Structured PR title: SPARK refs, component tags, and body.
+
+ ``leading`` — SPARK-NNNNN IDs and [MINOR]/[TRIVIAL] markers, in order.
+ ``components`` — all other bracket tags, in order.
+ ``text`` — body text following the bracket sequence.
+
+ >>> t = Title.parse("[SPARK-1234][SQL] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL'], 'Fix something')
+ >>> str(t)
+ '[SPARK-1234][SQL] Fix something'
+ >>> t = Title.parse("[SPARK-1234][SQL][FOLLOWUP] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'], 'Fix something')
+ >>> str(t)
+ '[SPARK-1234][SQL][FOLLOWUP] Fix something'
+ """
- # Replace multiple spaces with a single space, e.g. if no jira refs and/or components were
- # included
- clean_text = re.sub(r"\s+", " ", clean_text.strip())
+ def __init__(
+ self,
+ leading: List[str],
+ components: List[str],
+ text: str,
+ ) -> None:
+ self.leading = leading
+ self.components = components
+ self.text = text
+
+ @classmethod
+ def parse(cls, raw: str) -> "Title":
+ """Parse a PR title string into a :class:`Title`.
+
+ A title must open with a leading tag ([SPARK-NNNNN], [MINOR], or
+ [TRIVIAL]); otherwise :exc:`ValueError` is raised. Subsequent bracket
+ tokens (spaces trimmed, separated by optional whitespace) go to
+ ``components``. The remainder is ``text``.
+
+ >>> t = Title.parse("[SPARK-1234][SQL][TESTS] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'TESTS'], 'Fix something')
+ >>> t = Title.parse(" [ SPARK-1234 ] [ SQL ] [ TESTS ] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'TESTS'], 'Fix something')
+ >>> t = Title.parse("[SPARK-1234 ][ sql ][ followup ] Fix")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'], 'Fix')
+ >>> str(t)
+ '[SPARK-1234][SQL][FOLLOWUP] Fix'
+ >>> Title.parse("[MINOR] Fix typo").leading
+ ['MINOR']
+ >>> t = Title.parse("[spark-1234][sql][followup] Fix")
+ >>> t.leading, t.components
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'])
+ >>> Title.parse("[SPARK-1234][SPARK-5678][SQL] Fix").leading
+ ['SPARK-1234', 'SPARK-5678']
+ >>> Title.parse("[SPARK-1234][4.X][SQL] Fix").components
+ ['4.X', 'SQL']
+ >>> Title.parse("[SPARK-1234][SQL][4.2] Fix").components
+ ['SQL', '4.2']
+ >>> Title.parse("[SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: '[SQL] Fix'
+ >>> Title.parse("No brackets")
+ Traceback (most recent call last):
+ ...
+ ValueError: title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: 'No brackets'
+ >>> Title.parse("[SPARK-1234][SQL][SPARK-123] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN] tags must all appear before other tags: '[SPARK-1234][SQL][SPARK-123] Fix'
+ >>> Title.parse("[SPARK-1234][MINOR][SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist
+ >>> Title.parse("[MINOR][TRIVIAL][SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist
+ """
+ leading: List[str] = []
+ components: List[str] = []
+
+ raw = raw.strip()
+ m0 = _BRACKET_TAG_RE.match(raw)
+ first = m0.group(1).upper() if m0 else ""
+ if not (_SPARK_ID_RE.match(first) or first in _LEADING_TAGS):
+ raise ValueError(
+ "title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: %r" % raw
+ )
- return clean_text
+ past_leading = False
+ pos = 0
+ while pos < len(raw):
+ m = _BRACKET_TAG_RE.match(raw, pos)
+ if not m:
+ break
+ tag = m.group(1).upper()
+ if _SPARK_ID_RE.match(tag):
+ if past_leading:
+ raise ValueError(
+ "[SPARK-NNNNN] tags must all appear before other tags: %r" % raw
+ )
+ leading.append(tag)
+ elif tag in _LEADING_TAGS:
+ leading.append(tag)
+ else:
+ components.append(tag)
+ past_leading = True
+ pos = m.end()
+ while pos < len(raw) and raw[pos] == " ":
+ pos += 1
+
+ text = raw[pos:].lstrip()
+ markers = [t for t in leading if t in _LEADING_TAGS]
+ if len(markers) > 1 or (markers and len(leading) > len(markers)):
+ raise ValueError(
+ "[SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist"
+ )
+ return cls(leading, components, text)
+
+ def __str__(self) -> str:
+ parts = "".join("[%s]" % t for t in self.leading)
+ parts += "".join("[%s]" % c for c in self.components)
+ if not self.text:
+ return parts
+ return parts + (" " if parts else "") + self.text
+
+
+def prompt_for_components():
+ """
+ Prompt the committer for component(s) when the PR title lacks a primary
+ component. Each entered token is normalized via Component.find
+ (e.g. "DOCS" -> "DOC", "PYSPARK" -> "PYTHON"). Unrecognized tokens are
+ passed through as-is. Re-prompts until at least one entered token resolves
+ to a tag in PRIMARY_COMPONENTS. Returns an uppercase list of tags in
Review Comment:
Stale name — there is no `PRIMARY_COMPONENTS` constant in the module; primariness is encoded on each `Component` via the `primary` attribute. Suggested rewording:
```suggestion
to a primary Component (one with primary=True). Returns an uppercase list of tags in
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56026:
URL: https://github.com/apache/spark/pull/56026#issuecomment-4531259678
Thanks for the re-review @cloud-fan! Addressed all 5 inline findings in `fc2ee3d92`:
1. Stale `[PS]` example in `Component` docstring → swapped for `[DEPLOY]` (still non-primary).
2. Dead commented-out `SPARKR` registry entry → removed.
3. Parser-unreachable with-space aliases → dropped (`SPARK CORE`, `PROJECT INFRA`, `JAVA API`, `SPARK SHELL`, `WEB UI`, `FOLLOW UP`). Also flipped the `Block Manager` canonical from `BLOCK MANAGER` to `BLOCK_MANAGER` so it's parser-reachable. The corresponding underscore aliases remain so the interactive prompt still accepts them.
4. Missing doctest for empty-body case → added `Title.parse("[SPARK-1234]")` doctest covering the `__str__` no-text branch.
5. Stale `PRIMARY_COMPONENTS` reference in `prompt_for_components` docstring → reworded to "a primary Component (one with `primary=True`)".
Also corrected my earlier reply to @LuciferYang on the `[PROJECT INFRA]` thread, which was wrong (the with-space form wasn't actually reachable from `Title.parse` — only the underscore form is).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #56026: [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time
URL: https://github.com/apache/spark/pull/56026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3295918145
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
Review Comment:
Done — updated to use (still non-primary). Thanks!
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
Review Comment:
Done — removed the dead comment line.
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
Review Comment:
Good catch — went with the first option and dropped the parser-unreachable with-space variants (`SPARK CORE`, `PROJECT INFRA`, `JAVA API`, `SPARK SHELL`, `WEB UI`, `FOLLOW UP`). Also flipped `BLOCK MANAGER` → `BLOCK_MANAGER` so the canonical tag is parser-reachable. The remaining underscore aliases stay so the interactive prompt still accepts `SPARK_CORE`, `PROJECT_INFRA`, etc.
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
+_SPARK_ID_RE = re.compile(r"^SPARK-\d+$", re.IGNORECASE)
+_VERSION_TAG_RE = re.compile(r"^\d+\.(\d+|X)$")
+_LEADING_TAGS = frozenset({"MINOR", "TRIVIAL"})
+
+
+class Title:
Review Comment:
Added — `Title.parse("[SPARK-1234]")` doctest now covers both the empty-body parse and the `__str__` 'no text' branch.
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
+ remain recognized — they normalize and pass through validation — but
+ they must be paired with a primary tag (e.g. [SQL][TEST]). Status
+ markers are never primary. [WIP] is intentionally absent from the
+ registry: a WIP PR should be aborted at the earlier WIP warning, not
+ merged.
"""
- jira_refs = []
- components = []
- # If this is a Revert PR, no need to process any further
- if text.startswith('Revert "') and text.endswith('"'):
- return text
+ def __init__(self, tag, aliases=(), primary=False, jira_name=""):
+ self.tag = tag
+ self.aliases = frozenset(aliases)
+ self.primary = primary
+ self.jira_name = jira_name
- # If the string is compliant, no need to process any further
- if re.search(r"^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+", text):
- return text
+ def matches(self, token):
+ return token == self.tag or token in self.aliases
- # Extract JIRA ref(s):
- pattern = re.compile(r"(SPARK[-\s]*[0-9]{3,6})+", re.IGNORECASE)
- for ref in pattern.findall(text):
- # Add brackets, replace spaces with a dash, & convert to uppercase
- jira_refs.append("[" + re.sub(r"\s+", "-", ref.upper()) + "]")
- text = text.replace(ref, "")
+ @classmethod
+ def find(cls, token):
+ """Return the Component matching ``token`` (case-insensitive), or None."""
+ if token is None:
+ return None
+ token = token.strip().upper()
+ for c in COMPONENTS:
+ if c.matches(token):
+ return c
+ return None
- # Extract spark component(s):
- # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
- pattern = re.compile(r"(\[[\w\s,.-]+\])", re.IGNORECASE)
- for component in pattern.findall(text):
- components.append(component.upper())
- text = text.replace(component, "")
- # Cleanup any remaining symbols:
- pattern = re.compile(r"^\W+(.*)", re.IGNORECASE)
- if pattern.search(text) is not None:
- text = pattern.search(text).groups()[0]
- # Assemble full text (JIRA ref(s), module(s), remaining text)
- clean_text = "".join(jira_refs).strip() + "".join(components).strip() + " " + text.strip()
+# Full SPARK JIRA component list (sorted alphabetically by tag), followed
+# by status markers. Keep in sync with the components in JIRA — fetch the
+# current list with:
+# curl -s https://issues.apache.org/jira/rest/api/2/project/SPARK/components
+# A `primary=True` marker indicates the tag alone satisfies the merge-time
+# component requirement; non-primary JIRA components must be paired with a
+# primary one (e.g. [SQL][TEST], [PYTHON][PS], [CORE][SHUFFLE]). Status
+# markers leave `jira_name` empty.
+COMPONENTS = (
+ Component("BLOCK MANAGER", ("BLOCK_MANAGER",), jira_name="Block Manager"),
+ Component("BUILD", primary=True, jira_name="Build"),
+ Component("CONNECT", primary=True, jira_name="Connect"),
+ Component("CORE", ("SPARK CORE", "SPARK_CORE"), primary=True, jira_name="Spark Core"),
+ Component("DEPLOY", jira_name="Deploy"),
+ Component("DOC", ("DOCS", "DOCUMENTATION"), primary=True, jira_name="Documentation"),
+ Component("DOCKER", primary=True, jira_name="Spark Docker"),
+ Component("EC2", jira_name="EC2"),
+ Component("EXAMPLE", ("EXAMPLES",), jira_name="Examples"),
+ Component("GRAPHX", primary=True, jira_name="GraphX"),
+ Component("INFRA", ("PROJECT INFRA", "PROJECT_INFRA"), primary=True, jira_name="Project Infra"),
+ Component("IO", jira_name="Input/Output"),
+ Component("JAVA", ("JAVA API", "JAVA_API", "JAVAAPI"), jira_name="Java API"),
+ Component("K8S", ("KUBERNETES",), primary=True, jira_name="Kubernetes"),
+ Component("MESOS", jira_name="Mesos"),
+ Component("ML", primary=True, jira_name="ML"),
+ Component("MLLIB", primary=True, jira_name="MLlib"),
+ Component("OPTIMIZER", jira_name="Optimizer"),
+ Component("PROTOBUF", jira_name="Protobuf"),
+ Component("PS", primary=True, jira_name="Pandas API on Spark"),
+ Component("PYTHON", ("PYSPARK",), primary=True, jira_name="PySpark"),
+ Component("R", ("SPARKR",), primary=True, jira_name="R"),
+ Component("REPL", ("SHELL", "SPARK SHELL", "SPARK_SHELL"), jira_name="Spark Shell"),
+ Component("SCHEDULER", jira_name="Scheduler"),
+ Component("SDP", ("PIPELINES",), primary=True, jira_name="Declarative Pipelines"),
+ Component("SECURITY", primary=True, jira_name="Security"),
+ Component("SHUFFLE", jira_name="Shuffle"),
+ # Component("SPARKR", jira_name="SparkR"), # SPARKR is treated as an alias of R above
+ Component("SQL", primary=True, jira_name="SQL"),
+ Component("SS", primary=True, jira_name="Structured Streaming"),
+ Component("STREAMING", ("DSTREAM", "DSTREAMS"), primary=True, jira_name="DStreams"),
+ Component("SUBMIT", jira_name="Spark Submit"),
+ Component("TEST", ("TESTS", "TEST-ONLY", "TESTS-ONLY"), jira_name="Tests"),
+ Component("UI", ("WEBUI", "WEB UI", "WEB_UI"), primary=True, jira_name="Web UI"),
+ Component("WINDOWS", primary=True, jira_name="Windows"),
+ Component("YARN", primary=True, jira_name="YARN"),
+ # Status markers — recognized in PR titles, but not JIRA components.
+ Component("FOLLOWUP", ("FOLLOW-UP", "FOLLOW UP")),
+ Component("MINOR"),
+ Component("TRIVIAL"),
+)
+
+
+_BRACKET_TAG_RE = re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
+_SPARK_ID_RE = re.compile(r"^SPARK-\d+$", re.IGNORECASE)
+_VERSION_TAG_RE = re.compile(r"^\d+\.(\d+|X)$")
+_LEADING_TAGS = frozenset({"MINOR", "TRIVIAL"})
+
+
+class Title:
+ """Structured PR title: SPARK refs, component tags, and body.
+
+ ``leading`` — SPARK-NNNNN IDs and [MINOR]/[TRIVIAL] markers, in order.
+ ``components`` — all other bracket tags, in order.
+ ``text`` — body text following the bracket sequence.
+
+ >>> t = Title.parse("[SPARK-1234][SQL] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL'], 'Fix something')
+ >>> str(t)
+ '[SPARK-1234][SQL] Fix something'
+ >>> t = Title.parse("[SPARK-1234][SQL][FOLLOWUP] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'], 'Fix something')
+ >>> str(t)
+ '[SPARK-1234][SQL][FOLLOWUP] Fix something'
+ """
- # Replace multiple spaces with a single space, e.g. if no jira refs and/or components were
- # included
- clean_text = re.sub(r"\s+", " ", clean_text.strip())
+ def __init__(
+ self,
+ leading: List[str],
+ components: List[str],
+ text: str,
+ ) -> None:
+ self.leading = leading
+ self.components = components
+ self.text = text
+
+ @classmethod
+ def parse(cls, raw: str) -> "Title":
+ """Parse a PR title string into a :class:`Title`.
+
+ A title must open with a leading tag ([SPARK-NNNNN], [MINOR], or
+ [TRIVIAL]); otherwise :exc:`ValueError` is raised. Subsequent bracket
+ tokens (spaces trimmed, separated by optional whitespace) go to
+ ``components``. The remainder is ``text``.
+
+ >>> t = Title.parse("[SPARK-1234][SQL][TESTS] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'TESTS'], 'Fix something')
+ >>> t = Title.parse(" [ SPARK-1234 ] [ SQL ] [ TESTS ] Fix something")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'TESTS'], 'Fix something')
+ >>> t = Title.parse("[SPARK-1234 ][ sql ][ followup ] Fix")
+ >>> t.leading, t.components, t.text
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'], 'Fix')
+ >>> str(t)
+ '[SPARK-1234][SQL][FOLLOWUP] Fix'
+ >>> Title.parse("[MINOR] Fix typo").leading
+ ['MINOR']
+ >>> t = Title.parse("[spark-1234][sql][followup] Fix")
+ >>> t.leading, t.components
+ (['SPARK-1234'], ['SQL', 'FOLLOWUP'])
+ >>> Title.parse("[SPARK-1234][SPARK-5678][SQL] Fix").leading
+ ['SPARK-1234', 'SPARK-5678']
+ >>> Title.parse("[SPARK-1234][4.X][SQL] Fix").components
+ ['4.X', 'SQL']
+ >>> Title.parse("[SPARK-1234][SQL][4.2] Fix").components
+ ['SQL', '4.2']
+ >>> Title.parse("[SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: '[SQL] Fix'
+ >>> Title.parse("No brackets")
+ Traceback (most recent call last):
+ ...
+ ValueError: title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: 'No brackets'
+ >>> Title.parse("[SPARK-1234][SQL][SPARK-123] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN] tags must all appear before other tags: '[SPARK-1234][SQL][SPARK-123] Fix'
+ >>> Title.parse("[SPARK-1234][MINOR][SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist
+ >>> Title.parse("[MINOR][TRIVIAL][SQL] Fix")
+ Traceback (most recent call last):
+ ...
+ ValueError: [SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist
+ """
+ leading: List[str] = []
+ components: List[str] = []
+
+ raw = raw.strip()
+ m0 = _BRACKET_TAG_RE.match(raw)
+ first = m0.group(1).upper() if m0 else ""
+ if not (_SPARK_ID_RE.match(first) or first in _LEADING_TAGS):
+ raise ValueError(
+ "title must start with [SPARK-NNNNN], [MINOR], or [TRIVIAL]: %r" % raw
+ )
- return clean_text
+ past_leading = False
+ pos = 0
+ while pos < len(raw):
+ m = _BRACKET_TAG_RE.match(raw, pos)
+ if not m:
+ break
+ tag = m.group(1).upper()
+ if _SPARK_ID_RE.match(tag):
+ if past_leading:
+ raise ValueError(
+ "[SPARK-NNNNN] tags must all appear before other tags: %r" % raw
+ )
+ leading.append(tag)
+ elif tag in _LEADING_TAGS:
+ leading.append(tag)
+ else:
+ components.append(tag)
+ past_leading = True
+ pos = m.end()
+ while pos < len(raw) and raw[pos] == " ":
+ pos += 1
+
+ text = raw[pos:].lstrip()
+ markers = [t for t in leading if t in _LEADING_TAGS]
+ if len(markers) > 1 or (markers and len(leading) > len(markers)):
+ raise ValueError(
+ "[SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot coexist"
+ )
+ return cls(leading, components, text)
+
+ def __str__(self) -> str:
+ parts = "".join("[%s]" % t for t in self.leading)
+ parts += "".join("[%s]" % c for c in self.components)
+ if not self.text:
+ return parts
+ return parts + (" " if parts else "") + self.text
+
+
+def prompt_for_components():
+ """
+ Prompt the committer for component(s) when the PR title lacks a primary
+ component. Each entered token is normalized via Component.find
+ (e.g. "DOCS" -> "DOC", "PYSPARK" -> "PYTHON"). Unrecognized tokens are
+ passed through as-is. Re-prompts until at least one entered token resolves
+ to a tag in PRIMARY_COMPONENTS. Returns an uppercase list of tags in
Review Comment:
Done — replaced with 'a primary Component (one with primary=True)'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3281452915
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
Review Comment:
The allowlist has 23 entries, but SPARK's active JIRA components number ~32 (per `issues.apache.org/jira/rest/api/2/project/SPARK/components`). Several active components have no allowlist entry: `Declarative Pipelines` (would be `[PIPELINES]` per the "last-word-uppercased" convention, or `[SDP]` as 77 recent commits actually use), `Deploy`, `DStreams`, `Examples`, `Optimizer`, `Scheduler`, `Shuffle`, `Spark Docker`, `Spark Shell`, `Spark Submit`, `Windows`, `Block Manager`.
What's the selection rule for what goes in `COMPONENT_MARKERS`? A few possibilities:
1. **Full JIRA canonical list** — derive from `spark_jira_utils.list_components` (network-live) or snapshot it here, so any future JIRA addition is recognized.
2. **Curated "common subsystems" only** — explicitly document the criterion (commit volume? top-level visibility?) so the next maintainer knows when to add or remove an entry.
3. **Status quo** — keep the current set as-is and force committers to add a covering canonical tag for everything else (`[PYTHON][SDP]`, `[CORE][SHUFFLE]`, etc.).
If the answer is (3), it's worth saying so in the header comment — and especially worth confirming for `Declarative Pipelines`, since `[SDP]` is the largest single non-canonical tag in recent history and the SDP team has presumably been using it as a deliberate subsystem marker.
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
+ {
+ "AVRO",
+ "BUILD",
+ "CONNECT",
+ "CORE",
+ "DOC",
+ "DOCS",
+ "GRAPHX",
+ "INFRA",
+ "K8S",
+ "ML",
+ "MLLIB",
+ "PROTOBUF",
+ "PS",
+ "PYTHON",
+ "R",
+ "REPL",
+ "SECURITY",
+ "SQL",
+ "SS",
+ "STREAMING",
+ "TESTS",
+ "UI",
+ "YARN",
+ }
+)
+
+
+def title_missing_component(title):
+ """
+ Return True when the PR title has no bracket whose content is in
+ COMPONENT_MARKERS.
+
+ A compliant Spark PR title looks like: [SPARK-XXXXX][COMPONENT1][COMPONENT2] Rest,
+ where at least one [COMPONENT] is in COMPONENT_MARKERS. Status markers like
+ [MINOR] or [FOLLOWUP] are not in COMPONENT_MARKERS and don't satisfy the
+ requirement. Unknown tags like [FOO] also don't satisfy it. Revert PRs,
+ which reuse the original commit's title verbatim, are not flagged.
+
+ >>> title_missing_component("[SPARK-56853] Improve PATH Tests")
+ True
+ >>> title_missing_component("[SPARK-56853][SQL] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-56853][SQL][TESTS] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-1234][SPARK-5678] Title")
+ True
+ >>> title_missing_component("[SPARK-1234][SPARK-5678][CORE] Title")
+ False
+ >>> title_missing_component('Revert "[SPARK-48591][PYTHON] Simplify"')
+ False
+ >>> title_missing_component("Plain title with no brackets")
+ True
+ >>> title_missing_component("[SPARK-1234][MINOR] Fix typo")
+ True
+ >>> title_missing_component("[SPARK-1234][FOLLOWUP] Follow up")
+ True
+ >>> title_missing_component("[SPARK-1234][TRIVIAL] Trivial fix")
+ True
+ >>> title_missing_component("[SPARK-1234][SQL][MINOR] SQL typo fix")
+ False
+ >>> title_missing_component("[SPARK-1234][FOO] Unknown component")
+ True
+ >>> title_missing_component("[MINOR] Fix typo")
+ True
+ """
+ if title.startswith('Revert "') and title.endswith('"'):
+ return False
+ brackets = re.findall(r"\[([^\]]+)\]", title)
+ return not any(b.strip().upper() in COMPONENT_MARKERS for b in brackets)
+
+
+def insert_components_into_title(title, components):
+ """
+ Insert [COMPONENT1][COMPONENT2]... after the leading [SPARK-XXX] ref(s) in title.
+ If the title has no leading SPARK ref, the component tag is prepended. When the
+ remainder of the title starts with another [bracket] tag (e.g. an existing
+ sub-tag like [SDP]), the inserted block is joined to it without a space so all
+ leading brackets stay adjacent.
+
+ >>> insert_components_into_title("[SPARK-56853] Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title(
+ ... "[SPARK-1234][SPARK-5678] Title", ["CORE", "SQL"])
+ '[SPARK-1234][SPARK-5678][CORE][SQL] Title'
+ >>> insert_components_into_title("Random title", ["CORE"])
+ '[CORE] Random title'
+ >>> insert_components_into_title("[SPARK-56853]Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title("[SPARK-56953][SDP] Implement", ["SQL"])
+ '[SPARK-56953][SQL][SDP] Implement'
+ >>> insert_components_into_title(
+ ... "[SPARK-56953][SDP] Implement", ["SQL", "CORE"])
+ '[SPARK-56953][SQL][CORE][SDP] Implement'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE"])
+ '[CORE][MINOR] Fix typo'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE", "SQL"])
+ '[CORE][SQL][MINOR] Fix typo'
+ """
+ component_tag = "".join("[%s]" % c for c in components)
+ m = re.match(r"^((?:\[SPARK-[0-9]{3,6}\])+)\s*(.*)$", title)
+ if m and m.group(1):
+ refs, rest = m.group(1), m.group(2)
+ if not rest:
+ return "%s%s" % (refs, component_tag)
+ sep = "" if rest.startswith("[") else " "
+ return "%s%s%s%s" % (refs, component_tag, sep, rest)
+ sep = "" if title.startswith("[") else " "
+ return "%s%s%s" % (component_tag, sep, title)
+
+
+def prompt_for_components(title):
+ """
+ Prompt the committer for component(s) when the PR title lacks a recognized
+ [COMPONENT] tag. Validates that at least one entered component is in
+ COMPONENT_MARKERS. Re-prompts until valid input is provided. Returns an
+ uppercase list.
+ """
+ print_error("PR title is missing a recognized [COMPONENT] tag.")
Review Comment:
`print_error` is the project's red-text helper for fatal/error states. This message is a recoverable interactive prompt, not a failure — consider using `print` (or a less alarming color) so it doesn't read as "the script broke." Minor.
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
+ {
+ "AVRO",
+ "BUILD",
+ "CONNECT",
+ "CORE",
+ "DOC",
+ "DOCS",
+ "GRAPHX",
+ "INFRA",
+ "K8S",
+ "ML",
+ "MLLIB",
+ "PROTOBUF",
+ "PS",
+ "PYTHON",
+ "R",
+ "REPL",
+ "SECURITY",
+ "SQL",
+ "SS",
+ "STREAMING",
+ "TESTS",
+ "UI",
+ "YARN",
+ }
+)
+
+
+def title_missing_component(title):
+ """
+ Return True when the PR title has no bracket whose content is in
+ COMPONENT_MARKERS.
+
+ A compliant Spark PR title looks like: [SPARK-XXXXX][COMPONENT1][COMPONENT2] Rest,
+ where at least one [COMPONENT] is in COMPONENT_MARKERS. Status markers like
+ [MINOR] or [FOLLOWUP] are not in COMPONENT_MARKERS and don't satisfy the
+ requirement. Unknown tags like [FOO] also don't satisfy it. Revert PRs,
+ which reuse the original commit's title verbatim, are not flagged.
+
+ >>> title_missing_component("[SPARK-56853] Improve PATH Tests")
+ True
+ >>> title_missing_component("[SPARK-56853][SQL] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-56853][SQL][TESTS] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-1234][SPARK-5678] Title")
+ True
+ >>> title_missing_component("[SPARK-1234][SPARK-5678][CORE] Title")
+ False
+ >>> title_missing_component('Revert "[SPARK-48591][PYTHON] Simplify"')
+ False
+ >>> title_missing_component("Plain title with no brackets")
+ True
+ >>> title_missing_component("[SPARK-1234][MINOR] Fix typo")
+ True
+ >>> title_missing_component("[SPARK-1234][FOLLOWUP] Follow up")
+ True
+ >>> title_missing_component("[SPARK-1234][TRIVIAL] Trivial fix")
+ True
+ >>> title_missing_component("[SPARK-1234][SQL][MINOR] SQL typo fix")
+ False
+ >>> title_missing_component("[SPARK-1234][FOO] Unknown component")
+ True
+ >>> title_missing_component("[MINOR] Fix typo")
+ True
+ """
+ if title.startswith('Revert "') and title.endswith('"'):
+ return False
+ brackets = re.findall(r"\[([^\]]+)\]", title)
+ return not any(b.strip().upper() in COMPONENT_MARKERS for b in brackets)
+
+
+def insert_components_into_title(title, components):
Review Comment:
Worth adding a doctest for `insert_components_into_title("[SPARK-1234]", ["SQL"])` — that exercises the `if not rest:` branch which currently has no doctest coverage.
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
+ {
+ "AVRO",
+ "BUILD",
+ "CONNECT",
+ "CORE",
+ "DOC",
+ "DOCS",
+ "GRAPHX",
+ "INFRA",
+ "K8S",
+ "ML",
+ "MLLIB",
+ "PROTOBUF",
+ "PS",
+ "PYTHON",
+ "R",
+ "REPL",
+ "SECURITY",
+ "SQL",
+ "SS",
+ "STREAMING",
+ "TESTS",
+ "UI",
+ "YARN",
+ }
+)
+
+
+def title_missing_component(title):
+ """
+ Return True when the PR title has no bracket whose content is in
+ COMPONENT_MARKERS.
+
+ A compliant Spark PR title looks like: [SPARK-XXXXX][COMPONENT1][COMPONENT2] Rest,
+ where at least one [COMPONENT] is in COMPONENT_MARKERS. Status markers like
+ [MINOR] or [FOLLOWUP] are not in COMPONENT_MARKERS and don't satisfy the
+ requirement. Unknown tags like [FOO] also don't satisfy it. Revert PRs,
+ which reuse the original commit's title verbatim, are not flagged.
+
+ >>> title_missing_component("[SPARK-56853] Improve PATH Tests")
+ True
+ >>> title_missing_component("[SPARK-56853][SQL] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-56853][SQL][TESTS] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-1234][SPARK-5678] Title")
+ True
+ >>> title_missing_component("[SPARK-1234][SPARK-5678][CORE] Title")
+ False
+ >>> title_missing_component('Revert "[SPARK-48591][PYTHON] Simplify"')
+ False
+ >>> title_missing_component("Plain title with no brackets")
+ True
+ >>> title_missing_component("[SPARK-1234][MINOR] Fix typo")
+ True
+ >>> title_missing_component("[SPARK-1234][FOLLOWUP] Follow up")
+ True
+ >>> title_missing_component("[SPARK-1234][TRIVIAL] Trivial fix")
+ True
+ >>> title_missing_component("[SPARK-1234][SQL][MINOR] SQL typo fix")
+ False
+ >>> title_missing_component("[SPARK-1234][FOO] Unknown component")
+ True
+ >>> title_missing_component("[MINOR] Fix typo")
+ True
+ """
+ if title.startswith('Revert "') and title.endswith('"'):
+ return False
+ brackets = re.findall(r"\[([^\]]+)\]", title)
+ return not any(b.strip().upper() in COMPONENT_MARKERS for b in brackets)
+
+
+def insert_components_into_title(title, components):
+ """
+ Insert [COMPONENT1][COMPONENT2]... after the leading [SPARK-XXX] ref(s) in title.
+ If the title has no leading SPARK ref, the component tag is prepended. When the
+ remainder of the title starts with another [bracket] tag (e.g. an existing
+ sub-tag like [SDP]), the inserted block is joined to it without a space so all
+ leading brackets stay adjacent.
+
+ >>> insert_components_into_title("[SPARK-56853] Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title(
+ ... "[SPARK-1234][SPARK-5678] Title", ["CORE", "SQL"])
+ '[SPARK-1234][SPARK-5678][CORE][SQL] Title'
+ >>> insert_components_into_title("Random title", ["CORE"])
+ '[CORE] Random title'
+ >>> insert_components_into_title("[SPARK-56853]Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title("[SPARK-56953][SDP] Implement", ["SQL"])
+ '[SPARK-56953][SQL][SDP] Implement'
+ >>> insert_components_into_title(
+ ... "[SPARK-56953][SDP] Implement", ["SQL", "CORE"])
+ '[SPARK-56953][SQL][CORE][SDP] Implement'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE"])
+ '[CORE][MINOR] Fix typo'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE", "SQL"])
+ '[CORE][SQL][MINOR] Fix typo'
+ """
+ component_tag = "".join("[%s]" % c for c in components)
+ m = re.match(r"^((?:\[SPARK-[0-9]{3,6}\])+)\s*(.*)$", title)
+ if m and m.group(1):
+ refs, rest = m.group(1), m.group(2)
+ if not rest:
+ return "%s%s" % (refs, component_tag)
+ sep = "" if rest.startswith("[") else " "
+ return "%s%s%s%s" % (refs, component_tag, sep, rest)
+ sep = "" if title.startswith("[") else " "
+ return "%s%s%s" % (component_tag, sep, title)
+
+
+def prompt_for_components(title):
+ """
+ Prompt the committer for component(s) when the PR title lacks a recognized
+ [COMPONENT] tag. Validates that at least one entered component is in
+ COMPONENT_MARKERS. Re-prompts until valid input is provided. Returns an
+ uppercase list.
+ """
+ print_error("PR title is missing a recognized [COMPONENT] tag.")
+ print("Common components:")
+ print(" [BUILD] - build system")
+ print(" [CONNECT] - Spark Connect")
+ print(" [CORE] - Spark Core")
+ print(" [DOCS] - documentation changes")
+ print(" [GRAPHX] - GraphX")
+ print(" [INFRA] - project infrastructure")
+ print(" [K8S] - Kubernetes")
+ print(" [ML] - Spark ML (DataFrame-based)")
+ print(" [MLLIB] - MLlib (RDD-based)")
+ print(" [PS] - pandas API on Spark")
+ print(" [PYTHON] - PySpark")
+ print(" [R] - SparkR")
+ print(" [SQL] - Spark SQL")
+ print(" [SS] - Structured Streaming")
+ print(" [STREAMING] - DStream API")
+ print(" [TESTS] - test-only changes")
+ print(" [UI] - Spark Web UI")
+ print(" [YARN] - Hadoop YARN")
+ print("Current title: %s" % title)
+ while True:
+ raw = bold_input(
+ "Enter comma-separated component(s) to insert into the title (e.g. CORE,SQL): "
+ )
+ components = [c.strip().upper() for c in raw.split(",") if c.strip()]
+ if not components:
+ print_error("Component(s) cannot be empty. Please enter at least one.")
+ continue
+ if not any(c in COMPONENT_MARKERS for c in components):
Review Comment:
When the user enters a mix of canonical and unknown tags (e.g., `SQL,FOO` or a typo like `SQL,SQLLL`), the unknown tag is silently inserted into the final title. Consider warning so typos don't slip through — for example:
```python
unknown = [c for c in components if c not in COMPONENT_MARKERS]
if not any(c in COMPONENT_MARKERS for c in components):
print_error(
"At least one component must be a recognized Spark component. "
"Got: %s" % ", ".join(components)
)
continue
if unknown:
print(
"Warning: unrecognized component(s) will be inserted as-is: %s" % ", ".join(unknown)
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3282505190
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
+ {
+ "AVRO",
+ "BUILD",
+ "CONNECT",
+ "CORE",
+ "DOC",
+ "DOCS",
+ "GRAPHX",
+ "INFRA",
+ "K8S",
+ "ML",
+ "MLLIB",
+ "PROTOBUF",
+ "PS",
+ "PYTHON",
+ "R",
+ "REPL",
+ "SECURITY",
+ "SQL",
+ "SS",
+ "STREAMING",
+ "TESTS",
+ "UI",
+ "YARN",
+ }
+)
+
+
+def title_missing_component(title):
+ """
+ Return True when the PR title has no bracket whose content is in
+ COMPONENT_MARKERS.
+
+ A compliant Spark PR title looks like: [SPARK-XXXXX][COMPONENT1][COMPONENT2] Rest,
+ where at least one [COMPONENT] is in COMPONENT_MARKERS. Status markers like
+ [MINOR] or [FOLLOWUP] are not in COMPONENT_MARKERS and don't satisfy the
+ requirement. Unknown tags like [FOO] also don't satisfy it. Revert PRs,
+ which reuse the original commit's title verbatim, are not flagged.
+
+ >>> title_missing_component("[SPARK-56853] Improve PATH Tests")
+ True
+ >>> title_missing_component("[SPARK-56853][SQL] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-56853][SQL][TESTS] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-1234][SPARK-5678] Title")
+ True
+ >>> title_missing_component("[SPARK-1234][SPARK-5678][CORE] Title")
+ False
+ >>> title_missing_component('Revert "[SPARK-48591][PYTHON] Simplify"')
+ False
+ >>> title_missing_component("Plain title with no brackets")
+ True
+ >>> title_missing_component("[SPARK-1234][MINOR] Fix typo")
+ True
+ >>> title_missing_component("[SPARK-1234][FOLLOWUP] Follow up")
+ True
+ >>> title_missing_component("[SPARK-1234][TRIVIAL] Trivial fix")
+ True
+ >>> title_missing_component("[SPARK-1234][SQL][MINOR] SQL typo fix")
+ False
+ >>> title_missing_component("[SPARK-1234][FOO] Unknown component")
+ True
+ >>> title_missing_component("[MINOR] Fix typo")
+ True
+ """
+ if title.startswith('Revert "') and title.endswith('"'):
+ return False
+ brackets = re.findall(r"\[([^\]]+)\]", title)
+ return not any(b.strip().upper() in COMPONENT_MARKERS for b in brackets)
+
+
+def insert_components_into_title(title, components):
+ """
+ Insert [COMPONENT1][COMPONENT2]... after the leading [SPARK-XXX] ref(s) in title.
+ If the title has no leading SPARK ref, the component tag is prepended. When the
+ remainder of the title starts with another [bracket] tag (e.g. an existing
+ sub-tag like [SDP]), the inserted block is joined to it without a space so all
+ leading brackets stay adjacent.
+
+ >>> insert_components_into_title("[SPARK-56853] Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title(
+ ... "[SPARK-1234][SPARK-5678] Title", ["CORE", "SQL"])
+ '[SPARK-1234][SPARK-5678][CORE][SQL] Title'
+ >>> insert_components_into_title("Random title", ["CORE"])
+ '[CORE] Random title'
+ >>> insert_components_into_title("[SPARK-56853]Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title("[SPARK-56953][SDP] Implement", ["SQL"])
+ '[SPARK-56953][SQL][SDP] Implement'
+ >>> insert_components_into_title(
+ ... "[SPARK-56953][SDP] Implement", ["SQL", "CORE"])
+ '[SPARK-56953][SQL][CORE][SDP] Implement'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE"])
+ '[CORE][MINOR] Fix typo'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE", "SQL"])
+ '[CORE][SQL][MINOR] Fix typo'
+ """
+ component_tag = "".join("[%s]" % c for c in components)
+ m = re.match(r"^((?:\[SPARK-[0-9]{3,6}\])+)\s*(.*)$", title)
Review Comment:
```suggestion
m = re.match(r"^((?:\[SPARK-[0-9]+\])+)\s*(.*)$", title)
```
Current Spark JIRA numbers span 3-6 digits (e.g., legacy `SPARK-979` through current `SPARK-56979`). Once SPARK numbers reach 7 digits, the `{3,6}` upper bound silently breaks the matcher — `insert_components_into_title` falls through to the no-SPARK-ref prepend branch and produces `[COMPONENT][SPARK-1000000] Title` instead of `[SPARK-1000000][COMPONENT] Title`. JIRA numbering is monotonically increasing, so absent any future fix to this regex, the malformation is **inevitable** once that threshold is reached (hence MEDIUM, not LOW). Drop the upper bound; the leading `[SPARK-` already disambiguates from non-SPARK brackets.
##########
dev/merge_spark_pr.py:
##########
@@ -773,6 +773,169 @@ def standardize_jira_ref(text):
return clean_text
+# Recognized Spark component tags. A PR title must contain at least one of
+# these (within square brackets) to be merged. The list shown to the committer
+# in prompt_for_components is a subset of this set selected for prominence.
+# Status markers like [MINOR] or [FOLLOWUP] are intentionally absent and do
+# not satisfy the component requirement. [WIP] is also absent: a WIP PR
+# should be aborted at the earlier WIP warning, not merged.
+COMPONENT_MARKERS = frozenset(
+ {
+ "AVRO",
+ "BUILD",
+ "CONNECT",
+ "CORE",
+ "DOC",
+ "DOCS",
+ "GRAPHX",
+ "INFRA",
+ "K8S",
+ "ML",
+ "MLLIB",
+ "PROTOBUF",
+ "PS",
+ "PYTHON",
+ "R",
+ "REPL",
+ "SECURITY",
+ "SQL",
+ "SS",
+ "STREAMING",
+ "TESTS",
+ "UI",
+ "YARN",
+ }
+)
+
+
+def title_missing_component(title):
+ """
+ Return True when the PR title has no bracket whose content is in
+ COMPONENT_MARKERS.
+
+ A compliant Spark PR title looks like: [SPARK-XXXXX][COMPONENT1][COMPONENT2] Rest,
+ where at least one [COMPONENT] is in COMPONENT_MARKERS. Status markers like
+ [MINOR] or [FOLLOWUP] are not in COMPONENT_MARKERS and don't satisfy the
+ requirement. Unknown tags like [FOO] also don't satisfy it. Revert PRs,
+ which reuse the original commit's title verbatim, are not flagged.
+
+ >>> title_missing_component("[SPARK-56853] Improve PATH Tests")
+ True
+ >>> title_missing_component("[SPARK-56853][SQL] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-56853][SQL][TESTS] Improve PATH Tests")
+ False
+ >>> title_missing_component("[SPARK-1234][SPARK-5678] Title")
+ True
+ >>> title_missing_component("[SPARK-1234][SPARK-5678][CORE] Title")
+ False
+ >>> title_missing_component('Revert "[SPARK-48591][PYTHON] Simplify"')
+ False
+ >>> title_missing_component("Plain title with no brackets")
+ True
+ >>> title_missing_component("[SPARK-1234][MINOR] Fix typo")
+ True
+ >>> title_missing_component("[SPARK-1234][FOLLOWUP] Follow up")
+ True
+ >>> title_missing_component("[SPARK-1234][TRIVIAL] Trivial fix")
+ True
+ >>> title_missing_component("[SPARK-1234][SQL][MINOR] SQL typo fix")
+ False
+ >>> title_missing_component("[SPARK-1234][FOO] Unknown component")
+ True
+ >>> title_missing_component("[MINOR] Fix typo")
+ True
+ """
+ if title.startswith('Revert "') and title.endswith('"'):
+ return False
+ brackets = re.findall(r"\[([^\]]+)\]", title)
+ return not any(b.strip().upper() in COMPONENT_MARKERS for b in brackets)
+
+
+def insert_components_into_title(title, components):
+ """
+ Insert [COMPONENT1][COMPONENT2]... after the leading [SPARK-XXX] ref(s) in title.
+ If the title has no leading SPARK ref, the component tag is prepended. When the
+ remainder of the title starts with another [bracket] tag (e.g. an existing
+ sub-tag like [SDP]), the inserted block is joined to it without a space so all
+ leading brackets stay adjacent.
+
+ >>> insert_components_into_title("[SPARK-56853] Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title(
+ ... "[SPARK-1234][SPARK-5678] Title", ["CORE", "SQL"])
+ '[SPARK-1234][SPARK-5678][CORE][SQL] Title'
+ >>> insert_components_into_title("Random title", ["CORE"])
+ '[CORE] Random title'
+ >>> insert_components_into_title("[SPARK-56853]Improve PATH Tests", ["SQL"])
+ '[SPARK-56853][SQL] Improve PATH Tests'
+ >>> insert_components_into_title("[SPARK-56953][SDP] Implement", ["SQL"])
+ '[SPARK-56953][SQL][SDP] Implement'
+ >>> insert_components_into_title(
+ ... "[SPARK-56953][SDP] Implement", ["SQL", "CORE"])
+ '[SPARK-56953][SQL][CORE][SDP] Implement'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE"])
+ '[CORE][MINOR] Fix typo'
+ >>> insert_components_into_title("[MINOR] Fix typo", ["CORE", "SQL"])
+ '[CORE][SQL][MINOR] Fix typo'
+ """
+ component_tag = "".join("[%s]" % c for c in components)
+ m = re.match(r"^((?:\[SPARK-[0-9]{3,6}\])+)\s*(.*)$", title)
+ if m and m.group(1):
+ refs, rest = m.group(1), m.group(2)
+ if not rest:
+ return "%s%s" % (refs, component_tag)
+ sep = "" if rest.startswith("[") else " "
+ return "%s%s%s%s" % (refs, component_tag, sep, rest)
+ sep = "" if title.startswith("[") else " "
+ return "%s%s%s" % (component_tag, sep, title)
+
+
+def prompt_for_components(title):
+ """
+ Prompt the committer for component(s) when the PR title lacks a recognized
+ [COMPONENT] tag. Validates that at least one entered component is in
+ COMPONENT_MARKERS. Re-prompts until valid input is provided. Returns an
+ uppercase list.
+ """
+ print_error("PR title is missing a recognized [COMPONENT] tag.")
+ print("Common components:")
+ print(" [BUILD] - build system")
+ print(" [CONNECT] - Spark Connect")
+ print(" [CORE] - Spark Core")
+ print(" [DOCS] - documentation changes")
+ print(" [GRAPHX] - GraphX")
+ print(" [INFRA] - project infrastructure")
+ print(" [K8S] - Kubernetes")
+ print(" [ML] - Spark ML (DataFrame-based)")
+ print(" [MLLIB] - MLlib (RDD-based)")
+ print(" [PS] - pandas API on Spark")
+ print(" [PYTHON] - PySpark")
+ print(" [R] - SparkR")
+ print(" [SQL] - Spark SQL")
+ print(" [SS] - Structured Streaming")
+ print(" [STREAMING] - DStream API")
+ print(" [TESTS] - test-only changes")
+ print(" [UI] - Spark Web UI")
+ print(" [YARN] - Hadoop YARN")
+ print("Current title: %s" % title)
+ while True:
+ raw = bold_input(
+ "Enter comma-separated component(s) to insert into the title (e.g. CORE,SQL): "
+ )
+ components = [c.strip().upper() for c in raw.split(",") if c.strip()]
Review Comment:
```suggestion
components = list(dict.fromkeys(
c.strip().upper() for c in raw.split(",") if c.strip()
))
```
Entering `"SQL,SQL"` produces `[SQL][SQL]` in the title with the current code. `dict.fromkeys` deduplicates while preserving insertion order. Complements the existing typo-warning concern raised by another reviewer.
##########
dev/merge_spark_pr.py:
##########
Review Comment:
It seems tags like `[PROJECT INFRA]` were originally supported. Will any issues arise after this modification?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56026:
URL: https://github.com/apache/spark/pull/56026#issuecomment-4518477374
### `Title.parse` coverage on real-world titles
Tested against the latest **1000 commits** on `master` and **200 open PRs**:
| Source | Total | Bypassed* | Parse OK | Parse Fail |
|---|---|---|---|---|
| Commits | 1000 | 15 | 983 | 2 |
| Open PRs | 200 | 11 | 172 | 17 |
\* Bypassed = `Revert "..."` / `Reapply "..."` / `[WIP]` / `[DO-NOT-MERGE]` titles short-circuited in `main()` before `Title.parse` is called.
#### Failing commits (already on `master`, slipped past the old lenient script)
```
revert [SPARK-43752] and add test
[DOCS] Clarify DataFrames in quickstart
```
#### Failing open PRs (all malformed — exactly what this PR intends to block)
```
Add recursive read for partition
fix issue MapType as a type hint for schema 55900
WIP Spark 56763 sarutak 3.5 restore additional functionality r2
[SPARK-XXXXX][CORE] Add IO_URING transport mode
Fix UNBOUND_SQL_PARAMETER regression in PySpark 4.1.1
fix: the hive thrift server's ldap authentication pr... in...
Predicate Pushdown in Spark Structured Streaming (DataSource V2).
[SPARK-XXXXX][SQL] Public Column.toJson / Column.fromJson on the V2 catalog interface
fix: remove bare excepts and clean up lint issues (quantum-local-fixer)
[SQL][DML] Resolve Path Based Tables Read in Transactional Writes
Project DSv2 leaf node to Scan's schema
[SPARK-XXXXX][SQL][TESTS] Add DSv2 table refresh infrastructure and t…
[SPARK-XXXXX][CONNECT][TESTS] Add Connect DataFrame pinning tests
[SPARK-XXXXX][SQL][TESTS] Add DSv2 CACHE TABLE and write validation t…
[SQL] Update grammar/TableChange to allow CLUSTER BY expressions
[SPARK 55879] [Web UI] copy sql to clipboard
Implement `SET` command in single-pass analyzer
```
Breakdown of the 17 PR failures: 5 `[SPARK-XXXXX]` placeholders, 2 component-first (no SPARK ref), 1 `[SPARK 55879]` (space instead of hyphen), 9 with no leading bracket at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56026:
URL: https://github.com/apache/spark/pull/56026#issuecomment-4535858516
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3282571000
##########
dev/merge_spark_pr.py:
##########
Review Comment:
It seems tags like `[PROJECT INFRA]` were originally supported. Will any issues arise after this fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56026:
URL: https://github.com/apache/spark/pull/56026#issuecomment-4518681755
### End-to-end pipeline impact on real-world titles
Ran the full new title pipeline (parse → normalize via registry → prompt-if-no-primary → dedup → move version tags to head → move FOLLOWUP last → warn on unknown tags) against the same sample.
| | Commits (983) | PRs (172) |
|---|---|---|
| Need prompt for primary | 40 | 11 |
| Title would be modified | 190 | 37 |
| Has unknown tag(s) (warning only) | 9 | 7 |
#### Breakdown of "modified" titles
| Kind | Commits | PRs |
|---|---|---|
| Tag rename (`TESTS→TEST`, `FOLLOW-UP→FOLLOWUP`, `DOCS→DOC`, `WEBUI→UI`, `EXAMPLES→EXAMPLE`, `PYSPARK→PYTHON`, `SHELL→REPL`) | 187 | 25 |
| Reorder only (`FOLLOWUP` → last, version tag → head) | 3 | 4 |
| Whitespace / case fixes (e.g. missing space, lowercase inner tags) | 0 | 8 |
Examples:
- Reorder: `[SPARK-55897][SQL][4.0] ...` → `[SPARK-55897][4.0][SQL] ...`
- Whitespace: `[SPARK-56998]Add SECURITY.md ...` → `[SPARK-56998] Add SECURITY.md ...`
- Case: `[SPARK-56962][SS][RTM][StreamingShuffle][Part2] ...` → `[SPARK-56962][SS][RTM][STREAMINGSHUFFLE][PART2] ...`
#### Unknown tags
Recurring (worth considering for the registry):
| Tag | Commits | PRs |
|---|---|---|
| `DML` | 3 | 2 |
| `GEO` | 3 | 0 |
| `UDF` | 0 | 2 |
| `RTM` | 0 | 2 |
| `SHS` | 1 | 0 |
One-offs / typos:
- `WEBIUI` — typo of `WEBUI` (already aliased to primary `UI`)
- `REVERT`, `SPARK`, `STREAMINGSHUFFLE`, `PART1`, `PART2` — non-standard or one-off
For unknown tags the script just prints a warning and continues — the committer can decide whether to fix or proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56979][INFRA] Require COMPONENT tag in PR title at merge time [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56026:
URL: https://github.com/apache/spark/pull/56026#discussion_r3295918145
##########
dev/merge_spark_pr.py:
##########
@@ -700,77 +701,275 @@ def resolve_jira_issues(title, merge_branches, comment):
resolve_jira_issue(merge_branches, comment, jira_id)
-def standardize_jira_ref(text):
- """
- Standardize the [SPARK-XXXXX] [MODULE] prefix
- Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
- "[SPARK-XXX][MLLIB] Issue"
-
- >>> standardize_jira_ref(
- ... "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
- '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
- >>> standardize_jira_ref(
- ... "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
- '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
- >>> standardize_jira_ref("[MLlib] Spark 5954: Top by key")
- '[SPARK-5954][MLLIB] Top by key'
- >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
- '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
- >>> standardize_jira_ref(
- ... "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
- '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
- >>> standardize_jira_ref("[WIP] [SPARK-1146] Vagrant support for Spark")
- '[SPARK-1146][WIP] Vagrant support for Spark'
- >>> standardize_jira_ref(
- ... "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
- '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
- >>> standardize_jira_ref(
- ... "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
- '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
- >>> standardize_jira_ref(
- ... 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"')
- 'Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with F.lit"'
- >>> standardize_jira_ref("Additional information for users building from source code")
- 'Additional information for users building from source code'
+
+class Component:
+ """A Spark PR-title tag, paired with its canonical JIRA component name.
+
+ ``jira_name`` is the canonical name of the SPARK JIRA component (e.g.
+ "Documentation"); empty for status markers like [MINOR] that are not
+ JIRA components but are still recognized in PR titles.
+
+ ``tag`` is the preferred PR-title abbreviation (uppercase, no brackets,
+ e.g. "DOC"). ``aliases`` lists other accepted spellings that resolve to
+ the same component (e.g. "DOCS", "DOCUMENTATION" -> "DOC").
+
+ ``primary`` marks components whose presence alone satisfies the merge-time
+ requirement. Non-primary JIRA components (e.g. [TEST], [PS], [SHUFFLE])
Review Comment:
Done — updated to use `[DEPLOY]` (still non-primary). Thanks!
##########
dev/merge_spark_pr.py:
##########
Review Comment:
Correction to my earlier reply: in the latest revision, `[PROJECT INFRA]` (with a space) is not actually reachable from `Title.parse` — the bracket-tag regex `[A-Za-z0-9._-]+` excludes spaces, so `[PROJECT INFRA]` would have stayed in the body text. To make the registry honest about what is reachable, I dropped the with-space aliases in the latest commit; the underscore form `[PROJECT_INFRA]` still resolves to `[INFRA]`. Per a separate dry-run on the latest 1000 commits + 200 open PRs, none use the with-space form in practice, so the drop should be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org