You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pchintar (via GitHub)" <gi...@apache.org> on 2026/06/01 03:45:15 UTC
[PR] [SPARK-57190][SQL] Resolve API inconsistency for 4-argument regexp_replace [spark]
pchintar opened a new pull request, #56240:
URL: https://github.com/apache/spark/pull/56240
### What changes were proposed in this pull request?
Spark SQL already supports the 4-argument form of `regexp_replace`:
```sql
regexp_replace(str, regexp, replacement, position)
```
However, the corresponding Scala, PySpark, and Spark Connect APIs currently expose only the 3-argument variants.
This PR exposes the existing 4-argument functionality through:
* Scala API (`functions.regexp_replace`)
* PySpark API (`functions.regexp_replace`)
* Spark Connect API (`functions.regexp_replace`)
and adds corresponding Scala, PySpark, and Connect test coverage.
### Why are the changes needed?
The underlying SQL functionality already exists and is available through SQL expressions, but it is not accessible through the public Scala, PySpark, and Spark Connect APIs.
This creates an inconsistency between SQL and programmatic interfaces. Exposing the optional `position` argument aligns the public APIs with existing SQL functionality.
### Does this PR introduce *any* user-facing change?
Yes.
Users can now specify the optional `position` argument through the Scala, PySpark, and Spark Connect APIs.
Before:
```scala
regexp_replace(col("s"), "(\\d+)", "--")
```
After:
```scala
regexp_replace(col("s"), "(\\d+)", "--", 5)
```
Similarly, PySpark users can now call:
```python
F.regexp_replace("s", r"(\d+)", "--", 5)
```
### How was this patch tested?
Added test coverage in:
* `StringFunctionsSuite`
* `FunctionsTests.test_regexp_replace`
* `SparkConnectFunctionTests.test_string_functions_multi_args`
Verified with:
```bash
./build/sbt "sql-api/compile"
./build/sbt "sql/Test/compile"
./build/sbt \
"sql/testOnly org.apache.spark.sql.StringFunctionsSuite -- -z \"regex_replace / regex_extract\""
python3.11 -m pytest \
python/pyspark/sql/tests/test_functions.py::FunctionsTests::test_regexp_replace -v
python3.11 -m pytest \
python/pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_multi_args -v
```
### Was this patch authored or co-authored using generative AI tooling?
Co-written with GPT 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4618791008
thanks, merging to master/4.x!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4612494379
@cloud-fan @cloud-fan all the CI checks have passed as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4610564860
@cloud-fan & @shrirangmhalgi thnx for the review! I made some changes as per your feedback:
* I Updated the Scala `@since` annotations and the PySpark `versionchanged` annotation from `4.2.0` to `4.3.0`.
* I added a PySpark doctest example demonstrating the new 4-argument form of `regexp_replace` with the `position` parameter, and so the documentation now covers both the existing 3-argument usage and the new 4-argument usage.
* I kept the Scala parameter name as `pos` instead of `position` because `functions.scala` already defines overloaded public methods named `position(...)`; using `position` as the parameter name in this overload causes ambiguous resolution against those existing methods inside the same object. So, I updated the Scaladoc wording to explicitly reference `pos`, so the documentation now matches the signature clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56240:
URL: https://github.com/apache/spark/pull/56240#discussion_r3345649955
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -5401,6 +5411,16 @@ object functions {
def regexp_replace(e: Column, pattern: Column, replacement: Column): Column =
Column.fn("regexp_replace", e, pattern, replacement)
+ /**
+ * Replace all substrings of the specified string value that match regexp with rep, starting at
+ * `position`.
+ *
+ * @group string_funcs
+ * @since 4.2.0
Review Comment:
Same here — `@since` should be `4.3.0`.
```suggestion
* @since 4.3.0
```
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -5392,6 +5392,16 @@ object functions {
def regexp_replace(e: Column, pattern: String, replacement: String): Column =
regexp_replace(e, lit(pattern), lit(replacement))
+ /**
+ * Replace all substrings of the specified string value that match regexp with rep, starting at
+ * `position`.
+ *
+ * @group string_funcs
+ * @since 4.2.0
Review Comment:
`@since` should be `4.3.0`, not `4.2.0`: 4.2 is already cut (`branch-4.2` exists) and master is on `5.0.0-SNAPSHOT`, so this new overload first ships in 4.3.0. Same correction as [SPARK-56820](https://issues.apache.org/jira/browse/SPARK-56820) for `counter_diff`.
```suggestion
* @since 4.3.0
```
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -16350,14 +16350,19 @@ def regexp_extract_all(
@_try_remote_functions
def regexp_replace(
- string: "ColumnOrName", pattern: Union[str, Column], replacement: Union[str, Column]
+ string: "ColumnOrName",
+ pattern: Union[str, Column],
+ replacement: Union[str, Column],
+ position: Optional[Union[int, Column]] = None,
) -> Column:
r"""Replace all substrings of the specified string value that match regexp with replacement.
.. versionadded:: 1.5.0
.. versionchanged:: 3.4.0
Supports Spark Connect.
+ .. versionchanged:: 4.2.0
Review Comment:
`versionchanged` should be `4.3.0` — the `position` parameter first appears in 4.3.0, not the already-cut 4.2.0.
```suggestion
.. versionchanged:: 4.3.0
```
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -5392,6 +5392,16 @@ object functions {
def regexp_replace(e: Column, pattern: String, replacement: String): Column =
regexp_replace(e, lit(pattern), lit(replacement))
+ /**
+ * Replace all substrings of the specified string value that match regexp with rep, starting at
Review Comment:
Nit: the prose says "starting at `position`", but the parameter is named `pos` — the backtick'd `position` implies an identifier that doesn't exist in this signature. Consider matching the prose to the param name (or renaming `pos` to `position` to align with the PySpark binding).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4620242948
Thank you @shrirangmhalgi & @cloud-fan for your timely feedback & approval!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4620221642
@pchintar In Apache Spark, committers can merge directly after reviewing - they don't always use GitHub's formal "Approve" button. The merged commit itself is the approval. Welcome to the project! 🎉
Reference - https://github.com/apache/spark/commit/463d8043978f39260e2494f3daae56ca4ea6422d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4619600003
> thanks, merging to master/4.x!
thnx @cloud-fan I'm new to Spark, so is this approved yet bc I don't see the "Approved" in closed PRs for this?
<img width="1626" height="488" alt="Image 6-4-26 at 11 57 AM" src="https://github.com/user-attachments/assets/04383083-6fb9-4f14-aef8-963ce7237221" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4608866649
The change LGTM. It is a clean API addition that mirrors the existing 3-arg pattern.
`One minor suggestion`: The PySpark docstring mentions the `position` parameter but the `Examples` section only shows 3-arg usage. Consider adding a doctest for the 4-arg form so that users discover it via `help(F.regexp_replace)`:
```
>>> df = spark.createDataFrame([("100-200",)], ["str"])
>>> df.select(regexp_replace('str', r'(\d+)', '--', 5).alias('d')).collect()
[Row(d='100---')]
```
(Minor / non-blocking - just for docs completeness.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #56240:
URL: https://github.com/apache/spark/pull/56240#issuecomment-4602280622
cc @cloud-fan @shrirangmhalgi @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #56240: [SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace
URL: https://github.com/apache/spark/pull/56240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org