You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "brijrajk (via GitHub)" <gi...@apache.org> on 2026/05/28 04:50:42 UTC

[PR] [SPARK-40437][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

brijrajk opened a new pull request, #56178:
URL: https://github.com/apache/spark/pull/56178

   ### What changes were proposed in this pull request?
   
   `GroupState.setTimeoutDuration` previously accepted only an integer milliseconds value. This PR
   extends it to also accept a Spark interval string (e.g. `"5 minutes"`, `"1 hour 30 minutes"`,
   `"1.5 seconds"`), matching the behaviour of the Scala API's
   `GroupStateImpl.setTimeoutDuration(String)` overload.
   
   Changes:
   - Added `_parse_timeout_duration(duration: str) -> int` helper in
     `python/pyspark/sql/streaming/state.py` that converts a Spark interval string to milliseconds.
     Parsing behaviour mirrors Scala's `IntervalUtils.stringToInterval` and `IntervalUtils.getDuration`
     (31 days/month convention for structured streaming watermarks).
   - Updated `setTimeoutDuration` to accept `Union[int, str]` and call the helper when a string is
     passed.
   - Added `INVALID_TIMEOUT_DURATION_STRING` error class to
     `python/pyspark/errors/error-conditions.json`.
   - Added `python/pyspark/sql/tests/streaming/test_state.py` with 27 unit tests covering: all
     supported units, months/years (31-day convention), negative component offsets, fractional seconds,
     leading-dot decimals (`.5 seconds`), explicit `+`/`-` signs, whitespace between sign and
     quantity, the `interval` keyword prefix, compound durations, case-insensitivity, and various
     invalid-input cases.
   
   ### Why are the changes needed?
   
   The Scala API supports both `setTimeoutDuration(long durationMs)` and
   `setTimeoutDuration(String duration)`. The Python implementation only supported the integer form,
   leaving users unable to use human-readable interval strings as described in SPARK-40437.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. `GroupState.setTimeoutDuration` now also accepts a Spark interval string such as
   `"5 minutes"` or `"1 hour 30 minutes"`. The integer form continues to work unchanged.
   This change is relative to the unreleased master branch.
   
   ### How was this patch tested?
   
   27 new pure-Python unit tests in `python/pyspark/sql/tests/streaming/test_state.py`, covering
   both positive cases (all units, compound durations, fractional seconds, edge-case signs and
   whitespace) and negative cases (invalid strings, non-positive durations, wrong timeout mode).
   
   Tests can be run without a full Spark build:
   
       source .venv/bin/activate
       PYTHONPATH=python python3 -m unittest pyspark.sql.tests.streaming.test_state -v
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude (Anthropic)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #56178:
URL: https://github.com/apache/spark/pull/56178#discussion_r3348348533


##########
python/pyspark/sql/streaming/state.py:
##########
@@ -170,21 +248,13 @@ def remove(self) -> None:
         self._updated = False
         self._removed = True
 
-    def setTimeoutDuration(self, durationMs: int) -> None:
+    def setTimeoutDuration(self, durationMs: Union[int, str]) -> None:
         """
         Set the timeout duration in ms for this key.
         Processing time timeout must be enabled.

Review Comment:
   shall we add a `versionchanged` to doc that `str` is supported?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on code in PR #56178:
URL: https://github.com/apache/spark/pull/56178#discussion_r3348702672


##########
python/pyspark/sql/streaming/state.py:
##########
@@ -170,21 +248,13 @@ def remove(self) -> None:
         self._updated = False
         self._removed = True
 
-    def setTimeoutDuration(self, durationMs: int) -> None:
+    def setTimeoutDuration(self, durationMs: Union[int, str]) -> None:
         """
         Set the timeout duration in ms for this key.
         Processing time timeout must be enabled.

Review Comment:
   Good point, added! A `versionchanged` note is useful here because this is a behavioral change to an existing method — users upgrading from an older Spark version would not know that string durations are now accepted unless the API docs call it out explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56178:
URL: https://github.com/apache/spark/pull/56178#issuecomment-4773637736

   cc @HeartSaVioR 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on PR #56178:
URL: https://github.com/apache/spark/pull/56178#issuecomment-4582222148

   Could a committer please review this? It extends `GroupState.setTimeoutDuration` to accept a Spark interval string (e.g. `"5 minutes"`, `"1 hour 30 minutes"`) in addition to integer milliseconds, matching the existing Scala API overload (SPARK-40437).
   
   cc @zhengruifeng @itholic


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on code in PR #56178:
URL: https://github.com/apache/spark/pull/56178#discussion_r3348702672


##########
python/pyspark/sql/streaming/state.py:
##########
@@ -170,21 +248,13 @@ def remove(self) -> None:
         self._updated = False
         self._removed = True
 
-    def setTimeoutDuration(self, durationMs: int) -> None:
+    def setTimeoutDuration(self, durationMs: Union[int, str]) -> None:
         """
         Set the timeout duration in ms for this key.
         Processing time timeout must be enabled.

Review Comment:
   Good point, added! A `versionchanged` note is useful here because this is a behavioral change to an existing method — users upgrading from an older Spark version would not know that string durations are now accepted unless the API docs call it out explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56178:
URL: https://github.com/apache/spark/pull/56178#issuecomment-6243856360

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #56178: [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration
URL: https://github.com/apache/spark/pull/56178


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on PR #56178:
URL: https://github.com/apache/spark/pull/56178#issuecomment-4925961264

   @HeartSaVioR — pinging again in case this got lost. This extends `GroupState.setTimeoutDuration` in PySpark to accept a Spark interval string (e.g. `"5 minutes"`, `"1 hour 30 minutes"`) in addition to integer milliseconds, matching the existing Scala API overload. Would appreciate your review when you have bandwidth. Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on code in PR #56178:
URL: https://github.com/apache/spark/pull/56178#discussion_r3348702672


##########
python/pyspark/sql/streaming/state.py:
##########
@@ -170,21 +248,13 @@ def remove(self) -> None:
         self._updated = False
         self._removed = True
 
-    def setTimeoutDuration(self, durationMs: int) -> None:
+    def setTimeoutDuration(self, durationMs: Union[int, str]) -> None:
         """
         Set the timeout duration in ms for this key.
         Processing time timeout must be enabled.

Review Comment:
   Done — added `.. versionchanged:: 5.0.0` noting that `durationMs` now also accepts a Spark interval string. Also validated the docstring compiles cleanly under Sphinx.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "brijrajk (via GitHub)" <gi...@apache.org>.
brijrajk commented on code in PR #56178:
URL: https://github.com/apache/spark/pull/56178#discussion_r3454319886


##########
python/pyspark/sql/streaming/state.py:
##########
@@ -170,21 +248,13 @@ def remove(self) -> None:
         self._updated = False
         self._removed = True
 
-    def setTimeoutDuration(self, durationMs: int) -> None:
+    def setTimeoutDuration(self, durationMs: Union[int, str]) -> None:
         """
         Set the timeout duration in ms for this key.
         Processing time timeout must be enabled.

Review Comment:
   Done, added `versionchanged:: 4.3.0` (corrected from 5.0.0 — since this is a normally backported change, the right version is branch-4.x's 4.3.0).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-40437][SS][PYTHON] Support string representation of durationMs in GroupState.setTimeoutDuration [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56178:
URL: https://github.com/apache/spark/pull/56178#issuecomment-4612171145

   I think @HyukjinKwon and @HeartSaVioR should have more context as per the discussion in https://issues.apache.org/jira/browse/SPARK-40437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org