You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zeruibao (via GitHub)" <gi...@apache.org> on 2025/12/03 22:04:18 UTC
[PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithStateInPandas [spark]
zeruibao opened a new pull request, #53314:
URL: https://github.com/apache/spark/pull/53314
### What changes were proposed in this pull request?
Fix type handling of namedTuple for transfromWithStateInPandas.
### Why are the changes needed?
The customers hit the issue when using namedTuple as value of structType like
```
class BillingAddress(NamedTuple):
address_id: int
transaction_id: int
def handleInputRows(
self,
key: Any,
rows: Iterator[Row],
timerValues: TimerValues
) -> Iterator[Row]:
address: BillingAddress = BillingAddress(address_id = 1, transaction_id= 1)
address_list = []
address_list.append(address)
self.addresses.update((address_list,))
```
The `_serialize_to_bytes` cannot construct the namedTuple correctly and hit
```
File "/databricks/spark/python/pyspark/sql/streaming/stateful_processor_api_client.py", line 575, in normalize_value
return type(v)(normalize_value(e) for e in v)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BillingAddress.__new__() missing 3 required positional arguments: 'event_timestamp', 'processing_timestamp', and 'transaction_id'
```
It's because NamedTuple cannot accept generator as parameter.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
UT
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2587006200
##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -501,6 +501,11 @@ def normalize_value(v: Any) -> Any:
# Convert NumPy types to Python primitive types.
if isinstance(v, np.generic):
return v.tolist()
+ # Named tuples (collections.namedtuple or typing.NamedTuple) have a
+ # _fields attribute. Spark Row has __fields__. Both require positional
+ # arguments and cannot be instantiated with a generator expression.
+ if hasattr(v, '_fields') or hasattr(v, '__fields__'):
Review Comment:
Hey @gaogaotiantian, I use
```
if (
isinstance(v, Row) or
(isinstance(v, tuple) and hasattr(v, "_fields"))
):
```
instead. `isinstance(v, NamedTuple)` won’t work because `typing.NamedTuple` is a class factory, not a runtime parent of instances. Checking `isinstance(v, tuple)` and _fields is the correct way. Please take another look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192][SS][PYTHON] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2593997859
##########
python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py:
##########
@@ -1663,10 +1663,17 @@ def close(self) -> None:
# A stateful processor that contains composite python type inside Value, List and Map state variable
class PandasStatefulProcessorCompositeType(StatefulProcessor):
+ from typing import NamedTuple
Review Comment:
you are right. Just moved it the top!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192][SS][PYTHON] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2593927376
##########
python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py:
##########
@@ -1663,10 +1663,17 @@ def close(self) -> None:
# A stateful processor that contains composite python type inside Value, List and Map state variable
class PandasStatefulProcessorCompositeType(StatefulProcessor):
+ from typing import NamedTuple
Review Comment:
Can we move this to the top of the test module? Is there a specific reason that you want it in the class definition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2586835509
##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -501,6 +501,11 @@ def normalize_value(v: Any) -> Any:
# Convert NumPy types to Python primitive types.
if isinstance(v, np.generic):
return v.tolist()
+ # Named tuples (collections.namedtuple or typing.NamedTuple) have a
+ # _fields attribute. Spark Row has __fields__. Both require positional
+ # arguments and cannot be instantiated with a generator expression.
+ if hasattr(v, '_fields') or hasattr(v, '__fields__'):
Review Comment:
I think checking the type is much safer than checking an attribute, especially considering that `_fields` is a not a rare attribute name. If we know what type we are targeting, we should just check type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2586850105
##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -501,6 +501,11 @@ def normalize_value(v: Any) -> Any:
# Convert NumPy types to Python primitive types.
if isinstance(v, np.generic):
return v.tolist()
+ # Named tuples (collections.namedtuple or typing.NamedTuple) have a
+ # _fields attribute. Spark Row has __fields__. Both require positional
+ # arguments and cannot be instantiated with a generator expression.
+ if hasattr(v, '_fields') or hasattr(v, '__fields__'):
Review Comment:
sounds good! You are so fast. Did not get a chance to add a UT yet 😛 just convert to draft.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51920][SS][PYTHON] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #53314: [SPARK-51920][SS][PYTHON] Fix type handling of namedTuple for transfromWithState
URL: https://github.com/apache/spark/pull/53314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2587027074
##########
python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py:
##########
@@ -1663,10 +1663,17 @@ def close(self) -> None:
# A stateful processor that contains composite python type inside Value, List and Map state variable
class PandasStatefulProcessorCompositeType(StatefulProcessor):
+
+ from typing import NamedTuple
+ class Address(NamedTuple):
+ road_id: int
+ block_id: int
Review Comment:
sure sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5192] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "bogao007 (via GitHub)" <gi...@apache.org>.
bogao007 commented on code in PR #53314:
URL: https://github.com/apache/spark/pull/53314#discussion_r2587023915
##########
python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py:
##########
@@ -1663,10 +1663,17 @@ def close(self) -> None:
# A stateful processor that contains composite python type inside Value, List and Map state variable
class PandasStatefulProcessorCompositeType(StatefulProcessor):
+
+ from typing import NamedTuple
+ class Address(NamedTuple):
+ road_id: int
+ block_id: int
Review Comment:
For the test `Address` class, can we use different types? E.g. one with `int` type and the other one with `string` type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51920][SS][PYTHON] Fix type handling of namedTuple for transfromWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #53314:
URL: https://github.com/apache/spark/pull/53314#issuecomment-3641861778
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org