You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2025/09/12 01:50:47 UTC
[PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
ueshin opened a new pull request, #52321:
URL: https://github.com/apache/spark/pull/52321
### What changes were proposed in this pull request?
Supports complex types on observations.
### Why are the changes needed?
The observations didn't support complex types.
For example:
```py
>>> observation = Observation("struct")
>>> df = spark.range(10).observe(
... observation,
... F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
... )
```
- classic
```py
>>> df.collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]
>>> observation.get
{'struct': JavaObject id=o61}
```
- connect
```py
>>> df.collect()
Traceback (most recent call last):
...
pyspark.errors.exceptions.base.PySparkTypeError: [UNSUPPORTED_LITERAL] Unsupported Literal 'struct {
...
```
### Does this PR introduce _any_ user-facing change?
Yes, complex types are available on observations.
```py
>>> df.collect()
[Row(id=0), Row(id=1), Row(id=2), Row(id=3), Row(id=4), Row(id=5), Row(id=6), Row(id=7), Row(id=8), Row(id=9)]
>>>
>>> observation.get
{'struct': Row(rows=10, maxid=9)}
```
### How was this patch tested?
Added the related tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2345630820
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
For example of `Array`, `data_type` was added and `element_type` was deprecated.
https://github.com/apache/spark/blob/master/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto#L227-L244
but the creator of the message may follow it or may not, so this should be able to handle both cases.
Same for `Map` and `Struct`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2346760811
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
To minimize changes, it is not necessary to support these new data type fields. Currently, the new data type fields are only used in the requests from the Spark Connect Scala Client.
If we really want to support them, I think we need to consider:
- How can we enable the new data type fields in the responses while maintaining backward compatibility?
- How is this code path tested?
- Do the added tests cover it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2377517321
##########
python/pyspark/sql/tests/test_observation.py:
##########
@@ -190,10 +186,46 @@ def test_observe_on_commands(self):
]:
with self.subTest(command=command):
observation = Observation()
- observed_df = df.observe(observation, count(lit(1)).alias("cnt"))
+ observed_df = df.observe(observation, F.count(F.lit(1)).alias("cnt"))
action(observed_df)
self.assertEqual(observation.get, dict(cnt=50))
+ def test_observe_with_struct_type(self):
+ observation = Observation("struct")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"struct": Row(rows=10, maxid=9)})
+
+ def test_observe_with_array_type(self):
+ observation = Observation("array")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.array(F.count(F.lit(1))).alias("array"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"array": [10]})
+
+ def test_observe_with_map_type(self):
Review Comment:
This has been stable since this was merged. I guess ~another commit broke this~ or any environment changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52321:
URL: https://github.com/apache/spark/pull/52321#issuecomment-3289999832
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2377517321
##########
python/pyspark/sql/tests/test_observation.py:
##########
@@ -190,10 +186,46 @@ def test_observe_on_commands(self):
]:
with self.subTest(command=command):
observation = Observation()
- observed_df = df.observe(observation, count(lit(1)).alias("cnt"))
+ observed_df = df.observe(observation, F.count(F.lit(1)).alias("cnt"))
action(observed_df)
self.assertEqual(observation.get, dict(cnt=50))
+ def test_observe_with_struct_type(self):
+ observation = Observation("struct")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"struct": Row(rows=10, maxid=9)})
+
+ def test_observe_with_array_type(self):
+ observation = Observation("array")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.array(F.count(F.lit(1))).alias("array"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"array": [10]})
+
+ def test_observe_with_map_type(self):
Review Comment:
This has been stable since this was merged. I guess another commit broke this or any environment changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2347012535
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
Sure, I'll revert the changes and leave it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2346760811
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
To minimize changes, it is not necessary to support these new data type fields. They are still under development and not fully stabilized yet. Currently, the new data type fields are only used in the requests from the Spark Connect Scala Client.
If we really want to support them, I think we need to consider:
- How can we enable the new data type fields in the responses while maintaining backward compatibility?
- How is this code path tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #52321: [SPARK-53544][PYTHON] Support complex types on observations
URL: https://github.com/apache/spark/pull/52321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #52321:
URL: https://github.com/apache/spark/pull/52321#issuecomment-3283361084
cc @heyihong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2345625698
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
Curious when does data_type exist and when does it not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2377518844
##########
python/pyspark/sql/tests/test_observation.py:
##########
@@ -190,10 +186,46 @@ def test_observe_on_commands(self):
]:
with self.subTest(command=command):
observation = Observation()
- observed_df = df.observe(observation, count(lit(1)).alias("cnt"))
+ observed_df = df.observe(observation, F.count(F.lit(1)).alias("cnt"))
action(observed_df)
self.assertEqual(observation.get, dict(cnt=50))
+ def test_observe_with_struct_type(self):
+ observation = Observation("struct")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"struct": Row(rows=10, maxid=9)})
+
+ def test_observe_with_array_type(self):
+ observation = Observation("array")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.array(F.count(F.lit(1))).alias("array"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"array": [10]})
+
+ def test_observe_with_map_type(self):
Review Comment:
Also I can't repro this in my local.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2346760811
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
To minimize changes, it is not necessary to support these new data type fields. Currently, the new data type fields are only used in the requests from the Spark Connect Scala Client.
If we really want to support them, I think we need to consider:
- How can we enable the new data type fields in the responses while maintaining backward compatibility?
- How is this code path tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2418339686
##########
python/pyspark/sql/tests/test_observation.py:
##########
@@ -190,10 +186,46 @@ def test_observe_on_commands(self):
]:
with self.subTest(command=command):
observation = Observation()
- observed_df = df.observe(observation, count(lit(1)).alias("cnt"))
+ observed_df = df.observe(observation, F.count(F.lit(1)).alias("cnt"))
action(observed_df)
self.assertEqual(observation.get, dict(cnt=50))
+ def test_observe_with_struct_type(self):
+ observation = Observation("struct")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"struct": Row(rows=10, maxid=9)})
+
+ def test_observe_with_array_type(self):
+ observation = Observation("array")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.array(F.count(F.lit(1))).alias("array"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"array": [10]})
+
+ def test_observe_with_map_type(self):
Review Comment:
@ueshin another failure with JVM stacktrace
https://github.com/apache/spark/actions/runs/18387627760/job/52390087983
```
JVM stacktrace:
java.lang.IllegalArgumentException
at scala.Predef$.require(Predef.scala:324)
at org.apache.spark.sql.catalyst.util.ArrayBasedMapData.<init>(ArrayBasedMapData.scala:31)
at org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.build(ArrayBasedMapBuilder.scala:130)
at org.apache.spark.sql.catalyst.expressions.CreateMap.eval(complexTypeCreator.scala:260)
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:162)
at org.apache.spark.sql.catalyst.expressions.InterpretedUnsafeProjection.apply(InterpretedUnsafeProjection.scala:84)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2347012535
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
Sure, I'll revert the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2346760811
##########
python/pyspark/sql/connect/expressions.py:
##########
@@ -436,11 +442,48 @@ def _to_value(
assert dataType is None or isinstance(dataType, DayTimeIntervalType)
return DayTimeIntervalType().fromInternal(literal.day_time_interval)
elif literal.HasField("array"):
- elementType = proto_schema_to_pyspark_data_type(literal.array.element_type)
+ if literal.array.HasField("data_type"):
Review Comment:
How is this code path tested? Does the added test cover it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53544][PYTHON] Support complex types on observations [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52321:
URL: https://github.com/apache/spark/pull/52321#discussion_r2377491457
##########
python/pyspark/sql/tests/test_observation.py:
##########
@@ -190,10 +186,46 @@ def test_observe_on_commands(self):
]:
with self.subTest(command=command):
observation = Observation()
- observed_df = df.observe(observation, count(lit(1)).alias("cnt"))
+ observed_df = df.observe(observation, F.count(F.lit(1)).alias("cnt"))
action(observed_df)
self.assertEqual(observation.get, dict(cnt=50))
+ def test_observe_with_struct_type(self):
+ observation = Observation("struct")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.struct(F.count(F.lit(1)).alias("rows"), F.max("id").alias("maxid")).alias("struct"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"struct": Row(rows=10, maxid=9)})
+
+ def test_observe_with_array_type(self):
+ observation = Observation("array")
+
+ df = self.spark.range(10).observe(
+ observation,
+ F.array(F.count(F.lit(1))).alias("array"),
+ )
+
+ assertDataFrameEqual(df, [Row(id=id) for id in range(10)])
+
+ self.assertEqual(observation.get, {"array": [10]})
+
+ def test_observe_with_map_type(self):
Review Comment:
@ueshin this test occasionally fails (e.g. https://github.com/apache/spark/actions/runs/17988107433/job/51171506921)
```
======================================================================
FAIL [0.159s]: test_observe_with_map_type (pyspark.sql.tests.connect.test_parity_observation.DataFrameObservationParityTests.test_observe_with_map_type)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/sql/tests/test_observation.py", line 227, in test_observe_with_map_type
self.assertEqual(observation.get, {"map": {"count": 10}})
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'map': {'': 10}} != {'map': {'count': 10}}
- {'map': {'': 10}}
+ {'map': {'count': 10}}
? +++++
----------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org