You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/25 05:00:37 UTC
(spark) branch branch-4.x updated: [SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in golden files
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 27e2411b99f8 [SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in golden files
27e2411b99f8 is described below
commit 27e2411b99f8eaee3d563bcdb445bb91630f8d0f
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Tue Aug 25 13:00:09 2026 +0800
[SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in golden files
### What changes were proposed in this pull request?
Add a `ValueError` branch to `_scalar_str` in `python/pyspark/testing/goldenutils.py` so a `time64[ns]` scalar holding INT64_MIN renders as the marker `NaT collision`, beside the existing `temporal overflow` and raw-bytes fallbacks, and regenerate the four affected `from_pandas` type-scalar golden cells (`int64:standard` and `Int64:standard` x `time64[ns]`, safe and unsafe).
Rendering a `time64[ns]` scalar goes through pandas, because Python's `datetime.time` has only microsecond resolution. pandas reserves INT64_MIN as its NaT sentinel, so it refuses that one value with `ValueError: NaTType does not support time`. The branch is gated on both the type and the value, so any other `ValueError` still propagates.
### Why are the changes needed?
`PyArrowArrayFromPandasTypeScalarTests` currently fails on master. The four cells above read `ERRValueError`, which by convention means the conversion raised — but the conversion succeeds and only the rendering failed. The broad `try/except` in the cell helper recorded the rendering failure as if it were a conversion error, so narrowing that `try/except` in #58091 turned the mislabelling into a test error.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
`python/pyspark/tests/upstream/pyarrow/` passes (63 passed, 1 skipped), and a sweep over pyarrow 18-25 x pandas 2/3 is 16/16 with no `overrides` entries.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58266 from Spenserrrr/golden-nat-sentinel.
Authored-by: Spenser Sun <hs...@gmail.com>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
(cherry picked from commit 08e97ded9430aa7dc931936ebbfe5293404e6cba)
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
python/pyspark/testing/goldenutils.py | 16 +++++++--
..._pyarrow_array_from_pandas_type_scalar_safe.csv | 4 +--
...n_pyarrow_array_from_pandas_type_scalar_safe.md | 42 +++++++++++-----------
...yarrow_array_from_pandas_type_scalar_unsafe.csv | 4 +--
...pyarrow_array_from_pandas_type_scalar_unsafe.md | 42 +++++++++++-----------
5 files changed, 59 insertions(+), 49 deletions(-)
diff --git a/python/pyspark/testing/goldenutils.py b/python/pyspark/testing/goldenutils.py
index f9ba1f66fd98..6d60c4a8be92 100644
--- a/python/pyspark/testing/goldenutils.py
+++ b/python/pyspark/testing/goldenutils.py
@@ -316,13 +316,16 @@ class GoldenFileTestMixin:
"""
Render one PyArrow scalar for a golden cell via PyArrow's own ``str(scalar)``.
- Some values are valid in Arrow yet unrenderable via ``str``:
+ Some values that Arrow stores are unrenderable via ``str``:
- a temporal value past Python's ``datetime`` range (e.g. date32 past year 9999)
raises ``OverflowError`` -> the marker ``temporal overflow``;
- an unsafe ``binary``->``string`` cast relabels bytes without UTF-8 validation,
so non-UTF-8 bytes raise ``UnicodeDecodeError`` -> the raw bytes, so the golden
- still tracks what Arrow stored for them.
- An unexpected error (non-temporal overflow, non-string decode error) propagates.
+ still tracks what Arrow stored for them;
+ - a nanosecond ``time64`` holding INT64_MIN collides with pandas' NaT sentinel and
+ raises ``ValueError`` -> the marker ``NaT collision``.
+ An unexpected error (non-temporal overflow, non-string decode error, or a
+ ValueError from any other value) propagates.
"""
try:
return str(scalar).replace("\x00", "\\0")
@@ -341,6 +344,13 @@ class GoldenFileTestMixin:
if pa.types.is_string(scalar.type) or pa.types.is_large_string(scalar.type):
return repr(scalar.as_buffer().to_pybytes())
raise
+ except ValueError:
+ # A nanosecond time64 renders through pandas (Python's ``time`` is microsecond
+ # resolution), and pandas reads INT64_MIN as its NaT sentinel, so it refuses
+ # that one value. Any other ValueError is unexpected, so re-raise.
+ if pa.types.is_time(scalar.type) and scalar.value == pd.NaT.value:
+ return "NaT collision"
+ raise
@classmethod
def repr_arrow_value(
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
index a054d3a83712..0ef13693498d 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day] duration[us] time64[ns] string binary
-int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff', b'\xff\x [...]
+int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:59:59.999999, 23:47:16.854775, NaT collision]@time64[ns] ERR@ArrowType [...]
float64:standard ERR@ArrowInvalid ERR@ArrowInvalid [0.0, 1.5, -1.5]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string [b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us] [2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day, 2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:47:16.854775, NaT collision]@time64[ns] ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
int64:overflow ERR@ArrowInvalid [300, 1]@int64 [300.0, 1.0]@float32 [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us] [00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01', b'\x01']@binary
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
index 65476ad0e9e9..2157f49343ac 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64 | float32 | timestamp[us] | date32[day] | duration[us] | time64[ns] [...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------------------- [...]
-| int64:standard | ERR@ArrowInvalid | [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | ERR@ValueError [...]
-| float64:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | [0.0, 1.5, -1.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0, 1]@int64 | [1.0, 0.0, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| object:string | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| object:bytes | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| date:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-06-15]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| object:datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | [2024-06-15 18:30:00]@timestamp[us] | [2024-06-15]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| object:timedelta | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us] | ERR@ArrowTypeError [...]
-| time:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | [18:30:45]@time64[ns] [...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | ERR@ValueError [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| int64:overflow | ERR@ArrowInvalid | [300, 1]@int64 | [300.0, 1.0]@float32 | [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00.000300, 0:00:00.000001]@duration[us] | [00:00:00, 00:00:00]@time64[ns [...]
-| float64:fractional | ERR@ArrowInvalid | ERR@ArrowInvalid | [1.5, 2.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| float64:infinity | ERR@ArrowInvalid | ERR@ArrowInvalid | [inf, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| float64:precision | ERR@ArrowInvalid | ERR@ArrowInvalid | [1.1234568357467651]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError | [1704067200000000123]@int64 | ERR@ArrowNotImplementedError | ERR@ArrowInvalid | [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError | [00:00:00]@time64[ns] [...]
-| object:date-then-datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| object:datetime-then-date | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
\ No newline at end of file
+| source \ target | int8 | int64 | float32 | timestamp[us] | date32[day] | duration[us] | time64[ns] [...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------------------- [...]
+| int64:standard | ERR@ArrowInvalid | [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | [00:00:00, 00:00:00, 23:59:59. [...]
+| float64:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | [0.0, 1.5, -1.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0, 1]@int64 | [1.0, 0.0, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| object:string | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| object:bytes | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| date:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-06-15]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| object:datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | [2024-06-15 18:30:00]@timestamp[us] | [2024-06-15]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| object:timedelta | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us] | ERR@ArrowTypeError [...]
+| time:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError | [18:30:45]@time64[ns] [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | [00:00:00, 00:00:00, 23:47:16. [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| int64:overflow | ERR@ArrowInvalid | [300, 1]@int64 | [300.0, 1.0]@float32 | [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00.000300, 0:00:00.000001]@duration[us] | [00:00:00, 00:00:00]@time64[ns [...]
+| float64:fractional | ERR@ArrowInvalid | ERR@ArrowInvalid | [1.5, 2.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| float64:infinity | ERR@ArrowInvalid | ERR@ArrowInvalid | [inf, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| float64:precision | ERR@ArrowInvalid | ERR@ArrowInvalid | [1.1234568357467651]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError | [1704067200000000123]@int64 | ERR@ArrowNotImplementedError | ERR@ArrowInvalid | [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError | [00:00:00]@time64[ns] [...]
+| object:date-then-datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| object:datetime-then-date | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
index b04f68c6e346..32a24d6a661c 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day] duration[us] time64[ns] string binary
-int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18, -9.223372036854776e+18]@float32 [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [...]
+int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18, -9.223372036854776e+18]@float32 [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:59:59.999 [...]
float64:standard [0, 1, -1]@int8 [0, 1, -1]@int64 [0.0, 1.5, -1.5]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string [b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us] [2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day, 2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:47:16.854775, NaT collision]@time64[ns] ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
int64:overflow [44, 1]@int8 [300, 1]@int64 [300.0, 1.0]@float32 [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us] [00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01', b'\x01']@binary
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
index 05ddfff08f59..a6f12040dc1d 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64 | float32 | timestamp[us] | date32[day] | duration[us] [...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------ [...]
-| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0, 9.223372036854776e+18, -9.223372036854776e+18]@float32 | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@du [...]
-| float64:standard | [0, 1, -1]@int8 | [0, 1, -1]@int64 | [0.0, 1.5, -1.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0, 1]@int64 | [1.0, 0.0, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| object:string | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| object:bytes | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| date:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-06-15]@date32[day] | ERR@ArrowTypeError [...]
-| object:datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | [2024-06-15 18:30:00]@timestamp[us] | [2024-06-15]@date32[day] | ERR@ArrowTypeError [...]
-| object:timedelta | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us] [...]
-| time:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError [...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| int64:overflow | [44, 1]@int8 | [300, 1]@int64 | [300.0, 1.0]@float32 | [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00.000300, 0:00:00.000001]@duration[us] [...]
-| float64:fractional | [1, 2]@int8 | [1, 2]@int64 | [1.5, 2.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| float64:infinity | [0, 1]@int8 | [-9223372036854775808, 1]@int64 | [inf, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| float64:precision | [1]@int8 | [1]@int64 | [1.1234568357467651]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError | [1704067200000000123]@int64 | ERR@ArrowNotImplementedError | [2024-01-01 00:00:00]@timestamp[us] | [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError [...]
-| object:date-then-datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError [...]
-| object:datetime-then-date | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError [...]
\ No newline at end of file
+| source \ target | int8 | int64 | float32 | timestamp[us] | date32[day] | duration[us] [...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------ [...]
+| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0, 9.223372036854776e+18, -9.223372036854776e+18]@float32 | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@du [...]
+| float64:standard | [0, 1, -1]@int8 | [0, 1, -1]@int64 | [0.0, 1.5, -1.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0, 1]@int64 | [1.0, 0.0, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| object:string | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| object:bytes | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| date:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-06-15]@date32[day] | ERR@ArrowTypeError [...]
+| object:datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | [2024-06-15 18:30:00]@timestamp[us] | [2024-06-15]@date32[day] | ERR@ArrowTypeError [...]
+| object:timedelta | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us] [...]
+| time:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | ERR@ArrowTypeError | ERR@ArrowTypeError [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1, 9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid | [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| int64:overflow | [44, 1]@int8 | [300, 1]@int64 | [300.0, 1.0]@float32 | [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError | [0:00:00.000300, 0:00:00.000001]@duration[us] [...]
+| float64:fractional | [1, 2]@int8 | [1, 2]@int64 | [1.5, 2.5]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| float64:infinity | [0, 1]@int8 | [-9223372036854775808, 1]@int64 | [inf, 1.0]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| float64:precision | [1]@int8 | [1]@int64 | [1.1234568357467651]@float32 | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError | [1704067200000000123]@int64 | ERR@ArrowNotImplementedError | [2024-01-01 00:00:00]@timestamp[us] | [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError [...]
+| object:date-then-datetime | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError [...]
+| object:datetime-then-date | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowInvalid | ERR@ArrowTypeError | [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org