You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/16 09:00:00 UTC
[jira] [Updated] (SPARK-59571) Cover every TIME precision, null pattern and interval sign in ArrowCachedBatchSerializerSuite
[ https://issues.apache.org/jira/browse/SPARK-59571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-59571:
-----------------------------------
Labels: pull-request-available (was: )
> Cover every TIME precision, null pattern and interval sign in ArrowCachedBatchSerializerSuite
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-59571
> URL: https://issues.apache.org/jira/browse/SPARK-59571
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Minor
> Labels: pull-request-available
>
> h3. What
> Add two tests to \{{ArrowCachedBatchSerializerSuite}} for the Arrow cache serializer (SPARK-57268) over the \{{TIME}} and \{{INTERVAL DAY TO SECOND}} types:
> * \{{TIME(p)}} for p in 0, 3, 6 and 9, each at three null patterns (every 31st row null, no nulls, all nulls), a thousand rows in one partition. Each case checks that the cached frame answers what the uncached one did, that the cached \{{InMemoryRelation}} keeps \{{TimeType(p)}} with the declared precision, and that the serializer's own columnar read path yields a \{{TimeNanoVector}} per batch whose row and null counts add up to the input's. The values are truncated to the declared precision so a precision loss could not hide behind a value the type would round anyway.
> * \{{INTERVAL DAY TO SECOND}} with whole microseconds of both signs, at the same three null patterns, checking the same three things against a \{{DurationVector}}.
> h3. Why
> The suite today round-trips \{{TIME}} once, at precision 6, over two non-null rows, and checks its \{{LongColumnStats}}. Every precision of \{{TIME}} is written to the same \{{TimeNanoVector}}, and the precision travels only in the Arrow field metadata, so a regression that dropped it on the way back through the cache would leave the values intact and fail no existing test. The all-null column takes a branch of its own in the vector (no validity buffer), and negative intervals are not covered at all. \{{TIME}} is about to be enabled by default, and the Arrow cache is one of the paths it takes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org