[jira] [Commented] (SPARK-57581) Encode the TIME data type in Avro with a unit-correct logical type

["Max Gekk (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-57581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090363#comment-18090363 ] 

Max Gekk commented on SPARK-57581:
----------------------------------

Upstream dependency note: this ticket makes the Avro TIME encoding unit-correct for precision 0-6 by using time-micros (write micros, read micros->nanos). Making TIME(7-9) externally unit-correct as well is blocked upstream: Avro has no nanosecond time-of-day logical type (it only has timestamp-nanos / local-timestamp-nanos). That is tracked by AVRO-4043 ("Add time-nanos logical type", spec PR apache/avro#3125, Java PR apache/avro#3158). Until Avro ships time-nanos, TIME(7-9) values cannot be stored to Avro without truncation to microseconds. Linking AVRO-4043 as related.

> Encode the TIME data type in Avro with a unit-correct logical type
> ------------------------------------------------------------------
>
>                 Key: SPARK-57581
>                 URL: https://issues.apache.org/jira/browse/SPARK-57581
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Max Gekk
>            Assignee: Max Gekk
>            Priority: Major
>              Labels: pull-request-available
>
> h2. What
> Spark's Avro encoding for the TIME data type writes the internal value (nanoseconds since midnight, INT64) but annotates the column with the Avro {{time-micros}} logical type. The declared unit (microseconds) does not match the stored unit (nanoseconds).
> Relevant code:
> * {{SchemaConverters.scala}}: {{case t: TimeType => LogicalTypes.timeMicros().addToSchema(...)}} plus a {{spark.sql.catalyst.type = time(p)}} property.
> * {{AvroSerializer.scala}}: {{case (_: TimeType, LONG) => getter.getLong(ordinal)}} (writes raw nanos).
> * {{AvroDeserializer.scala}}: {{case (LONG, _: TimeType) => updater.setLong(ordinal, value)}} (reads raw nanos).
> Spark-to-Spark round-trips are correct because both the writer and reader treat the value as a raw Long and recover the precision from the {{spark.sql.catalyst.type}} property. However, the value is mislabeled for any consumer that honors the Avro {{time-micros}} logical type.
> h2. Why
> An external Avro reader (Hive, Trino, Flink, fastavro, etc.) decoding a Spark-written TIME column interprets the INT64 as microseconds-since-midnight, while it is actually nanoseconds-since-midnight - a 1000x error that also falls outside the valid micros-of-day range. This affects *all* precisions (0-9), not only 7-9, because even TIME(6) stores nanoseconds under {{time-micros}}.
> For comparison, the Parquet path is unit-correct: it emits TIME(MICROS) with a microsecond value for precision 0-6 and TIME(NANOS) with a nanosecond value for precision 7-9 (SPARK-57551), so Parquet files are interpretable by external tools.
> h2. Scope
> * Write path: emit a logical type whose unit matches the written value:
> ** precision 0-6: {{time-micros}}, write microseconds (nanos -> micros).
> ** precision 7-9: {{time-nanos}}, write nanoseconds (if the bundled Avro version exposes a nanosecond TIME logical type; otherwise document/choose an alternative encoding).
> * Read path: convert by the declared unit - {{time-micros}} -> micros -> nanos, {{time-nanos}} -> nanos (truncated to the requested precision). Keep honoring the {{spark.sql.catalyst.type}} property for precision fidelity.
> * Tests: assert that a plain (non-Spark) Avro decode yields the correct hour/minute/second/fraction, in addition to the existing Spark round-trip tests.
> h2. Backward compatibility (important)
> Changing the on-disk encoding is a format change. Avro files already written by current Spark store nanoseconds under {{time-micros}}; a new unit-correct reader would misread those legacy files (treating nanoseconds as microseconds). The fix must define how legacy files are detected/handled - e.g. keying off the {{spark.sql.catalyst.type}} property or a writer-version marker, or accepting the break given how recently TIME-in-Avro shipped. This migration question is the main design decision for the ticket.
> h2. Out of scope
> * The TIME precision extension itself (SPARK-57551).
> * Non-Avro datasources (Parquet/ORC already correct).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org