You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2026/06/19 21:49:29 UTC
[PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
MaxGekk opened a new pull request, #56622:
URL: https://github.com/apache/spark/pull/56622
### What changes were proposed in this pull request?
This PR extends the fractional-seconds precision of the `TIME` data type from a maximum of 6 (microseconds) to 9 (nanoseconds), so `TIME(p)` accepts `0 <= p <= 9`. The internal storage is already nanoseconds-since-midnight (`Long`, `TimeType.NANOS_PRECISION = 9` already exists), so this lifts the cap and closes the two remaining micros-only code paths:
- `TimeType.MAX_PRECISION` is raised from 6 to 9 (`DEFAULT_PRECISION` stays 6). The `UNSUPPORTED_TIME_PRECISION` error message and related scaladoc/comments are updated to `[0, 9]`.
- `SparkDateTimeUtils.stringToTime` now keeps the sub-microsecond digits (7-9), mirroring the timestamp parser, and `CAST(<string> AS TIME(p))` truncates the parsed value to the target precision (interpreted and codegen paths).
- `CurrentTime` accepts precisions up to `MAX_PRECISION`.
- Parquet I/O emits/reads `TIME(NANOS)` for precision 7..9 (and keeps `TIME(MICROS)` for 0..6) across `TimeTypeParquetOps`, `ParquetSchemaConverter`, the vectorized `ParquetVectorUpdaterFactory`, and the legacy row/write fallbacks. ORC and Avro already store the raw nanosecond `Long` with the catalyst type name preserved, so they round-trip 7..9 without production changes.
Casts that were already nanosecond-aware (`TIME(p1) -> TIME(p2)`, `TIME -> DECIMAL`, `TIME -> integral`, `TIME -> STRING`) work for 7..9 once the cap is lifted.
Out of scope (tracked separately): SQL golden files (SPARK-57563), casts to/from TIMESTAMP types (SPARK-57552 / SPARK-57554), and `TIME WITH TIME ZONE` (SPARK-51162).
### Why are the changes needed?
ANSI SQL (ISO/IEC 9075-2, 6.1 `<data type>`) makes the maximum `<time precision>` implementation-defined with the sole constraint that it is not less than 6, and requires the maximum of `<time precision>` and `<timestamp precision>` to be the same implementation-defined value. Spark already supports nanosecond timestamps, so to stay ANSI-consistent `TIME` must reach precision 9 in lockstep.
### Does this PR introduce _any_ user-facing change?
Yes. `TIME(7)`, `TIME(8)`, and `TIME(9)` can now be declared, parsed, used as literals, and round-tripped through Parquet/ORC/Avro. Previously these precisions raised `UNSUPPORTED_TIME_PRECISION`. The default precision of `TIME` is unchanged (6).
### How was this patch tested?
Extended existing TIME suites to cover precision 7..9 (`DataTypeParserSuite`, `DataTypeSuite`, `TimeExpressionsSuite`, `CastWithAnsiOn/OffSuite`, `TimeFormatterSuite`, `DateTimeUtilsSuite`, CSV/JSON/XML expression and function suites, `TimeTypeParquetOpsSuite`, `ParquetIOSuite`, `OrcQuerySuite`, `AvroSuite`/`AvroFunctionsSuite`, `PartitionedWriteSuite`, `RowJsonSuite`, `SparkConnectPlannerSuite`) and added nanosecond Parquet read and round-trip tests (covering both the vectorized and non-vectorized readers). Ran the affected suites locally; all pass. `dev/scalastyle` and scalafmt are clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3449088420
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -165,8 +165,17 @@ public ParquetVectorUpdater getUpdater(ColumnDescriptor descriptor, DataType spa
return new LongUpdater();
} else if (canReadAsDecimal(descriptor, sparkType)) {
return new LongToDecimalUpdater(descriptor, (DecimalType) sparkType);
- } else if (sparkType instanceof TimeType) {
- return new LongAsNanosUpdater();
+ } else if (sparkType instanceof TimeType &&
+ isTimeTypeMatched(LogicalTypeAnnotation.TimeUnit.NANOS)) {
Review Comment:
Thanks @cloud-fan -- agreed the tightening is a user-visible change in principle, and you're right that the code did ship in 4.1.0/4.1.1/4.1.2.
One thing I'd like your read on before adding the flag: the entire TIME type is gated behind `spark.sql.timeType.enabled`, which is `.internal()` and `createWithDefault(Utils.isTesting)` -- i.e. off by default in production -- since it was introduced in 4.1.0:
```scala
val TIME_TYPE_ENABLED =
buildConf("spark.sql.timeType.enabled")
.internal()
.doc("When true, the TIME data type is supported.")
.version("4.1.0")
.booleanConf
.createWithDefault(Utils.isTesting)
```
So in the released versions the lenient vectorized path was never reachable on default settings -- hitting it requires flipping this internal/undocumented flag and then reading a non-`TIME`-annotated INT64 column with an explicit `TIME` read schema (a preview/test scenario). My read is that the legacy escape hatch + migration-guide entry are aimed at GA, default-on, user-visible behavior, so they may not be warranted for a change confined to a still-internal, default-off preview type -- similar to the reasoning on the Avro follow-up (SPARK-57581). The change itself is also strictly safer: it turns a silent 1000x-wrong micros mis-decode into a loud failure that matches the row-based reader's `requireCompatibleParquetType`.
That said, I'm happy to add it if you'd prefer the belt-and-suspenders option -- it's a small `.internal()` `spark.sql.legacy.parquet.lenientTimeInt64Read` (default `false`, `version("4.3.0")`) that falls back to the pre-PR `LongAsNanosUpdater` regardless of annotation, plus a `docs/sql-migration-guide.md` note. Let me know which way you'd like to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "davidm-db (via GitHub)" <gi...@apache.org>.
davidm-db commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4767542572
> The MICROS/NANOS bifurcation for `TimeType` is duplicated into three fallback branches that are effectively dead: `ParquetTypeOps(TimeType)` always returns `Some(TimeTypeParquetOps(...))`, so the framework-first dispatch (`convertField` / `makeWriter` / `newConverter`) always routes `TimeType` to `TimeTypeParquetOps` and these `case _: TimeType` fallbacks never run. The live logic is in `TimeTypeParquetOps`.
>
> * `ParquetSchemaConverter.scala:716` (`convertFieldDefault`)
> * `ParquetWriteSupport.scala:310` (`makeWriterDefault`)
> * `ParquetRowConverter.scala:529` (`newConverterDefault`)
>
> I'd prefer not to expand the dead copies further in this PR. Tracking the cleanup (drop the shadowed `TimeType` fallbacks so the framework op is the single source of truth, or confirm they're intentionally kept during the types-framework migration) under [SPARK-57582](https://issues.apache.org/jira/browse/SPARK-57582), a sub-task of the Types framework umbrella [SPARK-53504](https://issues.apache.org/jira/browse/SPARK-53504).
>
> cc @davidm-db
@MaxGekk fine by me to remove in the separate PR! The parquet change was merged after my PR that removed the flag and dead parts of the code, thus the confusion! I can follow-up to remove dead code from the parquet ops as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56622: [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9)
URL: https://github.com/apache/spark/pull/56622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3449784126
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -165,8 +165,17 @@ public ParquetVectorUpdater getUpdater(ColumnDescriptor descriptor, DataType spa
return new LongUpdater();
} else if (canReadAsDecimal(descriptor, sparkType)) {
return new LongToDecimalUpdater(descriptor, (DecimalType) sparkType);
- } else if (sparkType instanceof TimeType) {
- return new LongAsNanosUpdater();
+ } else if (sparkType instanceof TimeType &&
+ isTimeTypeMatched(LogicalTypeAnnotation.TimeUnit.NANOS)) {
Review Comment:
oh I see, off-by-default feature does not count, so we still have the chance to fix it without considering backward compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4760000469
@cloud-fan @HyukjinKwon @uros-b Could you review this PR, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3449059739
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -165,8 +165,17 @@ public ParquetVectorUpdater getUpdater(ColumnDescriptor descriptor, DataType spa
return new LongUpdater();
} else if (canReadAsDecimal(descriptor, sparkType)) {
return new LongToDecimalUpdater(descriptor, (DecimalType) sparkType);
- } else if (sparkType instanceof TimeType) {
- return new LongAsNanosUpdater();
+ } else if (sparkType instanceof TimeType &&
+ isTimeTypeMatched(LogicalTypeAnnotation.TimeUnit.NANOS)) {
Review Comment:
This now requires a `TIME(MICROS)`/`TIME(NANOS)` annotation, where the old `else if (sparkType instanceof TimeType)` matched any INT64 column and decoded it as micros. The tightening itself is good — it matches the row-based reader's `requireCompatibleParquetType` and turns a silent mis-decode into a loud failure — but it's a user-facing behavior change: a job that read such a file via the vectorized reader will now throw.
Per Spark convention, could we add a legacy escape hatch + migration note?
- a `spark.sql.legacy.parquet.lenientTimeInt64Read` (`.internal()`, `booleanConf`, `.createWithDefault(false)`, `.version("4.3.0")`, modeled on `LEGACY_PARQUET_RETURN_NULL_STRUCT_IF_ALL_FIELDS_MISSING`) that, when true, falls back here to the pre-PR lenient `LongAsNanosUpdater` (raw INT64 -> micros) regardless of annotation;
- a `docs/sql-migration-guide.md` entry noting the change and the restore flag.
(Version 4.3.0 assuming this backports to branch-4.x.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4758849809
@srielau Could you look at the PR, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3447604380
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -1376,6 +1388,17 @@ class ExpressionParserSuite extends AnalysisTest {
fragment = "time '12-13.14'",
start = 0,
stop = 14))
+
+ // More than 9 fractional-second digits is rejected.
Review Comment:
Intentional, and it mirrors how TIMESTAMP literals vs casts already behave in this same `AstBuilder` path:
- A **literal**'s fractional precision *is* its digit count (ANSI ISO/IEC 9075-2, Subclause 5.3, Syntax Rule 26), so `TIME '...1234567890'` declares a `TIME(10)` literal, which is out of range -> `INVALID_TIME_LITERAL_PRECISION`. This is the same rule that makes `TIMESTAMP_NTZ '...1234567890'` raise `INVALID_TIMESTAMP_LITERAL_PRECISION`.
- A **cast** targets an explicit `TIME(p)` whose resolution is capped at nanoseconds; the 10th digit is sub-nanosecond and simply cannot be represented, so the shared `parseTimestampString` drops digits beyond the 9th (general string->datetime parse leniency). `CAST('...1234567890' AS TIMESTAMP_NTZ(9))` behaves the same way.
So the split is by design: the literal path validates the ANSI literal precision, while the cast path inherits the parser's truncation of unrepresentable sub-nanosecond digits. I can add a short cast test asserting the >9 truncation to make the intent explicit if you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3447564860
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
Review Comment:
Minor performance note here, regarding per-value math.pow in the vectorized hot loop. TimeUpdater.toTruncatedNanos → truncateTimeToPrecision recomputes math.pow(10, scale).toLong (a float pow) plus an assert for every decoded value. The precision is constant per column, so the factor (and the assert) are invariant. In the vectorized reader — which exists precisely for throughput — this adds avoidable cost per value. Worth precomputing factor once in the TimeUpdater constructor (the row converter is less hot but could do the same).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4757395995
The MICROS/NANOS bifurcation for `TimeType` is duplicated into three fallback branches that are effectively dead: `ParquetTypeOps(TimeType)` always returns `Some(TimeTypeParquetOps(...))`, so the framework-first dispatch (`convertField` / `makeWriter` / `newConverter`) always routes `TimeType` to `TimeTypeParquetOps` and these `case _: TimeType` fallbacks never run. The live logic is in `TimeTypeParquetOps`.
- `ParquetSchemaConverter.scala:716` (`convertFieldDefault`)
- `ParquetWriteSupport.scala:310` (`makeWriterDefault`)
- `ParquetRowConverter.scala:529` (`newConverterDefault`)
I'd prefer not to expand the dead copies further in this PR. Tracking the cleanup (drop the shadowed `TimeType` fallbacks so the framework op is the single source of truth, or confirm they're intentionally kept during the types-framework migration) under [SPARK-57582](https://issues.apache.org/jira/browse/SPARK-57582), a sub-task of the Types framework umbrella [SPARK-53504](https://issues.apache.org/jira/browse/SPARK-53504).
cc @davidm-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4756785111
Thanks for the careful review, @shrirangmhalgi!
On the Avro `time-micros` design note: you're right that the value is mislabeled - Spark writes the internal nanoseconds-since-midnight `Long` but annotates the column with the `time-micros` logical type, so a consumer that honors that annotation would misread it.
Two clarifications on scope:
- It isn't specific to TIME(7-9). The value is stored as nanoseconds under `time-micros` for **all** precisions (even TIME(6)), so external readers are already affected today. This PR doesn't change the Avro path (`SchemaConverters`/`AvroSerializer`/`AvroDeserializer` are untouched); Spark-to-Spark round-trips stay correct because both sides treat the `Long` as raw nanos and recover precision from the `spark.sql.catalyst.type` property.
- A `time-nanos`-for-7-9-only bifurcation would make 7-9 externally correct but leave 0-6 still mislabeled. Making the encoding fully unit-correct is also a format change with a backward-compatibility angle: a new reader would misread Avro files already written by current Spark (nanos stored under `time-micros`), so legacy files need a detection/migration story.
Since it's pre-existing, broader than this change, and needs that migration decision, I've filed a follow-up to do the full unit-correct Avro encoding (all precisions): [SPARK-57581](https://issues.apache.org/jira/browse/SPARK-57581). I'd prefer to keep this PR focused on the precision-cap extension and tackle Avro there - does that work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3447568566
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala:
##########
@@ -1376,6 +1388,17 @@ class ExpressionParserSuite extends AnalysisTest {
fragment = "time '12-13.14'",
start = 0,
stop = 14))
+
+ // More than 9 fractional-second digits is rejected.
Review Comment:
One question here about Cast vs. literal handling of >9 fractional digits. Right now, it seems a bit inconsistent. e.g. CAST('12:34:56.1234567890' AS TIME(9)) silently drops digits beyond the 9th (per parseTimestampString), while the literal TIME '12:34:56.1234567890' raises INVALID_TIME_LITERAL_PRECISION. Is this intentional and consistent with rest of the codebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56622:
URL: https://github.com/apache/spark/pull/56622#discussion_r3447603488
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
Review Comment:
Good catch - done in a9f4569. `truncateTimeToPrecision` now indexes a precomputed power-of-ten table instead of calling `math.pow` per value (helps the row-converter and cast paths too), and the vectorized `TimeUpdater` hoists the factor into its constructor via a `long[]` lookup, so the hot decode loop is pure long arithmetic with no per-value `pow`/assert. The factor table's length encodes `NANOS_PRECISION`, so the Java side needs no cross-language constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57551][SQL] Extend the TIME data type precision to nanoseconds (up to 9) [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56622:
URL: https://github.com/apache/spark/pull/56622#issuecomment-4767902885
All CI passed successfully:
<img width="757" height="93" alt="Screenshot 2026-06-22 at 13 43 42" src="https://github.com/user-attachments/assets/cfed139c-3222-4f46-850b-c15ef857683b" />
Merging to master/4.x. Thank you, @HyukjinKwon @srielau @uros-b @davidm-db @cloud-fan @shrirangmhalgi for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org