You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "q8webmaster (via GitHub)" <gi...@apache.org> on 2026/06/13 22:59:26 UTC
[PR] [parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility) [paimon]
q8webmaster opened a new pull request, #8230:
URL: https://github.com/apache/paimon/pull/8230
## What does this PR do?
Paimon currently emits a Parquet `TIMESTAMP(isAdjustedToUTC, unit=MILLIS)` annotation for `TIMESTAMP` and `TIMESTAMP_WITH_LOCAL_TIME_ZONE` columns with precision ≤ 3.
The [Iceberg v2 specification](https://iceberg.apache.org/spec/#parquet) only permits `INT64 MICROS` for the `timestamp` and `timestamptz` logical types. `MILLIS` is only valid under Iceberg v3.
This causes Iceberg-aware query engines to reject Paimon-written Parquet files for any such column with a schema compatibility error similar to:
```
Field ts's type INT64 in parquet file … is incompatible with type
timestamp(6) with time zone defined in table schema
```
## Root cause
`ParquetSchemaConverter.createTimestampWithLogicalType` selects `TimeUnit.MILLIS` for `precision <= 3` and `TimeUnit.MICROS` for `3 < precision <= 6`. The Iceberg v2 spec does not allow `MILLIS`; only `MICROS` and `NANOS` are valid for timestamp columns.
## Changes
| File | Change |
|---|---|
| `ParquetSchemaConverter.java` | Emit `MICROS` annotation for `precision <= 3` instead of `MILLIS` |
| `ParquetRowDataWriter.java` | `TimestampMillsWriter.writeTimestamp`: call `value.toMicros()` (= `millisecond × 1000`) so the stored `INT64` matches the `MICROS` annotation unit |
The **reader path** (`MILLIS → precision=3`, `MICROS → precision=6`) is intentionally left unchanged so that Parquet files written by older Paimon versions remain readable without error.
## Backward compatibility
- **Reading old files** (MILLIS annotation): unaffected — the reader maps `MILLIS → precision=3` as before.
- **Writing new files**: `TIMESTAMP(n≤3)` columns now produce `MICROS`-annotated `INT64` values (`millisecond × 1000`).
- **Mixed tables**: a table whose files span a Paimon upgrade will contain both MILLIS (old) and MICROS (new) files. Both remain readable, but Iceberg-aware engines that enforce strict annotation checking on **all** files may still reject the old files until the table is rebuilt. Affected tables should be rebuilt from scratch after upgrading.
- **Schema round-trip**: `TIMESTAMP(3)` written then read back by Paimon will return `TIMESTAMP(6)` (since the MICROS annotation maps to precision 6 in the reader). The `testPaimonParquetSchemaConvert` test is updated to reflect this.
## Tests
- `testLowPrecisionTimestampUseMicrosAnnotation` — verifies that `createTimestampWithLogicalType` produces a `MICROS` annotation for all precision values 0–3, for both `TIMESTAMP` and `TIMESTAMP_WITH_LOCAL_TIME_ZONE`.
- `testPaimonParquetSchemaConvert` — updated expected round-trip result to account for `TIMESTAMP(3)` reading back as `TIMESTAMP(6)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8230:
URL: https://github.com/apache/paimon/pull/8230#discussion_r3457495381
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java:
##########
@@ -223,8 +223,8 @@ private SimpleColStats toTimestampStats(Statistics<?> stats, int precision) {
if (precision <= 3) {
LongStatistics longStats = (LongStatistics) stats;
return new SimpleColStats(
- Timestamp.fromEpochMillis(longStats.getMin()),
- Timestamp.fromEpochMillis(longStats.getMax()),
+ Timestamp.fromMicros(longStats.getMin()),
Review Comment:
This makes footer stats decoding depend only on the Paimon field precision, but existing files written before this change still use TIMESTAMP_MILLIS and store min/max in epoch milliseconds. The PR description says those files remain readable; however any stats extraction path (for example migrate/clone or metadata regeneration) would now decode an old value like 1704067200123 as epoch micros and produce a 1970-era bound. Can we inspect the Parquet logical unit from stats.type().getLogicalTypeAnnotation() / column metadata, like the reader does, and keep MILLIS for legacy files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility) [paimon]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8230:
URL: https://github.com/apache/paimon/pull/8230#issuecomment-5769456704
This pull request has had no activity for 90 days and is being marked stale. If you'd like to keep it open, please leave a comment or push a new commit. Thank you for your contribution to Apache Paimon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8230:
URL: https://github.com/apache/paimon/pull/8230#discussion_r3419568562
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -299,11 +299,7 @@ private static Comparable<?> toParquetObject(
} else if (value instanceof Timestamp) {
Timestamp timestamp = (Timestamp) value;
int precision = getTimestampPrecision(type);
- if (precision <= 3) {
- // milliseconds
- return timestamp.getMillisecond();
- } else if (precision <= 6) {
- // microseconds
+ if (precision <= 6) {
return timestamp.toMicros();
Review Comment:
[P1] This makes Parquet predicate pushdown use epoch micros for every `TIMESTAMP(0..3)` column, but existing files written before this change still store the same logical columns as `TIMESTAMP_MILLIS` with epoch-millis footer values. When a query with a timestamp predicate reads those old files, the Parquet row-group filter compares the micros literal against millis min/max values and can drop the row group entirely, so Paimon returns missing rows even though the normal reader can still decode the file. We need to either build the pushed filter from the file annotation (MILLIS vs MICROS) or disable timestamp pushdown for these low-precision columns to preserve backward compatibility.
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java:
##########
@@ -223,8 +223,8 @@ private SimpleColStats toTimestampStats(Statistics<?> stats, int precision) {
if (precision <= 3) {
LongStatistics longStats = (LongStatistics) stats;
return new SimpleColStats(
- Timestamp.fromEpochMillis(longStats.getMin()),
- Timestamp.fromEpochMillis(longStats.getMax()),
+ Timestamp.fromMicros(longStats.getMin()),
Review Comment:
[P2] The stats extractor also needs to decode based on the Parquet file annotation, not only the Paimon field precision. Old Paimon files with `TIMESTAMP(0..3)` have `TIMESTAMP_MILLIS` footer stats, so reading them with `fromMicros` turns their min/max into values about 1000x too small. This affects any path that re-extracts stats from existing Parquet files, such as migration/metadata rebuild workflows. Since `stats.type()` carries the primitive logical annotation, can we branch on its `TimeUnit` here the same way the vector reader does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8230:
URL: https://github.com/apache/paimon/pull/8230#discussion_r3457497363
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -299,11 +299,7 @@ private static Comparable<?> toParquetObject(
} else if (value instanceof Timestamp) {
Timestamp timestamp = (Timestamp) value;
int precision = getTimestampPrecision(type);
- if (precision <= 3) {
- // milliseconds
- return timestamp.getMillisecond();
- } else if (precision <= 6) {
- // microseconds
+ if (precision <= 6) {
return timestamp.toMicros();
Review Comment:
This filter literal is now always epoch micros for TIMESTAMP(0..6), but old Paimon files with precision 0..3 still store epoch milliseconds. The filter is built in ParquetFileFormat from the table type before ParquetReaderFactory opens each file, so it cannot distinguish TIMESTAMP_MILLIS files from new TIMESTAMP_MICROS files. For an old file, ts = 2024-01-01 becomes 1704067200000000 while the row-group stats/data are around 1704067200000, so Parquet can incorrectly prune matching row groups. We need to build the timestamp predicate after seeing the file schema, or otherwise avoid this pushdown for legacy low-precision timestamp files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org