You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "AbdelrahmanElhawary (via GitHub)" <gi...@apache.org> on 2026/07/09 11:16:15 UTC
[PR] Arrow: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns [iceberg]
AbdelrahmanElhawary opened a new pull request, #17143:
URL: https://github.com/apache/iceberg/pull/17143
### Problem Context
Apache Iceberg supports reading Parquet files with `TIMESTAMP_MILLIS` annotations by converting the millisecond values to Iceberg's internal microsecond representation ($\times 1000$). While this scaling logic works correctly for `PLAIN` encoded pages (via `TimestampMillisReader`), it is completely bypassed when a column is entirely dictionary-encoded (e.g., columns with duplicate or low-cardinality values, like a batch extraction timestamp).
When this optimization occurs, the values are displayed as incorrect dates in the year 1970 because raw millisecond values are treated as microseconds.
Closes #17135
### Root Cause Analysis
In `VectorizedArrowReader#read`, the engine checks whether a column segment produces a dictionary-encoded vector:
```
boolean dictEncoded = vectorizedColumnIterator.producesDictionaryEncodedVector();
if (vectorizedColumnIterator.hasNext()) {
if (dictEncoded) {
vectorizedColumnIterator.dictionaryBatchReader().nextBatch(vec, -1, nullabilityHolder);
} else {
switch (readType) { ... }
}
}
```
If `dictEncoded` is true, the reader completely bypasses the type-specific switch statement—which normally maps to `ReadType.TIMESTAMP_MILLIS` and uses the correct `TimestampMillisReader`. Instead, it shortcuts by populating a generic `IntVector` with raw dictionary IDs and attaches the raw Parquet `Dictionary` object straight to the `VectorHolder` returned to Spark.
When Spark eventually decodes these IDs via` Dictionary#decodeToLong(id)`, it receives unscaled milliseconds from the raw Parquet metadata, resulting in corrupted timestamps.
### Solution
The fix intercepts the raw Parquet `Dictionary` inside `VectorizedArrowReader#setRowGroupInfo` right after initialization.
If the column's modern `LogicalTypeAnnotation` indicates it is a `TIMESTAMP` with `TimeUnit.MILLIS` precision, the dictionary is wrapped in a proxy wrapper. This proxy intercepts calls to` decodeToLong(int id)` and scales the returned values to microseconds.
This approach resolves the bug gracefully:
It fixes the issue on the optimized dictionary-passthrough path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Arrow: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17143:
URL: https://github.com/apache/iceberg/pull/17143#issuecomment-5310410647
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Arrow: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17143:
URL: https://github.com/apache/iceberg/pull/17143#issuecomment-5228942043
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Arrow: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17143: Arrow: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns
URL: https://github.com/apache/iceberg/pull/17143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org