You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "AbdelrahmanElhawary (via GitHub)" <gi...@apache.org> on 2026/07/07 20:47:27 UTC

[I] Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader [iceberg]

AbdelrahmanElhawary opened a new issue, #17135:
URL: https://github.com/apache/iceberg/issues/17135

   ### Problem Context
   Apache Iceberg supports reading Parquet files with `TIMESTAMP_MILLIS` annotations by converting the millisecond values to Iceberg's internal microsecond representation ($\times 1000$). While this scaling logic works correctly for `PLAIN` encoded pages (via `TimestampMillisReader`), it is completely bypassed when a column is entirely dictionary-encoded (e.g., columns with duplicate or low-cardinality values, like a batch extraction timestamp).
   When this optimization occurs, the values are displayed as incorrect dates in the year 1970 because raw millisecond values are treated as microseconds.
   
   ### Root Cause Analysis
   
   In `VectorizedArrowReader#read`, the engine checks whether a column segment produces a dictionary-encoded vector:
   
   ```
   boolean dictEncoded = vectorizedColumnIterator.producesDictionaryEncodedVector();
   if (vectorizedColumnIterator.hasNext()) {
     if (dictEncoded) {
       vectorizedColumnIterator.dictionaryBatchReader().nextBatch(vec, -1, nullabilityHolder);
     } else {
       switch (readType) { ... }
     }
   }
   ```
   
   If `dictEncoded` is true, the reader completely bypasses the type-specific switch statement—which normally maps to `ReadType.TIMESTAMP_MILLIS` and uses the correct `TimestampMillisReader`. Instead, it shortcuts by populating a generic `IntVector` with raw dictionary IDs and attaches the raw Parquet `Dictionary` object straight to the `VectorHolder` returned to Spark.
   
   When Spark eventually decodes these IDs via` Dictionary#decodeToLong(id)`, it receives unscaled milliseconds from the raw Parquet metadata, resulting in corrupted timestamps.
   
   ### Solution
   The fix intercepts the raw Parquet `Dictionary` inside `VectorizedArrowReader#setRowGroupInfo` right after initialization.
   
   If the column's modern `LogicalTypeAnnotation` indicates it is a `TIMESTAMP` with `TimeUnit.MILLIS` precision, the dictionary is wrapped in a proxy wrapper. This proxy intercepts calls to` decodeToLong(int id)` and scales the returned values to microseconds.
   
   This approach resolves the bug gracefully:
   
   It fixes the issue on the optimized dictionary-passthrough path.
   
   ### Changes
   `VectorizedArrowReader.java`: Added a check using ### LogicalTypeAnnotation to detect `TimeUnit.MILLIS` timestamps inside `setRowGroupInfo`.
   
   Wrapped this.dictionary in an anonymous proxy class that applies the `* 1000L bit-shift` multiplier inside `decodeToLong`.
   
   ### How to Test
   Write an Iceberg table where a timestamp column contains identical values (forcing Parquet's writer optimization to select `PLAIN_DICTIONARY` encoding instead of `PLAIN`).
   
   Read the table using Spark with vectorization enabled (`spark.sql.iceberg.vectorized_read.enabled=true`).
   
   Before Fix: Values display as 1970-01-21...
   
   After Fix: Values accurately decode to their current, modern calendar dates.
   
   ### Exact code change
   From : 
   ```
     @Override
     public void setRowGroupInfo(PageReadStore source, Map<ColumnPath, ColumnChunkMetaData> metadata) {
       ColumnChunkMetaData chunkMetaData = metadata.get(ColumnPath.get(columnDescriptor.getPath()));
       this.dictionary =
           vectorizedColumnIterator.setRowGroupInfo(
               source.getPageReader(columnDescriptor),
               !ParquetUtil.hasNonDictionaryPages(chunkMetaData));
     }
   ```
   To : 
   ```
     @Override
     public void setRowGroupInfo(PageReadStore source, Map<ColumnPath, ColumnChunkMetaData> metadata) {
       ColumnChunkMetaData chunkMetaData = metadata.get(ColumnPath.get(columnDescriptor.getPath()));
       this.dictionary =
           vectorizedColumnIterator.setRowGroupInfo(
               source.getPageReader(columnDescriptor),
               !ParquetUtil.hasNonDictionaryPages(chunkMetaData));
   
       // Modern, non-deprecated check using LogicalTypeAnnotation
       boolean isTimestampMillis = false;
       if (columnDescriptor != null && columnDescriptor.getPrimitiveType() != null) {
         org.apache.parquet.schema.LogicalTypeAnnotation annotation = 
             columnDescriptor.getPrimitiveType().getLogicalTypeAnnotation();
         
         if (annotation instanceof org.apache.parquet.schema.LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
           org.apache.parquet.schema.LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampAnnotation = 
               (org.apache.parquet.schema.LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) annotation;
           
           isTimestampMillis = timestampAnnotation.getUnit() == org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit.MILLIS;
         }
       }
   
       if (this.dictionary != null && isTimestampMillis) {
           final Dictionary backingDictionary = this.dictionary;
           
           this.dictionary = new Dictionary(backingDictionary.getEncoding()) {
               @Override
               public long decodeToLong(int id) {
                   return backingDictionary.decodeToLong(id) * 1000L;
               }
   
               @Override
               public int decodeToInt(int id) { return backingDictionary.decodeToInt(id); }
   
               @Override
               public float decodeToFloat(int id) { return backingDictionary.decodeToFloat(id); }
   
               @Override
               public double decodeToDouble(int id) { return backingDictionary.decodeToDouble(id); }
   
               @Override
               public org.apache.parquet.io.api.Binary decodeToBinary(int id) { return backingDictionary.decodeToBinary(id); }
   
               @Override
               public int getMaxId() { return backingDictionary.getMaxId(); }
           };
       }
     }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader [iceberg]

Posted by "cokuwuragema (via GitHub)" <gi...@apache.org>.
cokuwuragema commented on issue #17135:
URL: https://github.com/apache/iceberg/issues/17135#issuecomment-4908698154

   [iceberg_dict_fix.zip](https://github.com/user-attachments/files/29766473/iceberg_dict_fix.zip)
   Man, I ran into that same 1970 date glitch with my Spark jobs. Turns out the dictionary path was totally skipping the microsecond scaling logic. I put together a small patch that wraps the dictionary in a proxy to handle the millis conversion properly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary closed issue #17135: Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader
URL: https://github.com/apache/iceberg/issues/17135


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader [iceberg]

Posted by "AbdelrahmanElhawary (via GitHub)" <gi...@apache.org>.
AbdelrahmanElhawary commented on issue #17135:
URL: https://github.com/apache/iceberg/issues/17135#issuecomment-6335911769

   thanks for continuing the fix @binarybrain-009 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Fix TIMESTAMP_MILLIS scaling for dictionary-encoded columns in Vectorized Spark Reader [iceberg]

Posted by "binarybrain-009 (via GitHub)" <gi...@apache.org>.
binarybrain-009 commented on issue #17135:
URL: https://github.com/apache/iceberg/issues/17135#issuecomment-6284676487

   @AbdelrahmanElhawary #17143 was closed by the stale bot before anyone reviewed it. If you aren't planning to reopen it, I'd like to pick this up and keep you as co-author. I'd fix it in `GenericArrowVectorAccessorFactory`, which is where every reader of a dictionary-encoded vector decodes its values: `TIMESTAMP_MILLIS` currently shares `DictionaryLongAccessor` with plain longs, so it would get its own accessor that converts to microseconds, the same way INT96 timestamps have `DictionaryTimestampInt96Accessor`. The test would write a fully dictionary-encoded millis column and check that the vectorized reader returns microseconds, with and without UTC adjustment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org