[GitHub] [iceberg] stevenzwu commented on a change in pull request #3015: Add data_file.spec_id to metadata tables

[GitBox <gi...@apache.org>]

stevenzwu commented on a change in pull request #3015:
URL: https://github.com/apache/iceberg/pull/3015#discussion_r753631713



##########
File path: core/src/main/java/org/apache/iceberg/BaseFile.java
##########
@@ -235,45 +235,48 @@ public void put(int i, Object value) {
         this.format = FileFormat.valueOf(value.toString());
         return;
       case 3:
-        this.partitionData = (PartitionData) value;
+        this.partitionSpecId = (value != null) ? (Integer) value : -1;

Review comment:
       @szehon-ho @RussellSpitzer @rdblue @openinx  FYI, shifting order is a breaking change that caused Flink failing to restore from checkpoint. It is a not big deal for us this time as we are still in testing phase. I just like to call out that we need to be more careful in the future.
   
   ```
   java.lang.ClassCastException: class org.apache.iceberg.PartitionData cannot be cast to class java.lang.Integer (org.apache.iceberg.PartitionData is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @3e063fd4; java.lang.Integer is in module java.base of loader 'bootstrap')
   	at org.apache.iceberg.BaseFile.put(BaseFile.java:238)
   	at org.apache.iceberg.avro.ValueReaders$IndexedRecordReader.set(ValueReaders.java:746)
   	at org.apache.iceberg.avro.ValueReaders$IndexedRecordReader.set(ValueReaders.java:715)
   	at org.apache.iceberg.avro.ValueReaders$StructReader.read(ValueReaders.java:669)
   	at org.apache.iceberg.avro.ValueReaders$StructReader.read(ValueReaders.java:669)
   	at org.apache.iceberg.data.avro.DecoderResolver.resolveAndRead(DecoderResolver.java:48)
   	at org.apache.iceberg.avro.GenericAvroReader.read(GenericAvroReader.java:69)
   	at org.apache.iceberg.avro.ProjectionDatumReader.read(ProjectionDatumReader.java:74)
   	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:250)
   	at org.apache.iceberg.avro.AvroIterable$AvroReuseIterator.next(AvroIterable.java:202)
   	at org.apache.iceberg.io.CloseableIterable$4$1.next(CloseableIterable.java:113)
   	at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:66)
   	at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
   	at org.apache.iceberg.io.CloseableIterable$4$1.hasNext(CloseableIterable.java:108)
   	at org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:355)
   	at org.apache.iceberg.relocated.com.google.common.collect.Lists.newArrayList(Lists.java:143)
   	at org.apache.iceberg.relocated.com.google.common.collect.Lists.newArrayList(Lists.java:130)
   	at org.apache.iceberg.flink.sink.FlinkManifestUtil.readDataFiles(FlinkManifestUtil.java:60)
   	at org.apache.iceberg.flink.sink.FlinkManifestUtil.readCompletedFiles(FlinkManifestUtil.java:105)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:212)
   	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.initializeState(IcebergFilesCommitter.java:156)
   	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:118)
   	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:290)
   	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:441)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585)
   	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540)
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org