Re: [PR] [core] Fix that ignore-delete option is not compatible with old delete records [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on code in PR #3139:
URL: https://github.com/apache/paimon/pull/3139#discussion_r1547351885


##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueFileReaderFactory.java:
##########
@@ -144,7 +144,12 @@ private RecordReader<KeyValue> createRecordReader(
                     new ApplyDeletionVectorReader<>(fileRecordReader, deletionVector.get());
         }
 
-        return new KeyValueDataFileRecordReader(fileRecordReader, keyType, valueType, level);
+        return new KeyValueDataFileRecordReader(
+                fileRecordReader,
+                keyType,
+                valueType,
+                level,
+                CoreOptions.fromMap(schema.options()).ignoreDelete());

Review Comment:
   Can be a class member.



##########
paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileRecordReader.java:
##########
@@ -50,11 +56,15 @@ public RecordIterator<KeyValue> readBatch() throws IOException {
             return null;
         }
 
-        return iterator.transform(
-                internalRow ->
-                        internalRow == null
-                                ? null
-                                : serializer.fromRow(internalRow).setLevel(level));
+        RecordIterator<KeyValue> transformed =
+                iterator.transform(
+                        internalRow ->
+                                internalRow == null
+                                        ? null
+                                        : serializer.fromRow(internalRow).setLevel(level));
+        // In older version, the delete records might be written into data file even when

Review Comment:
   Document clear: Version 0.7



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org