You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2023/05/16 07:08:36 UTC

[GitHub] [hudi] danny0405 commented on a diff in pull request #8721: [HUDI-6217]Fix Spark reads the deleted data whose _hoodie_operation is D

danny0405 commented on code in PR #8721:
URL: https://github.com/apache/hudi/pull/8721#discussion_r1194710953


##########
hudi-common/src/main/java/org/apache/hudi/common/model/BaseAvroPayload.java:
##########
@@ -85,13 +85,20 @@ public boolean canProduceSentinel() {
    */
   protected static boolean isDeleteRecord(GenericRecord genericRecord) {
     final String isDeleteKey = HoodieRecord.HOODIE_IS_DELETED_FIELD;
+    final String isDeleteKeyMeta = HoodieRecord.OPERATION_METADATA_FIELD;
     // Modify to be compatible with new version Avro.
     // The new version Avro throws for GenericRecord.get if the field name
     // does not exist in the schema.
-    if (genericRecord.getSchema().getField(isDeleteKey) == null) {
-      return false;
+    if (genericRecord.getSchema().getField(isDeleteKey) != null) {
+      Object deleteMarker = genericRecord.get(isDeleteKey);
+      return (deleteMarker instanceof Boolean && (boolean) deleteMarker);
     }
-    Object deleteMarker = genericRecord.get(isDeleteKey);
-    return (deleteMarker instanceof Boolean && (boolean) deleteMarker);
+
+    if (genericRecord.getSchema().getField(isDeleteKeyMeta) != null) {
+      Object deleteMarker = genericRecord.get(isDeleteKeyMeta);
+      return (deleteMarker != null && "D".equals(deleteMarker.toString()));

Review Comment:
   Nice catch, can you write a test case for Spark then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org