You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/01 08:49:14 UTC

[GitHub] [hudi] aditiwari01 opened a new issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

aditiwari01 opened a new issue #2756:
URL: https://github.com/apache/hudi/issues/2756


   While creating HoodieRecordPayloads from log files in case of MOR tables, the payloads are created without any orderingVal (even if specified while writing data). Due to this the precombine function could result in any payload irrespective of its orderingVal.
   
   Attaching a sample script to reproduce the issue.
   
   In this example, for key "key1", 1st insert is with ts=1000. Then we update with ts=2000. Thenn we updated with ts=500. Ideally after last update if we snnapshot query the table, we must get key1 with ts=2000 (since our ordering field is ts). However it shows entry of ts=1000 because from logs it ignores ts=2000 and only picks up ts=500.
   
   Also AFAIU, the same flow will be used while compaction and then we might lose data forever.
   
   [Hudi_sample_commands.txt](https://github.com/apache/hudi/files/6242206/Hudi_sample_commands.txt)
   
   A workaround is to implement a HoodiePayloadClass and hardcode ordering field in the costructor only. But we can not keep it parameterised since properties are not available in the constructor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] aditiwari01 commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-813447613


   The same behaviour is seen for both snapshot query over spark and hive query over _rt table, since both of them uses same read path.
   
   To be more specific, the issue seems to be in class `AbstractHoodieLogRecordScanner`, method `createHoodieRecord`.
   
   Here we are creating HoodieRecord from IndexRecord without ordering val, and hence setting orderingVal=0 for all Records, irrespective of PayloadClass used.
   
   I'm thinking about getting precombine field from hoodie meta client and use that as orderingVal while creating the Hoodie Record. Does it sound fine?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-812703790


   CC @n3nash 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] aditiwari01 edited a comment on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
aditiwari01 edited a comment on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-812516535


   I think I couldn't explain myself. I am using DefaultHoodieRecordPayload only. I have attached sample command regardinng same.
   
   The issue is not with "combineAndGetUpdateValue", rather with "preCombine".
    As per my uderstanding, combineAndGetUpdateValue is used to merge record from parquet and in memory record, whereas preCombine is used to dedupe multiple records in memory with same key. The preCombine function uses orderingVal field to sort and while creating record from log file we do not set this ordering field. And hence the issue. 
   
   The constructors are as foolows:
   
   1. DefaultHoodieRecordPayload(Option<GenericRecord> record) {this(recordl, 0);}
   2. DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {super(record, orderingVal)}
   
   In the read path we only call the 1st constructor and hence lose the ordering value.
   
   Also, if we compact after each commit we dont see this issue since "combineAndGetUpdateValue" works absolutely fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] aditiwari01 commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-813006720


   Can we change the tag for the issue? Or is there anythinng required from my side?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan closed issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #2756:
URL: https://github.com/apache/hudi/issues/2756


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-813522558


   yes, you are right. we do have a bug here w/ DefaultHoodieRecordPayload. With OverwriteWithLatestAvroPayload, we always choose the latest and so we didn't have this issue. DefaultHoodieRecordPayload class is something which was added in last few months and hence we didn't catch this. thanks for flagging this. 
   btw, DefaultHoodieRecordPayload uses a diff config named "hoodie.payload.ordering.field" to resolve combineAndGetUpdate. 
   
   Your approach seems fine to me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-812496813


   yes, this is expected. if you are using OverwriteWithLatestAvroPayload as your payload class, combineAndGetUpdateValue does not honor ordering value. And so we added another payload for this purpose. DefaultHoodieRecordPayload. While using this, ensure you set "hoodie.payload.ordering.field" accordingly. Let me know how it goes. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] aditiwari01 commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-812516535


   I think I couldn't explain myself. I am using DefaultHoodieRecordPayload only. I am attached sample command regardinng same.
   
   The issue is not with "combineAndGetUpdateValue", rather with "preCombine".
    As per my uderstanding, combineAndGetUpdateValue is used to merge record from parquet and in memory record, whereas preCombine is used to dedupe multiple records in memory with same key. The preCombine function uses orderingVal field to sort and while creating record from log file we do not set this ordering field. And hence the issue. 
   
   The constructors are as foolows:
   
   1. DefaultHoodieRecordPayload(Option<GenericRecord> record) {this(recordl, 0);}
   2. DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {super(record, orderingVal)}
   
   In the read path we only call the 1st constructor and hence lose the ordering value.
   
   Also, if we compact after each commit we dont see this issue since "combineAndGetUpdateValue" works absolutely fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #2756: OrderingVal not being honoured for payloads in log files (for MOR table)

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2756:
URL: https://github.com/apache/hudi/issues/2756#issuecomment-813524391


   Have filed a tracking [jira](https://issues.apache.org/jira/browse/HUDI-1763). Feel free to raise a PR :) once again, thanks for raising it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org