You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/28 07:21:12 UTC

[GitHub] [hudi] abhibhat98 opened a new issue #1676: [SUPPORT] Updating all the records per key rather than the one with higher timestamp

abhibhat98 opened a new issue #1676:
URL: https://github.com/apache/hudi/issues/1676


   For multiple keys in the same update, only the one with higher PRECOMBINE_FIELD_OPT_KEY is considered. 
   
   The default payload is OverwriteWithLatestAvroPayload which, in preCombine, picks the payload with greatest ordering value. I can change this logic in the custom payload to pick any other value, however is it possible to update a values for same keys, rather than choosing 1?
   
   eg. I have this current snapshot of table
   K1 , V1
   
   If a batch comes with the following values,
   K1,V2
   K1,V3
   K1,V4
   
   Assuming V4 > V3,V2, can I still update all of them one by one so that I have the history? For now, only V4 is updated and in case of CDC / events, I lose the history for V2/V3.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar edited a comment on issue #1676: [SUPPORT] Updating all the records per key rather than the one with higher timestamp

Posted by GitBox <gi...@apache.org>.
vinothchandar edited a comment on issue #1676:
URL: https://github.com/apache/hudi/issues/1676#issuecomment-636343512


   @abhibhat98 trying to understand your use case bit more.. do you want to retain all the versions forever? If so probably better to have a separate table that has an append  model using insert() operation? 
   
   If you use upsert then you have to have infinite data retention to get all the versions (even if we fix precombing to log all versions). 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on issue #1676: [SUPPORT] Updating all the records per key rather than the one with higher timestamp

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1676:
URL: https://github.com/apache/hudi/issues/1676#issuecomment-636343512


   @abhibhat98 trying to understand your user case bit more.. do you want to retain all the versions forever? If so probably better to have a separate table that has an append  model using insert() operation? 
   
   If you use upsert then you have to have infinite data retention to get all the versions (even if we fix precombing to log all versions). 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on issue #1676: [SUPPORT] Updating all the records per key rather than the one with higher timestamp

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1676:
URL: https://github.com/apache/hudi/issues/1676#issuecomment-640377675


   @abhibhat98 : Pinging to see if you can answer the above question ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar closed issue #1676: [SUPPORT] Updating all the records per key rather than the one with higher timestamp

Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #1676:
URL: https://github.com/apache/hudi/issues/1676


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org