You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/02 16:25:25 UTC

[GitHub] [hudi] melin opened a new issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

melin opened a new issue #4938:
URL: https://github.com/apache/hudi/issues/4938


   Example：The user listens for events that trigger incremental calculations of jobs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin commented on issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

Posted by GitBox <gi...@apache.org>.
melin commented on issue #4938:
URL: https://github.com/apache/hudi/issues/4938#issuecomment-1059982991


   > Hudi already has commit callbacks. wouldn't that suffice? or are you looking for more fine grained events ? https://hudi.apache.org/docs/configurations/#Write-commit-callback-configs
   
   It's what I need, Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4938:
URL: https://github.com/apache/hudi/issues/4938#issuecomment-1057466436


   sorry, I am not very clear on the requirement. 
   We already have commit callbacks. Would that suffice or are you looking for something else. 
   can you throw some light please. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin closed issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

Posted by GitBox <gi...@apache.org>.
melin closed issue #4938:
URL: https://github.com/apache/hudi/issues/4938


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4938:
URL: https://github.com/apache/hudi/issues/4938#issuecomment-1059243349


   Hudi already has commit callbacks. wouldn't that suffice? or are you looking for more fine grained events ? 
   https://hudi.apache.org/docs/configurations/#Write-commit-callback-configs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin commented on issue #4938: [SUPPORT]Add snapshot,  send event notification, user can customize listener

Posted by GitBox <gi...@apache.org>.
melin commented on issue #4938:
URL: https://github.com/apache/hudi/issues/4938#issuecomment-1057570270


   > sorry, I am not very clear on the requirement. We already have commit callbacks. Would that suffice or are you looking for something else. can you throw some light please.
   
   There is a need. The upstream system periodically writes Hudi data to the object storage. The downstream system needs to know the changes of Hudi data. Currently, the upstream system can only periodically query Hudi snapshot information to determine whether Hudi table data has changed. Then the incremental calculation of the job is triggered. If there is a snapshot change notification mechanism (including partition change information). Users can customize the Listener to write the change message to Kafka, and the downstream system only needs to fetch the message from Kafka to know the data change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org