You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/11 10:35:29 UTC
[GitHub] [hudi] zhangyue19921010 opened a new issue #3973: [SUPPORT]Hudi Multi-Writer with archive
zhangyue19921010 opened a new issue #3973:
URL: https://github.com/apache/hudi/issues/3973
Just ask a question about Hudi multi-write:
For a Hudi table named table1, and Spark job 1 and Spark job 2 are writing table1 at the same time with multi-write related configs enabled.
--> Table1 has 10 commit files under .hoodie dictionary.
--> Job1 is writing data with 10 commits view's active timeline
--> Job2 is finished writing and do archive works which will archive commits from 10 to 1(Exaggerated expression)
The same circumstance of archive async some how.
Under this circumstance, could something wrong with writing job1 ?
For example, update, compaction or something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3973: [SUPPORT]Hudi Multi-Writer with archive
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3973:
URL: https://github.com/apache/hudi/issues/3973#issuecomment-997118683
@zhangyue19921010 : sorry about the late turn around.
Archive will not step into commits which are part of compaction plan. So, multiple writers should not cause any issues.
Also, in example above, job1 will deal with only latest file slice. so archiving by job2 should not have any impact.
Let me know if you have any more clarifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 closed issue #3973: [SUPPORT]Hudi Multi-Writer with archive
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 closed issue #3973:
URL: https://github.com/apache/hudi/issues/3973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #3973: [SUPPORT]Hudi Multi-Writer with archive
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on issue #3973:
URL: https://github.com/apache/hudi/issues/3973#issuecomment-1000142445
Hi @nsivabalan Thanks a lot for your response :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3973: [SUPPORT]Hudi Multi-Writer with archive
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3973:
URL: https://github.com/apache/hudi/issues/3973#issuecomment-966780188
@nsivabalan could you take this? recall you're more familiar with multi-writer part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org