You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/04/19 08:15:14 UTC
[GitHub] [hudi] stream2000 opened a new issue, #8498: [DISCUSS] [Flink] Should we support start a new instant even there is no data in the last batch to support multi writer?
stream2000 opened a new issue, #8498:
URL: https://github.com/apache/hudi/issues/8498
**_Tips before filing an issue_**
In flink multi-writer scenario, if one of the writers has no data, it will keep its inflight instant in the timeline. But incremental clean and archive will be blocked by the oldest inflight commit in the timeline.
So to make clean and archive can move forward should we support starting a new instant even there is no data in the last batch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8498: [DISCUSS] [Flink] Should we support start a new instant when there is no data in the last batch to support multi writer?
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8498:
URL: https://github.com/apache/hudi/issues/8498#issuecomment-1515632473
Agree, this can avoid some corner cases for table services, the trade off is there maybe some useless commits on the timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on issue #8498: [DISCUSS] [Flink] Should we support start a new instant even there is no data in the last batch to support multi writer?
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on issue #8498:
URL: https://github.com/apache/hudi/issues/8498#issuecomment-1514327671
@danny0405 Hey danny could you have look at this? We have a simple multi writer implementation for append mode and have faced the problem that some the table service can't move forward due to a no-data multi writer retaining its inflight instant. And we're trying to fix it by allow flink to commit on an empty batch.
Maybe I can submit a demo pr the show how we're trying to fix it and you can review that pr directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 closed issue #8498: [DISCUSS] [Flink] Should we support start a new instant when there is no data in the last batch to support multi writer?
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 closed issue #8498: [DISCUSS] [Flink] Should we support start a new instant when there is no data in the last batch to support multi writer?
URL: https://github.com/apache/hudi/issues/8498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org