You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "eric666666 (via GitHub)" <gi...@apache.org> on 2024/05/17 09:03:20 UTC
[I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
eric666666 opened a new issue, #3345:
URL: https://github.com/apache/paimon/issues/3345
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
Paimon 0.9 snapshot
### Compute Engine
Flink
### Minimal reproduce step
If use flink bounded source which like jdbc or other, with checkpoint enabled,flink do not commit last snapshot commit.
I review the source code find org.apache.paimon.flink.sink.CommitterOperator caused it.
I do not know why their should skip commit snapshot if flink streaming mode enable checkpoint.
### What doesn't meet your expectations?
Missing last commit
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on issue #3345:
URL: https://github.com/apache/paimon/issues/3345#issuecomment-2146480990
As I know, flink 15+ version has support checkpointing with finished tasks (https://flink.apache.org/2022/07/11/flip-147-support-checkpoints-after-tasks-finished-part-one/#support-checkpointing-with-finished-tasks). so commit will be executed once in `notifyCheckpointComplete`. And this was true in my tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3345:
URL: https://github.com/apache/paimon/issues/3345#issuecomment-2120578069
Flink 1.14?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
Posted by "eric666666 (via GitHub)" <gi...@apache.org>.
eric666666 commented on issue #3345:
URL: https://github.com/apache/paimon/issues/3345#issuecomment-2117095166
> @tsreaper @JingsongLi Please have a look.
I remove the condition, to fix the problem. But I look at the code, the conditions here seem to be intended, so I don't know if I should do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
Posted by "eric666666 (via GitHub)" <gi...@apache.org>.
eric666666 commented on issue #3345:
URL: https://github.com/apache/paimon/issues/3345#issuecomment-2141212551
> Flink 1.14?
No, Flink 1.18.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Flink bounded source with checkpoint missing last snapshot commit [paimon]
Posted by "eric666666 (via GitHub)" <gi...@apache.org>.
eric666666 commented on issue #3345:
URL: https://github.com/apache/paimon/issues/3345#issuecomment-2117088730
@tsreaper @JingsongLi
Please having a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org