You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "FangYongs (via GitHub)" <gi...@apache.org> on 2023/07/24 07:51:34 UTC
[GitHub] [incubator-paimon] FangYongs opened a new issue, #1635: [Feature] Create new tag for flink savepoint
FangYongs opened a new issue, #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Currently Paimon supports tag to store data which can be associated with flink savepoint. When Flink perform a savepoint on sink operator, it can create a tag automatically after create a snapshot.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] FangYongs commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1672737847
@liming30 I reviewed this PR overall, there are two issues and you can consider whether it need to create some new issues for them:
1. How to get the specified tag name or id for given savepoint path, to that users can rollback the table to that tag
2. We need to validate the tag in the table when job is resumed from given savepoint
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi closed issue #1635: [Feature] Create new tag for flink savepoint
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #1635: [Feature] Create new tag for flink savepoint
URL: https://github.com/apache/incubator-paimon/issues/1635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1667323450
Hi @FangYongs , can you explain what scenario requires this ability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1677343104
Thanks @FangYongs , I see. We can document this in the PR~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] FangYongs commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1668970661
@JingsongLi This is used for data incremental correction for streaming processing. For example, there's a Flink streaming etl job reads data from kafka and writes results to Paimon and we will regularly trigger savepoints in Flink to ensure job recovery in case of disaster.
We sometimes need to redo data for kafka from specified timestamp, and resume Flink streaming etl job from the specified offset. At present, we cannot incrementally recover and can only discard the results of previous computation.
But with Flink savepoint and tag in Paimon, we can rollback data in Paimon to given tag and resume Flink job from given savepoint to incremental streaming process without lose any data.
The detail of incremental recover is described in `b) Increment Recover` of [PIP-5](https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] liming30 commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "liming30 (via GitHub)" <gi...@apache.org>.
liming30 commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1652866289
At present, the type of checkpoint cannot be distinguished in the operator, so I created an issue([FLINK-32698](https://issues.apache.org/jira/browse/FLINK-32698)) in the flink community. Maybe we can wait for this issue to be discussed before proceeding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] FangYongs commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1652878121
@liming30 I think we can change the implementation of `PrepareCommitOperator` in paimon not `extends AbstractStreamOperator`, then you can implement method `OperatorSnapshotFutures snapshotState(long checkpointId, long timestamp, CheckpointOptions checkpointOptions, CheckpointStreamFactory storageLocation) throws Exception` in it which has `CheckpointOptions` as parameter and you can get `SnapshotType`.
At the same time, you can push [FLINK-32698](https://issues.apache.org/jira/browse/FLINK-32698), I think it will be a long time after it is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] liming30 commented on issue #1635: [Feature] Create new tag for flink savepoint
Posted by "liming30 (via GitHub)" <gi...@apache.org>.
liming30 commented on issue #1635:
URL: https://github.com/apache/incubator-paimon/issues/1635#issuecomment-1649197828
@FangYongs I also reproduce this problem when testing locally, can you assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org