You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lkokhreidze (via GitHub)" <gi...@apache.org> on 2024/08/07 06:56:18 UTC
[I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
lkokhreidze opened a new issue, #10892:
URL: https://github.com/apache/iceberg/issues/10892
### Apache Iceberg version
1.5.2
### Query engine
Flink
### Please describe the bug 🐞
When Flink's state is restored, Iceberg File Committer gets the max committed checkpoint ID from the table's metadata as seen [here](https://github.com/apache/iceberg/blob/main/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java#L189). When recovering the job from the older savepoint, this behaviour might cause a silent data loss for Kafka Source (the one we have observed). Consider the following case:
We have Kafka topic which is materialised as Iceberg table with checkpoint interval of 30 seconds.
We create job savepoints twice per day, so we could rollback the job to older state if something bad happens. Snapshots for the Iceberg table is kept for the last 2 days.
If during the normal operation the job is at checkpoint 10000, if we would need to restore the job to the yesterdays version from the savepoint, then checkpoint would be rolled back to 9000 (as an example). Savepoint also contains offsets of the Kafka source which were committed when savepoint was triggered. Upon recovery from the savepoint, Kafka source seeks the topic-partition offset based on committed offset in the savepoint.
Iceberg File Committer though, upon recovery, would still get the max checkpoint ID as 10000 from table's and it would skip committing any files to iceberg table due to [this](https://github.com/apache/iceberg/blob/main/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java#L240) check. The problem is that there's no guarantee that Kafka source will consume exact same amount of data as before. So it might consume lag faster now due to increased parallelism, therefore in just 500 checkpoints it might consume the whole topic lag, while Iceberg File Committer would skip the committing anything new to the table since ongoing checkpoint ID will still be less then max checkpoint stored in the table metadata.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277123841
I would also share how I implemented the Flink Iceberg sink at my previous company (Netflix) before the OSS implementation here. We generate UUID for each checkpoint entry/commit which are added to snapshot metadata (similar to checkpointId). During state restore, committer use the UUIDs to check if the restored checkpoints/attempts have been committed or not. There won't be `maxCommittedCheckpointId` in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277051765
this is indeed a problem.
let's first clarify what should be the expected behavior. When users rewind the job from an older checkpoint/savepoint to reprocess older data, it indicates users are ok with duplicates.
> If this makes sense, I was wondering, if it would be possible to actually rollback the table snapshot to the version that we get from the Flink's checkpoint/savepoint upon recovery.
this is a logical thinking. but it may not be always safe and has some other implications to downstream.
* if there are multiple jobs write to the same table. Rewinding the Iceberg table state to an earlier snapshot just based on one job is incorrect.
* If downstream jobs (streaming or batch) does incremental processing on this Iceberg table, automatic rewinding table state in the upstream writer job is not safe.
I prefer this type of Iceberg table state manipulation as a manual admin action (instead of automatic from the Flink writer job).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "lkokhreidze (via GitHub)" <gi...@apache.org>.
lkokhreidze commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2602397744
I know I am late, but I am still planning to look into this issue, sorry for the delay :(
FYI - there was a bug in Flink's adaptive scheduler that was causing Flink to recover with empty checkpoint under certain conditions. Flink's bug also leads to the same data loss scenario as described in this issue.
https://issues.apache.org/jira/browse/FLINK-34518
We were able to mitigate the issue by implementing the custom Flink operator that upon recovery rollbacks the Iceberg table to the snapshot version that matches the recovered checkpoint ID. We only have one writer for the Iceberg tables so that worked fine for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2278393063
> I would say no. multiple jobs writing to the same table is a good example.
Theoretically it is possible to revert only the changes done by the given job (for append only tables). Still not sure we have to do so, but we could.
> BTW, this can be done by snapshots metadata table query with a filter on the summary column, right?
That's the easy part. Harder to find the correct checkpointId, operatorId, jobId - this needs state parsing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2302445320
> @pvary I'd be happy to help if I can be helpful. Can you maybe expand on the 1st point? How do you envision it being implemented? Is it just as simple as when recovery is detected, remove the max checkpoint ID check and let Iceberg committer commit the data?
We will still need to have `maxCommittedCheckpointId` in `initializeState` to prevent recommitting data to the Iceberg table on state restore, but in the `notifyCheckpointComplete` we need to have a different check. We need to correlate the current deployment and the last commit created by this deployment - we have to be aware of the lack of guarantees on the [CheckpointListener/notifyCheckpointComplete](https://nightlies.apache.org/flink/flink-docs-release-1.20/api/java//org/apache/flink/api/common/state/CheckpointListener.html#notifyCheckpointComplete-long-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277069747
Let's understand the motivation of `maxCommittedCheckpointId`, it was used to check if there are any unsuccessful commits from restored checkpoint state. But this problem shows a limitation of this approach.
let's also clarify what should be the expected behavior. When users rewind the job from an older checkpoint/savepoint to reprocess older data, it indicates users are accepting duplicates.
One potential fix is to reset the `maxCommittedCheckpointId` to the restored checkpointId after the check. Any uncommitted attempts (before or at the restored checkpointId) should be committed as usual. Any uncommitted attempts after the restored checkpointId should be discarded.
Let's look at this particular scenario. Let's the previous job completed checkpoint 10. But checkpoint 10 have commit failures (say network issue with the catalog service). So the c is 9. The job was rewinded to checkpoint 6.
after checking what are the committed checkpoints, committer should reset `maxCommittedCheckpointId` to the restored checkpointId `6`. Uncommitted checkpoints 10 from previous job should be discarded.
But it is also not an ideal situation if we are looking at the table snapshot history. we should say this order (from earliest to latest) regarding the committed Flink checkpoints.
```
5, 6, 7, 8, 9, (job rewinded), 7, 8, ...
```
It won't be a monotonically increased checkpointIds anymore, which would violate the assumption of `getMaxCommittedCheckpointId` method.
I see two potential options forward.
1. We can fix the `getMaxCommittedCheckpointId` so that it only looks back until the monotonically decreasing ends. e.g. when `getMaxCommittedCheckpointId` runs for the new job that restored from the 2nd checkpoint 8, it should return `8` as the backward searching should stop at the 2nd checkpointId `7`. the snapshot before it has an higher checkpointId of '9'. back tracing should stop in this case.
2. To me, this is also an indication of the limitation of using checkpointId to check successful/unsuccessful commits. When I implemented the Flink Iceberg sink at my previous company (Netflix) before the OSS implementation here, I generates an UUID for each checkpoint entry/commit. During state restore, committer use the UUIDs to check if the restored checkpoints/attempts have been committed or not. There we don't care about the monotonically increasing checkpoint Ids.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-3814630773
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "lkokhreidze (via GitHub)" <gi...@apache.org>.
lkokhreidze commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2275481345
Thanks for the reply @pvary ! Just for my understanding - is there technical reason why we can't call [ManageSnapshots#setCurrentSnapshot](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/ManageSnapshots.java#L53) API when recovery happens?
Flow could be something like this:
1. When job state is restored, get the snapshot ID from the Flink state.
2. If current snapshot id of the table and one restored from the flink state is the same, do nothing, all seems good.
3. If snapshot ID from the Flink's state is different, do the following:
- Get history of the snapshots and make sure that we still have that snapshot id in history, if not fail the recovery with explicit error msg.
- If snapshot ID is available in the history, use [ManageSnapshots#setCurrentSnapshot](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/ManageSnapshots.java#L53) API to set the current snapshot of the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "rodmeneses (via GitHub)" <gi...@apache.org>.
rodmeneses commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2302492563
> For the 3rd point:
>
> * Flink uses two phase commits. In the 1st phase the data is written to a temp manifest file, and the file path is stored into the state.
>
> So if there is a failure between the 2 commit phases, it could happen that the data is available in the temp manifest file, but not yet committed to the Iceberg table. This should be considered before throwing an error.
>
> But if the current snapshot of the Iceberg table is newer than the checkpoint we restore from, then it would be fine to revert to the given snapshot - we need to throw an exception if there was any concurrent writes to the table in the meantime (some other writers might written data independently).
>
> Also we should examine what we can do with the new IcebergSink. @rodmeneses could you please chime in?
Both the current FlinkSink and the incoming IcebergSink shares the same logic for `getMaxCommittedCheckpointId`. Both of them are also skipping checkpoints before this maxCheckpointId. In the new IcebergSink, we are also calling `signalAlreadyCommitted` for each checkpoint that is being skipped.
So I'd say that the same issue will also impact the new IcebergSink.
@lkokhreidze Thanks for reporting this and volunteering to work in a fix. I'd say the first thing would be to create unit test to reproduce this issue. Once you have them, kinda let me know and I can port them over to the new IcebergSink implementation.
cc: @pvary @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10892: Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss
URL: https://github.com/apache/iceberg/issues/10892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "jonhope3 (via GitHub)" <gi...@apache.org>.
jonhope3 commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-3090738950
Hi folks,
Checking on this as we're seeing this issue as well. Just a simple job that does Kafka -> Deserialize -> Write to Iceberg on Glue/S3.
We're on Flink 1.19 with Iceberg 1.9.1 runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277963876
> whether we think that the Iceberg table state is part of the Flink state, or not.
I would say no. multiple jobs writing to the same table is a good example.
> I have seen requests to help me rewind the state of the Iceberg table to the given savepoint, when rolling back was needed. We needed to read the state with custom java code to extract the checkpointId from the given state. This was hard for such a simple goal. Extracting an UUID would be even harder.
that is a good example where checkpointId is easier than UUID.
BTW, this can be done by `snapshots` metadata table query with a filter on the `summary` column, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2276410555
For the 3rd point:
- Flink uses two phase commits. In the 1st phase the data is written to a temp manifest file, and the file path is stored into the state.
So if there is a failure between the 2 commit phases, it could happen that the data is available in the temp manifest file, but not yet committed to the Iceberg table. This should be considered before throwing an error.
But if the current snapshot of the Iceberg table is newer than the checkpoint we restore from, then it would be fine to revert to the given snapshot - we need to throw an exception if there was any concurrent writes to the table in the meantime (some other writers might written data independently).
Also we should examine what we can do with the new IcebergSink. @rodmeneses could you please chime in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "lkokhreidze (via GitHub)" <gi...@apache.org>.
lkokhreidze commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2272758965
I hope the explanation makes sense and I am not thinking about it in a completely wrong way.
If this makes sense, I was wondering, if it would be possible to actually rollback the table snapshot to the version that we get from the Flink's checkpoint upon recovery... I realize there might be some reason why it's not done like that, but for me it seems logical thing to do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "lkokhreidze (via GitHub)" <gi...@apache.org>.
lkokhreidze commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2272765144
This behaviour might also explain the issue reported [here](https://github.com/apache/iceberg/issues/9089).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from the savepoint might trigger a silent data loss when Kafka source is used [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2275318331
Maybe, when restoring state, we could collect commits created by this job/operator, but with newer checkpointId, and we could create a "revert" commit for them...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2277177813
The main question is whether we think that the Iceberg table state is part of the Flink state, or not.
How do other sinks behave with this regard?
I have seen requests to help me rewind the state of the Iceberg table to the given savepoint, when rolling back was needed. a
We needed to read the state with custom java code to extract the checkpointId from the given state. This was hard for such a simple goal. Extracting an UUID would be even harder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2290980634
Discussed the issue with @stevenzwu yesterday in detail.
We agreed, that we should separate out the problems above.
1. We need to prevent the sink to silently not commit data when there is already one committed to the table with the same checkpointId - this is only desired when restoring the state (as we can't be sure where the previous deployment failed). In every other case, when there are new incoming records, they need to be committed to the Iceberg table.
2. We need to find a way for the users to easily (manually or automatically) revert back changes done by the given job to the state what was created by the given checkpoint/savepoint.
While in this specific case 1. and 2. was correlated, there might be cases when the desired behaviour is different, so we should not tie the 2 problems together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "lkokhreidze (via GitHub)" <gi...@apache.org>.
lkokhreidze commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2301542040
@pvary I'd be happy to help if I can be helpful. Can you maybe expand on the 1st point? How do you envision it being implemented? Is it just as simple as when recovery is detected, remove the max checkpoint ID check and let Iceberg committer commit the data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Restoring the Flink streaming job from and older checkpoint/savepoint might trigger a silent data loss [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10892:
URL: https://github.com/apache/iceberg/issues/10892#issuecomment-3752291802
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org