[jira] [Comment Edited] (FLINK-31593) Update reference data for Migration Tests

["Roman Khachatryan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17708612#comment-17708612 ] 

Roman Khachatryan edited comment on FLINK-31593 at 4/4/23 11:28 PM:
--------------------------------------------------------------------

I looked into StatefulJobWBroadcastStateMigrationITCase failiure which fails CI for your PRs.

It failed because TM was unable to find the referenced sst file:

 
{code:java}
java.io.FileNotFoundException: /tmp/junit6895540657846214402/checkpoints_e6e27005-a3fb-4c23-81e8-f6a2e827079b/4b3009354700b5a3a966f015556c8fb2/taskowned/1d9945c1-44b9-4d7c-9b20-d5ff90e68d7b (No   such file or directory) {code}
The file is indeed referenced in _metadata and is missing in resources (stateful-scala-with-broadcast-udf-migration-itcase-flink1.17-rocksdb-checkpoint/_metadata).

It was likely generated during changelog materialization (becuase it uses taskowned/ dir). Materialization is not deterministic, which might explain why it didn't show up earlier.

 

It should be [possible|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/#upgrading-existing-jobs] to restore from such a checkpoint (given all the files are present) even with changelog disabled.

Alternatively, we can just generate the checkpoints with changelog (or materialization) disabled. 

 

Am I missing something [~mapohl] , [~Yanfei Lei] ?

 

(if the above sounds reasonable, [~mapohl]  please let me know if I can help in checkpoints generation)

 


was (Author: roman_khachatryan):
[~mapohl] I looked into StatefulJobWBroadcastStateMigrationITCase failiure which fails CI for your PRs.

It failed because TM was unable to find the referenced sst file:

 
{code:java}
java.io.FileNotFoundException: /tmp/junit6895540657846214402/checkpoints_e6e27005-a3fb-4c23-81e8-f6a2e827079b/4b3009354700b5a3a966f015556c8fb2/taskowned/1d9945c1-44b9-4d7c-9b20-d5ff90e68d7b (No   such file or directory) {code}
The file is indeed referenced in _metadata and is missing in resources (stateful-scala-with-broadcast-udf-migration-itcase-flink1.17-rocksdb-checkpoint/_metadata).

It was likely generated during changelog materialization (becuase it uses taskowned/ dir). Materialization is not deterministic, which might explain why it didn't show up earlier.

 

It should be [possible|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/#upgrading-existing-jobs] to restore from such a checkpoint (given all the files are present) even with changelog disabled.

Alternatively, we can just generate the checkpoints with changelog (or materialization) disabled. 

 

Am I missing something?

 

(if the above sounds reasonable, [~mapohl]  please let me know if I can help in checkpoints generation)

 

> Update reference data for Migration Tests
> -----------------------------------------
>
>                 Key: FLINK-31593
>                 URL: https://issues.apache.org/jira/browse/FLINK-31593
>             Project: Flink
>          Issue Type: Sub-task
>            Reporter: Matthias Pohl
>            Assignee: Matthias Pohl
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: FLINK-31593.StatefulJobSavepointMigrationITCase.create_snapshot.log, FLINK-31593.StatefulJobSavepointMigrationITCase.verify_snapshot.log
>
>
> # Update {{CURRENT_VERSION in TypeSerializerUpgradeTestBase}}  with the new version. This will likely fail some tests because snapshots are missing for that version. Generate them, for example in {{TypeSerializerUpgradeTestBase.}} 
>  # (major/minor only) Update migration tests in master to cover migration from new version: (search for usages of FlinkV{{{}ersion{}}})
>  ** AbstractOperatorRestoreTestBase
>  ** CEPMigrationTest
>  ** BucketingSinkMigrationTest
>  ** FlinkKafkaConsumerBaseMigrationTest
>  ** ContinuousFileProcessingMigrationTest
>  ** WindowOperatorMigrationTest
>  ** StatefulJobSavepointMigrationITCase
>  ** StatefulJobWBroadcastStateMigrationITCase



--
This message was sent by Atlassian Jira
(v8.20.10#820010)