You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "lincoln-lil (via GitHub)" <gi...@apache.org> on 2024/03/18 10:05:20 UTC
[PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
lincoln-lil opened a new pull request, #24517:
URL: https://github.com/apache/flink/pull/24517
The data was generated based on release-1.19.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2016860210
@snuyanzin sorry to ping you here, do you still remember this data generation during 1.18 release ( https://github.com/apache/flink/pull/23710)?
Is there any exception when do `generate-migration-test-data` via maven?
I debugged generating `org.apache.flink.test.checkpointing.StatefulJobSnapshotMigrationITCase` locally via `MigrationTestsSnapshotGenerator` with following args:
```
--dir
/Users/lilin/work/git/flink/flink-tests
--version
1.19
--classes
org.apache.flink.test.checkpointing.StatefulJobSnapshotMigrationITCase
```
but will fail to load the specified class...
Before diving into each line of the generator code, try asking if you have encountered similar issues before, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2028808821
@masteryhx Could you take a look again when you have time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2041294975
Rebased latest 1.19 and triggered a new ci run before merging...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2041294668
@masteryhx Thanks for reviewing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2009918611
@masteryhx I've tried manually regenerate the specified test
`mvn clean package -Pgenerate-migration-test-data -Dgenerate.classes=StatefulJobSnapshotMigrationITCase -Dgenerate.version=1.19 -nsu -Dfast -DskipTests`
a strange log says the test class does not exist:
```
main:
[java] 743 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for module /Users/lilin/work/git/flink/flink-tests and version 1.19
[java] 749 [main] WARN org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Class StatefulJobSnapshotMigrationITCase does not exist.
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:42 min
[INFO] Finished at: 2024-03-20T23:35:58+08:00
```
Do you know what's wrong here? (If you know this, I haven't had time to dive into it yet)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2011312055
@masteryhx thank you!
Another input: there was no snapshot file generated for this case related data directory in previous releases, so I'm a bit skeptical that it's due to other changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2003442315
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d51dd5314e81eba834216de77a7ef93912996132",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d51dd5314e81eba834216de77a7ef93912996132",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d51dd5314e81eba834216de77a7ef93912996132 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2006256748
@masteryhx Thanks for looking into it and the tips! I'll try regrenerating data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2028805753
After some debugging, seems `MigrationTestsSnapshotGenerator` cannot work for the four failure tests ( https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58514&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=9218 ) even it logged generating succeed:
```
main:
[java] 1296 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for module /Users/lilin/work/git/flink/flink-tests and version 1.19
[java] 1345 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for org.apache.flink.test.checkpointing.StatefulJobWBroadcastStateMigrationITCase
[java] 1867 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Finish test data generating for org.apache.flink.test.checkpointing.StatefulJobWBroadcastStateMigrationITCase
```
So manually generate the corresponding savepoint/snapshot files by modifying the 4 cases that failed to generate automatically, let's complete this updating work first. (The problem on `MigrationTestsSnapshotGenerator` tracked by https://issues.apache.org/jira/browse/FLINK-34972)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "masteryhx (via GitHub)" <gi...@apache.org>.
masteryhx commented on PR #24517:
URL: https://github.com/apache/flink/pull/24517#issuecomment-2011189235
> @masteryhx I've tried manually regenerate the specified test `mvn clean package -Pgenerate-migration-test-data -Dgenerate.classes=StatefulJobSnapshotMigrationITCase -Dgenerate.version=1.19 -nsu -Dfast -DskipTests`
>
> a strange log says the test class does not exist:
>
> ```
> main:
> [java] 743 [main] INFO org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Start test data generating for module /Users/lilin/work/git/flink/flink-tests and version 1.19
> [java] 749 [main] WARN org.apache.flink.test.migration.MigrationTestsSnapshotGenerator [] - Class StatefulJobSnapshotMigrationITCase does not exist.
> [INFO] Executed tasks
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:42 min
> [INFO] Finished at: 2024-03-20T23:35:58+08:00
> ```
>
> Do you know what's wrong here? (If you know this, I haven't had time to dive into it yet)
1. The generate.classes should be configured as its complete path, e.g. org.apache.flink.test.checkpointing.StatefulJobSnapshotMigrationITCase
2. I just also tried to generate it but seems no result. I took a quick look and haven't found the reason. Maybe there are some issues for the generator logic. You could also debug this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34712][release] Generate reference data for state migration tests based on release-1.19.0 [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil merged PR #24517:
URL: https://github.com/apache/flink/pull/24517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org