You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/10/13 07:48:58 UTC
[GitHub] [flink] curcur opened a new pull request #17462: Metadata fix
curcur opened a new pull request #17462:
URL: https://github.com/apache/flink/pull/17462
## What is the purpose of the change
Write metadata after materialization, so that the log can be replayed based on the meta data during restore.
## Verifying this change
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749082756
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
Why is the test single thread? Because the materialization is triggered manually from the main thread now, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested in a separate ticket.
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
Why is the test single thread? Because the materialization is triggered manually from the main thread now, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749136798
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
I think it's very fragile to rely on
```
this.asyncOperationsThreadPool = Executors.newDirectExecutorService();
```
in constructor of a different class (`MockEnvironment`) in a different module.
That class is also used in many other tests. This change adds an **implicit** contract that its default `asyncOperationsThreadPool` **must** be direct.
Could you explain what was the motivation for the change in the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747958349
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
Can you be more explicit what is the difference between `forEach` and `for-loop` here?
Why foreach is blocking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r748043572
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
Sure, sorry, I mean the whole function is executed by the main task thread and this is blocking normal processing.
And `forEach` can be slower than a simple `for` loop, so it makes sense to shorten this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-966958130
I don't fully understand regarding the tests:
Why did they work before, were they broken?
Why missing flag reset for PQ states didn't fail anything?
And which tests exactly do you mean, is it `ChangelogDelegateHashMapTest.testMaterializedRestore`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749048095
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
It is a bit confusing unless digging into details how `triggerMaterialization()` is implemented. Because we also call schedule materialization in `triggerMaterialization()`.
Here, I mean each scheduling of materialization is triggered manually. It improves the readabilities of the test. I am a bit inclined to explicit commented here or if you have better suggestions to improve the comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510)
* e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r748044406
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
Yes, but can't one understand this from the code itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
To answer the questions above:
1. Why is the test single thread? Because the materialization is triggered **manually** from the main thread now, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested in a separate ticket.
2. For test, I mean `ChangelogDelegateHashMapTest.testMaterializedRestore`. You can compare against what has been changed on this test. As explained previously, now it includes both changelog + materialization part vs previously, it only includes materialization
If you still have questions, let's sync up offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747999451
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/AbstractChangelogState.java
##########
@@ -90,6 +90,10 @@ public void setCurrentNamespace(N namespace) {
return delegatedState.getStateIncrementalVisitor(recommendedMaxNumberOfReturnedRecords);
}
+ public void resetMetaDataWritten() {
Review comment:
I've changed `resetMetaDataWritten` to `resetWritingMetaFlag` (in this way, probably we do not need a java doc). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r746869320
##########
File path: .idea/vcs.xml
##########
@@ -20,6 +20,5 @@
</component>
<component name="VcsDirectoryMappings">
<mapping directory="$PROJECT_DIR$" vcs="Git" />
- <mapping directory="$PROJECT_DIR$/docs/themes/book" vcs="Git" />
</component>
-</project>
+</project>
Review comment:
I guess this change shouldn't belong to PR, please remove it if so.
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/AbstractChangelogState.java
##########
@@ -90,6 +90,10 @@ public void setCurrentNamespace(N namespace) {
return delegatedState.getStateIncrementalVisitor(recommendedMaxNumberOfReturnedRecords);
}
+ public void resetMetaDataWritten() {
Review comment:
1. add javadoc?
2. **nit**: a more abstract name, something like `onChangelogTruncated`? (so that details aren't exposed, and it shouldn't be renamed if they change in the future)
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
1. Please use `ChangelogState` instead of `AbstractChangelogState`
2. Please add a call for `priorityQueueStatesByName` (item above will force to implement it for PQ states)
3. Although this doesn't happen very often, it blocks processing, so I'd rather use for-loop
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
Could you explain this change? Shouldn't we make sure that materialization is completed at this point?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
The comment in code doesn't add much information IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r763868794
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogDelegateHashMapTest.java
##########
@@ -80,4 +80,12 @@ public void testMaterializedRestore() throws Exception {
ChangelogStateBackendTestUtils.testMaterializedRestore(
getStateBackend(), env, streamFactory);
}
+
+ @Test
+ public void testMaterializedRestorePriorityQueue() throws Exception {
Review comment:
Okay I see there's a dependency problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510)
* e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-988451245
squash commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r730566823
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -100,44 +100,6 @@
"Recognized shortcut names are 'jobmanager' and 'filesystem'.")
.build());
- /** Whether to enable state change log. */
Review comment:
Ah, I see, thanks. We should still add an annotation now, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
To answer the questions above:
1. Why is the test single thread? Because the materialization is triggered **manually** from the main thread now, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested in a separate ticket.
2. For test, I mean `ChangelogDelegateHashMapTest.testMaterializedRestore`. You can compare against what has been changed on this test. As explained previously, now it includes both changelog + materiliazation part.
If you still have questions, let's sync up offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
To answer the questions above:
1. Why is the test single thread? Because the materialization is triggered manually from the main thread, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested as well.
2. For test, I mean `ChangelogDelegateHashMapTest.testMaterializedRestore`. You can compare against what has been changed on this test. As explained previously, now it includes both changelog + materiliazation part.
If you still have questions, let's sync up offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
To answer the questions above:
1. Why is the test single thread? Because the materialization is triggered **manually** from the main thread now, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested as well.
2. For test, I mean `ChangelogDelegateHashMapTest.testMaterializedRestore`. You can compare against what has been changed on this test. As explained previously, now it includes both changelog + materiliazation part.
If you still have questions, let's sync up offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749129094
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
I think the page you are referring to compares indexed loops with java-1.5 foreach loops, but not with functional `forEach` from java 8 (and it's also not backed by any benchmarks).
In general, 1.8 loops are slower because of lambdas and function calls.
Yes, it may be neglibible, or may not. I don't think we can assume that it will,;and benchmarking seems non-trivial. So I'd just stay with faster for-loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r730565730
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -100,44 +100,6 @@
"Recognized shortcut names are 'jobmanager' and 'filesystem'.")
.build());
- /** Whether to enable state change log. */
Review comment:
Thanks for pointing this out @Airblader
These config options have not been visible to users yet. Changes were reverted before 1.14 release and re-enabled after. So technically, they are still new changes and do not break things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
}, {
"hash" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685)
* ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-966795230
> Thanks for the PR @curcur, I've left some comments, PTAL.
>
> PR description says it's verified by unit tests, but I couldn't find any (and the existing tests don't cover the change I believe).
The existing unit tests including a piece of materialized snapshot + non-materialized changelog
Recovery comes from both materialized snapshot + non-materialized changelog
It can not recover correctly if meta is missing.
> Commit messages are missing component; it's optional, but just "write metadata after materialization` is very abstract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-966795230
Thanks @rkhachatryan !
> Thanks for the PR @curcur, I've left some comments, PTAL.
>
> PR description says it's verified by unit tests, but I couldn't find any (and the existing tests don't cover the change I believe).
The existing unit tests including a piece of materialized snapshot + non-materialized changelog
Recovery comes from both materialized snapshot + non-materialized changelog
It can not recover correctly if meta is missing.
You can try it out by commenting the meta written, it will fail the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749101875
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
My point is that comment is the same as code, it basically repeats the method name (and "manually" is implied by the presense of the call itself).
https://flink.apache.org/contributing/code-style-and-quality-common.html#comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
* 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107)
* b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
}, {
"hash" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27767",
"triggerID" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27767)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r748102898
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
I don't see any changes related to threads in this PR, could you elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747958349
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
Can you be more explicit what is the difference between `forEach` and `for-loop` here?
Why it is blocking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968497357
@rkhachatryan , Let's sync up offline?
To answer the question above:
1. Why is the test single thread? Because the materialization is triggered manually from the main thread, the mailbox executor is sycn mailbox and async pool is a direct executor. Now everything is executed in a single thread. This is easier to reason about for a unit test. I am writing a ITTest as well. So multi-thread will be tested as well.
2. For test, I mean `ChangelogDelegateHashMapTest.testMaterializedRestore`. You can compare against what has been changed on this test. As explained previously, now it includes both changelog + materiliazation part.
If you still have questions, let's sync up offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747958911
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
Now everything happens in a single main thread, including the async phase.
No need of awaiting any more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r762895415
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -609,6 +609,11 @@ private ChangelogSnapshotState completeRestore(
((ChangelogState) changelogState).resetWritingMetaFlag();
}
+ for (ChangelogKeyGroupedPriorityQueue<?> priorityQueueState :
+ priorityQueueStatesByName.values()) {
+ priorityQueueState.resetWritingMetaFlag();
+ }
+
Review comment:
I think this change should be covered by a test (I mentioned this [earlier](https://github.com/apache/flink/pull/17462#issuecomment-966958130)).
It was already missed in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109)
* 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
* 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107)
* b51f758aa256fbc88737f0e12a53e5ed82cd067f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r746876858
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
This comment doesn't add much information IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747957300
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/AbstractChangelogState.java
##########
@@ -90,6 +90,10 @@ public void setCurrentNamespace(N namespace) {
return delegatedState.getStateIncrementalVisitor(recommendedMaxNumberOfReturnedRecords);
}
+ public void resetMetaDataWritten() {
Review comment:
for function naming, I think `resetMetaDataWritten` is more descriptive for what it is doing than `onChangelogTruncated`. I would prefer to keep it as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747958590
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
it means each materialization is triggered manually instead of automatically scheduling the next one.
This is to differentiate `periodicMaterializationManager` can schedule automatically even if triggerMaterialization is called separately. But in this test, only manually triggering is possible,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747999451
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/AbstractChangelogState.java
##########
@@ -90,6 +90,10 @@ public void setCurrentNamespace(N namespace) {
return delegatedState.getStateIncrementalVisitor(recommendedMaxNumberOfReturnedRecords);
}
+ public void resetMetaDataWritten() {
Review comment:
I've changed `resetMetaDataWritten` to `resetWritingMetaFlag`. WDYT?
java doc adds to the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496)
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17462: Metadata fix
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17462: Metadata fix
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur merged pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur merged pull request #17462:
URL: https://github.com/apache/flink/pull/17462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r747958590
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
it means each materialization is triggered manually instead of automatically schedule the next one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749046291
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
1. Is it for sure that `forEach` is slower than `for` loop in java? I've searched a bit and found contradict conclusions:
https://answers-short.com/en/Q%26A/page=48105dc27b0dd6653125b9bc3aa0257a
2. Even it is true, given there are very limited number of states, an't the differences really negligible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109)
* 26811181b8b06860cf50b28a8193114fc455bf37 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-968702444
I see, you removed 2nd `awaitMaterialization` call, thanks for clarifying.
I think it makes sense to preserve the scenario with two materializations (shouldn't be difficult to add a flag). WDYT?
I don't see the tests for PQ states, am I missing something?
Sure, let's discuss this offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r762071777
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +601,15 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ for (InternalKvState<K, ?, ?> changelogState : keyValueStatesByName.values()) {
+ checkState(changelogState instanceof ChangelogState);
+ ((ChangelogState) changelogState).resetWritingMetaFlag();
Review comment:
The loop is still through `keyValueStatesByName` only which only contains KV states.
What I meant is to loop through `priorityQueueStatesByName` as well which contains PQ states.
nit: checkState is unnecessary because JRE will check this on the next line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r763873697
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogDelegateHashMapTest.java
##########
@@ -80,4 +80,12 @@ public void testMaterializedRestore() throws Exception {
ChangelogStateBackendTestUtils.testMaterializedRestore(
getStateBackend(), env, streamFactory);
}
+
+ @Test
+ public void testMaterializedRestorePriorityQueue() throws Exception {
Review comment:
There are two reasons (we discussed this for `testMaterializedRestore`)
1. dependency
2. should expose a way to ignore trigger materialization because non-changelog does not support trigger materialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-987800906
Thanks for updating the PR @curcur, could you please squahs the commits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
}, {
"hash" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27767",
"triggerID" : "ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27685)
* ddc8a47426d41b78f448556c5b9a8bbf3ad8c19b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27767)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r746875368
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -600,8 +600,14 @@ private ChangelogSnapshotState completeRestore(
System.currentTimeMillis(),
streamFactory,
CHECKPOINT_OPTIONS),
- // TODO: add metadata to log FLINK-23170.
- upTo));
+ upTo);
+
+ // log metadata after materialization is triggered
+ keyValueStatesByName
+ .values()
+ .forEach(t -> ((AbstractChangelogState) t).resetMetaDataWritten());
Review comment:
1. Please use `ChangelogState` instead of `AbstractChangelogState`
2. Please add a call for `priorityQueueStatesByName` (item above will force to implement it for PQ states)
3. Although this doesn't happen very often, it blocks processing, so I'd rather use for-loop instead of `forEach`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
* 881206c906525fb7f4d1f9391bc3b196838d62fa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r728368143
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -100,44 +100,6 @@
"Recognized shortcut names are 'jobmanager' and 'filesystem'.")
.build());
- /** Whether to enable state change log. */
Review comment:
This class has no API annotation, so technically this isn't a breaking change, but in reality it kind of is. We should probably first deprecate this before removing it?
Also we should add an API annotation to the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107)
* b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r730579014
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -100,44 +100,6 @@
"Recognized shortcut names are 'jobmanager' and 'filesystem'.")
.build());
- /** Whether to enable state change log. */
Review comment:
Yes, I totally agree. I was about to say so :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r730837114
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##########
@@ -100,44 +100,6 @@
"Recognized shortcut names are 'jobmanager' and 'filesystem'.")
.build());
- /** Whether to enable state change log. */
Review comment:
BTW, the config moved to `StateChangelogOptions` is annotated as @PublicEvolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-966795230
Thanks @rkhachatryan !
> Thanks for the PR @curcur, I've left some comments, PTAL.
>
> PR description says it's verified by unit tests, but I couldn't find any (and the existing tests don't cover the change I believe).
The existing unit tests including a piece of materialized snapshot + non-materialized changelog
Recovery comes from both materialized snapshot + non-materialized changelog
It can not recover correctly if meta is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r748102898
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 22));
keyedBackend.setCurrentKey(3);
state.update(new StateBackendTestBase.TestPojo("u3", 3));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
-
Review comment:
I don't see any changes related to threads in this PR, could you elaborate?
Also I'm not sure it's not too fragile to rely on this. But at least, please add a comments at points where the materialization is expected to be copmleted and why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002)
* 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r749048095
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogStateBackendTestUtils.java
##########
@@ -158,16 +155,15 @@ public static void testMaterializedRestore(
keyedBackend.setCurrentKey(2);
state.update(new StateBackendTestBase.TestPojo("u2", 2));
- awaitMaterialization(keyedBackend, env.getMainMailboxExecutor());
+ // trigger materialization manually
+ periodicMaterializationManager.triggerMaterialization();
Review comment:
It is a bit confusing unless digging into details how `triggerMaterialization()` is implemented. Because we also call schedule materialization in `triggerMaterialization()`.
Here, I mean each scheduling of materialization is triggered manually. It improves the readabilities of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942025793
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit de953e065ada40720bd5118aa8fb43842b27733b (Wed Oct 13 07:51:18 UTC 2021)
**Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
}, {
"hash" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "459916eaca61c9210c403ee4f4125cfbf3590fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
* 459916eaca61c9210c403ee4f4125cfbf3590fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496)
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17462:
URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "de953e065ada40720bd5118aa8fb43842b27733b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002",
"triggerID" : "de953e065ada40720bd5118aa8fb43842b27733b",
"triggerType" : "PUSH"
}, {
"hash" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107",
"triggerID" : "881206c906525fb7f4d1f9391bc3b196838d62fa",
"triggerType" : "PUSH"
}, {
"hash" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109",
"triggerID" : "b51f758aa256fbc88737f0e12a53e5ed82cd067f",
"triggerType" : "PUSH"
}, {
"hash" : "26811181b8b06860cf50b28a8193114fc455bf37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510",
"triggerID" : "26811181b8b06860cf50b28a8193114fc455bf37",
"triggerType" : "PUSH"
}, {
"hash" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496",
"triggerID" : "e5dc41ce526335e5a1ef9adf8a434f89dd3e4679",
"triggerType" : "PUSH"
}, {
"hash" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600",
"triggerID" : "11561c01f7d5f1f92c2fb98db39ee5db34c7afab",
"triggerType" : "PUSH"
}, {
"hash" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679",
"triggerID" : "af61b462d33f2f903a74bc4e7295a022a32fac1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11561c01f7d5f1f92c2fb98db39ee5db34c7afab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27600)
* af61b462d33f2f903a74bc4e7295a022a32fac1a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27679)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r763803557
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogDelegateHashMapTest.java
##########
@@ -80,4 +80,12 @@ public void testMaterializedRestore() throws Exception {
ChangelogStateBackendTestUtils.testMaterializedRestore(
getStateBackend(), env, streamFactory);
}
+
+ @Test
+ public void testMaterializedRestorePriorityQueue() throws Exception {
Review comment:
Do we really need to add this method to every descendant of `HashMapStateBackendTest`? Can't we only add it to `HashMapStateBackendTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #17462: [FLINK-23170] Write metadata after materialization
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #17462:
URL: https://github.com/apache/flink/pull/17462#discussion_r763873697
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/state/changelog/ChangelogDelegateHashMapTest.java
##########
@@ -80,4 +80,12 @@ public void testMaterializedRestore() throws Exception {
ChangelogStateBackendTestUtils.testMaterializedRestore(
getStateBackend(), env, streamFactory);
}
+
+ @Test
+ public void testMaterializedRestorePriorityQueue() throws Exception {
Review comment:
There are two reasons (we discussed when testMaterializedRestore)
1. dependency
2. should expose a way to ignore trigger materialization because non-changelog does not support trigger materialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org