You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/01/11 09:15:37 UTC
[GitHub] [flink] Myasuka opened a new pull request #18324: Introduce incremental/full checkpoint size stats
Myasuka opened a new pull request #18324:
URL: https://github.com/apache/flink/pull/18324
## What is the purpose of the change
Introduce incremental/full checkpoint size stats.
## Brief change log
- Introduce `getIncrementalStateSize` for state objects.
- Let `PlaceHolderStateHandler` hold state size information
- Enable changelog related state handles also consider incremental state size.
## Verifying this change
TODO
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788743464
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI. And we can explain the large incremental checkpoint size compared with short async duration in documentations of changelog state-backend.
I do think we need to think of it carefully how we define the incremental state size fo changelog state-backend.
If the materialization begins after `chk-9` and completes before `chk-11`. Will `chk-11` include that part?
Moreover, If we think the incremental checkpoint size is the incremental size compared with last completed checkpoint, what about the case that `chk-11` failed globally but `chk-12` succeed in the end, will `chk-12` include that materialized part?
Maybe we can change the definiation of incremental checkpoint size to "The incremental checkpoint size compared with last checkpoint (not the last completed one)", we can say only `chk-11` would include that materialization part no matter whether it succeed in the end.
By doing so, if we can avoid to reupload SST files in the future, we can also apply the semantics of incremental checkpoint size.
** update ** The semantics is different here for current incremental RocksDB checkpoint, as the incremental checkpoint size is actually "The incremental checkpoint size compared with last **completed** checkpoint".
Because of we did not implemet the feature of avoid reuploading SST files, the semantics is different here.
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI. And we can explain the large incremental checkpoint size compared with short async duration in documentations of changelog state-backend.
I do think we need to think of it carefully how we define the incremental state size fo changelog state-backend.
If the materialization begins after `chk-9` and completes before `chk-11`. Will `chk-11` include that part?
Moreover, If we think the incremental checkpoint size is the incremental size compared with last completed checkpoint, what about the case that `chk-11` failed globally but `chk-12` succeed in the end, will `chk-12` include that materialized part?
Maybe we can change the definiation of incremental checkpoint size to "The incremental checkpoint size compared with last checkpoint (not the last completed one)", we can say only `chk-11` would include that materialization part no matter whether it succeed in the end.
By doing so, if we can avoid to reupload SST files in the future, we can also apply the semantics of incremental checkpoint size.
**update** The semantics is different here for current incremental RocksDB checkpoint, as the incremental checkpoint size is actually "The incremental checkpoint size compared with last **completed** checkpoint".
Because of we did not implemet the feature of avoid reuploading SST files, the semantics is different here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790617559
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -113,6 +124,19 @@ public long getStateSize() {
+ nonMaterialized.stream().mapToLong(StateObject::getStateSize).sum();
}
+ @Override
+ public long getIncrementalStateSize() {
+ long incrementalStateSize =
+ incrementalMaterializeSize == undefinedIncrementalMaterializeSize
+ ? materialized.stream()
+ .mapToLong(StateObject::getIncrementalStateSize)
+ .sum()
+ : incrementalMaterializeSize;
Review comment:
Reached consensus in [another thread](https://github.com/apache/flink/pull/18324#discussion_r790249924).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034)
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784658077
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html
##########
@@ -45,12 +45,13 @@
>
<thead>
<tr>
- <th nzWidth="60px"></th>
+ <th nzWidth="70px"></th>
<th><strong>Name</strong></th>
<th><strong>Acknowledged</strong></th>
<th><strong>Latest Acknowledgment</strong></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th><strong>Incremental Checkpoint Data Size</strong></th>
+ <th><strong>Full Checkpoint Data Size</strong></th>
Review comment:
Agree, I will add a tooltip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784658411
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SubtaskStateStats.java
##########
@@ -85,6 +88,7 @@
this.subtaskIndex = subtaskIndex;
checkArgument(stateSize >= 0, "Negative state size");
this.stateSize = stateSize;
+ this.incrementalStateSize = incrementalStateSize;
Review comment:
Okay, I will.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784314617
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SubtaskStateStats.java
##########
@@ -85,6 +88,7 @@
this.subtaskIndex = subtaskIndex;
checkArgument(stateSize >= 0, "Negative state size");
this.stateSize = stateSize;
+ this.incrementalStateSize = incrementalStateSize;
Review comment:
Add `checkState` similar to `stateSize`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a403ce534516301bbcd2c1088f60f343e3ae903 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784298983
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -113,6 +124,19 @@ public long getStateSize() {
+ nonMaterialized.stream().mapToLong(StateObject::getStateSize).sum();
}
+ @Override
+ public long getIncrementalStateSize() {
+ long incrementalStateSize =
+ incrementalMaterializeSize == undefinedIncrementalMaterializeSize
+ ? materialized.stream()
+ .mapToLong(StateObject::getIncrementalStateSize)
+ .sum()
+ : incrementalMaterializeSize;
Review comment:
Depending on how we define "incremental state size", materialized part should be included or not:
1. if it's everything that was uploaded for **this** checkpoint, then it should
1. if it's the difference from the previous checkpoint, it should **not** be included
Right?
It seems problematic to find out what exactly was uploaded for **this** checkpoint because multiple checkpoints will likely include the same materialized state, and therefore report the same incremental state multiple times.
Besides that, the 2nd option seems more intuitive to me personally.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790734827
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
Yes, let's only report non-preemptively uploaded state and document it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789407543
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
"Async Checkpointed Data Size" is fine for me.
But I think metrics/things from UI users can learn (May not include in 1.15, but eventually, we should include)
- How much Data Size Exploding
- When a checkpoint includes a new Materialization
- Materialization size
- changelog sizes from the last complete checkpoint (that can roughly infer restore time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788432483
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
1. If `materializate-1` happens before `chk-10`, even `chk-10` failed finally, we would still get a large number of incremental state size. And the next succss `chk-11` would be small. I prefer this plan, WDYT?
2. I think we could leverage materialization id to judge the delta state handles and simplify the logic to a map of `<materization id, materizated state size>`
3. If we can use materizaton id to distiguish, the out-of-order problem could be solved.
4. I think this is reasonable as we have two parts here.
5. We can add documentation in changelog part then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
* d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755)
* 6fecb367871725b9fa2a88158b1844453a7230ae UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "1022317281",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784635071
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -113,6 +124,19 @@ public long getStateSize() {
+ nonMaterialized.stream().mapToLong(StateObject::getStateSize).sum();
}
+ @Override
+ public long getIncrementalStateSize() {
+ long incrementalStateSize =
+ incrementalMaterializeSize == undefinedIncrementalMaterializeSize
+ ? materialized.stream()
+ .mapToLong(StateObject::getIncrementalStateSize)
+ .sum()
+ : incrementalMaterializeSize;
Review comment:
I prefer the 2nd option and I actually caulcate it in this way. Actually, the `incrementalMaterializeSize` would not always be computed each time. `ChangelogKeyedStateBackend` would hold `lastCompletedHandles` and leverage that to caculate the incremental state size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
* 647c5b7e76e310ff363a31eb9de04c544f2effd9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789409368
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
How about "Async Persist Checkpoint Data Size", which only represents the async persist checkpoint data size between the checkpoint trigger and complete.
For the last three questions, I think we need to introduce another two columns to describe. Currently, we can only know materialization happened if the full checkpoint size decreased suddenly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "8a640ee27a79bcd67e3a9e9e42130beb445af173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a640ee27a79bcd67e3a9e9e42130beb445af173",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* 8a640ee27a79bcd67e3a9e9e42130beb445af173 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "1022317281",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789465033
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
The data uploaded during the async phase is (usually) created during the sync phase. So "Async Persist Checkpoint Data Size" is not very precise. The current UI does distinguish duration of sync and async phases; also nothing prevents backend from persisting everything during the sync phase.
Something like "Foreground persist data size" would be more precise, but it would confuse non-changelog users I guess. WDYT?
So maybe "Sync/async Persist Checkpoint Data Size"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d06095f451f4d1de35ad6271d9ba2dbaf0295500 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308)
* ce08a46add96553021558f2af3ecebc74a93e4e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
* d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755)
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
* e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034)
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r791624395
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html
##########
@@ -217,14 +227,26 @@
>
<thead>
<tr>
- <th nzWidth="60px"></th>
+ <th nzWidth="70px"></th>
<th><strong>ID</strong></th>
<th><strong>Status</strong></th>
<th><strong>Acknowledged</strong></th>
<th><strong>Trigger Time</strong></th>
<th><strong>Latest Acknowledgement</strong></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th>
+ <strong>
+ Checkpointed Data Size
+ <i
+ class="header-icon"
+ nz-icon
+ nz-tooltip
+ nzTooltipTitle="This column shows the persisted data size during the sync and async phases of that checkpoint, the value could be different from full checkpoint data size if incremental checkpoint or changelog is enabled."
Review comment:
```suggestion
nzTooltipTitle="The size of data persisted during the sync and async phases of a checkpoint. It's usually smaller than Full checkpoint data size if incremental checkpoints or changelog is enabled"
```
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html
##########
@@ -288,7 +311,19 @@
<tr>
<th></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th>
+ <strong>
+ Checkpointed Data Size
+ <i
+ class="header-icon"
+ nz-icon
+ nz-tooltip
+ nzTooltipTitle="This column shows the persisted data size during the sync and async phases of that checkpoint, the value could be different from full checkpoint data size if incremental checkpoint or changelog is enabled."
Review comment:
```suggestion
nzTooltipTitle="The size of data persisted during the sync and async phases of a checkpoint. It's usually smaller than Full checkpoint data size if incremental checkpoints or changelog is enabled"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a403ce534516301bbcd2c1088f60f343e3ae903 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244)
* d06095f451f4d1de35ad6271d9ba2dbaf0295500 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d06095f451f4d1de35ad6271d9ba2dbaf0295500 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784655215
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
I prefer the 2nd option, and we should count materialized state size in incremental state size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8d8e4393265d5b28db2881f6afc3c8048639ef Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954)
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789407543
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
"Async Checkpointed Data Size" is fine for me.
But I think metrics/things from UI users can learn (May not include in 1.15, but eventually we should include)
- Data Exploding
- When a checkpoint includes a new Materialization
- Materialization size
- changelog sizes from the last complete checkpoint (that can roughly infer restore time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789540788
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
"Checkpointed Data Size" still doesn't include materialization, right?
Sounds good to me then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a403ce534516301bbcd2c1088f60f343e3ae903 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244)
* d06095f451f4d1de35ad6271d9ba2dbaf0295500 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
* d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788743464
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI. And we can explain the large incremental checkpoint size compared with short async duration in documentations of changelog state-backend.
I do think we need to think of it carefully how we define the incremental state size fo changelog state-backend.
If the materialization begins after `chk-9` and completes before `chk-11`. Will `chk-11` include that part?
Moreover, If we think the incremental checkpoint size is the incremental size compared with last completed checkpoint, what about the case that `chk-11` failed globally but `chk-12` succeed in the end, will `chk-12` include that materialized part?
Maybe we can change the definiation of incremental checkpoint size to "The incremental checkpoint size compared with last checkpoint (not the last completed one)", we can say only `chk-11` would include that materialization part no matter whether it succeed in the end.
By doing so, if we can avoid to reupload SST files in the future, we can also apply the semantics of incremental checkpoint size.
**update:** The semantics is different here for current incremental RocksDB checkpoint, as the incremental checkpoint size is actually "The incremental checkpoint size compared with last **completed** checkpoint".
Because of we did not implemet the feature of avoid reuploading SST files, the semantics is different here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788792675
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
> If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI.
Why do we need to know this? (we can infer by the reduced total size but why?)
I'd propose to derive the definition from the user needs, which I desribed above (maybe you can add or correct?).
If so, "incremental checkpoint size" should be the size of data uploaded during the async phase of the given checkpoint.
This should work for: Changelog, RocksDB with re-upload, RocksDB without re-upload, or any other incremental state backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18324: Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009749855
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1a403ce534516301bbcd2c1088f60f343e3ae903 (Tue Jan 11 09:20:29 UTC 2022)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790720255
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointStats.java
##########
@@ -63,6 +93,7 @@
* @param totalSubtaskCount Total number of subtasks for the checkpoint.
* @param taskStats Task stats for each involved operator.
* @param numAcknowledgedSubtasks Number of acknowledged subtasks.
+ * @param checkpointedSize Total checkpointed size over all subtasks.
* @param stateSize Total checkpoint state size over all subtasks.
Review comment:
Maybe it would be enough to mention sync and async phases as per discussion [above](https://github.com/apache/flink/pull/18324#discussion_r790718434).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784312461
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
(ditto [1st comment](https://github.com/apache/flink/pull/18324#discussion_r784298983)): I think we should NOT count materialized state size in incremental state size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789535143
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
After discussion with Yuan, we can use two columns, one stay the same as "Checkpointed Data Size", the other would be "Full Checkpoint Data Size". WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788743464
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI. And we can explain the large incremental checkpoint size compared with short async duration in documentations of changelog state-backend.
I do think we need to think of it carefully how we define the incremental state size fo changelog state-backend.
If the materialization begins after `chk-9` and completes before `chk-11`. Will `chk-11` include that part?
Moreover, If we think the incremental checkpoint size is the incremental size compared with last completed checkpoint, what about the case that `chk-11` failed globally but `chk-12` succeed in the end, will `chk-12` include that materialized part?
Maybe we can change the definiation of incremental checkpoint size to "The incremental checkpoint size compared with last checkpoint (not the last completed one)", we can say only `chk-11` would include that materialization part no matter whether it succeed in the end.
By doing so, if we can avoid to reupload SST files in the future, we can also apply the semantics of incremental checkpoint size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
* 647c5b7e76e310ff363a31eb9de04c544f2effd9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789407543
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
"Async Checkpointed Data Size" is fine for me.
But I think metrics/things from UI users can learn (May not include in 1.15, but eventually we should include)
- How much Data Size Exploding
- When a checkpoint includes a new Materialization
- Materialization size
- changelog sizes from the last complete checkpoint (that can roughly infer restore time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788418063
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
I have tried to refactor to add a new non-default method in `CompositeStateHandle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8d8e4393265d5b28db2881f6afc3c8048639ef Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790661113
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointStats.java
##########
@@ -63,6 +93,7 @@
* @param totalSubtaskCount Total number of subtasks for the checkpoint.
* @param taskStats Task stats for each involved operator.
* @param numAcknowledgedSubtasks Number of acknowledged subtasks.
+ * @param checkpointedSize Total checkpointed size over all subtasks.
* @param stateSize Total checkpoint state size over all subtasks.
Review comment:
How about make `checkpointedSize` as `Total persisted data size over all subtasks of this checkpoint.` and leave `stateSize` as before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18324: Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a403ce534516301bbcd2c1088f60f343e3ae903 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784638145
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -50,17 +50,28 @@
class ChangelogStateBackendHandleImpl implements ChangelogStateBackendHandle {
private static final long serialVersionUID = 1L;
+ private static final long undefinedIncrementalMaterializeSize = -1L;
Review comment:
Hmm, I will take a look at this posoal. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784305375
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java
##########
@@ -204,6 +203,21 @@ public long getStateSize() {
return size;
}
+ @Override
+ public long getIncrementalStateSize() {
+ long size = StateUtil.getStateSize(metaStateHandle);
+
+ for (StreamStateHandle sharedStateHandle : sharedState.values()) {
+ size += sharedStateHandle.getIncrementalStateSize();
Review comment:
I guess this only works because `PlaceholderStreamStateHandle.getIncrementalStateSize` returns `0`, right?
But backend isn't requried to return placeholder IMO; in fact, it currently doesn't - without FLINK-25395/ #18297 (In the future, the latter PR quite likely will be reverted I think).
WDYT about computing incremental state size externally (in `RocksIncrementalSnapshotStrategy`) and storing it in metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788417725
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java
##########
@@ -204,6 +203,21 @@ public long getStateSize() {
return size;
}
+ @Override
+ public long getIncrementalStateSize() {
+ long size = StateUtil.getStateSize(metaStateHandle);
+
+ for (StreamStateHandle sharedStateHandle : sharedState.values()) {
+ size += sharedStateHandle.getIncrementalStateSize();
Review comment:
Already persist in the new IncrementalRemoteStateHandle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788679509
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059)
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790720660
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html
##########
@@ -81,6 +82,9 @@
{{ checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] | humanizeDuration }}
</td>
<td *ngIf="checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] < 0">n/a</td>
+ <td>
+ {{ checkPointDetail['tasks'][vertex.id]['checkpointed_size'] | humanizeBytes }}
+ </td>
<td>{{ checkPointDetail['tasks'][vertex.id]['state_size'] | humanizeBytes }}</td>
Review comment:
I see your point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
* e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034)
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r791624395
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html
##########
@@ -217,14 +227,26 @@
>
<thead>
<tr>
- <th nzWidth="60px"></th>
+ <th nzWidth="70px"></th>
<th><strong>ID</strong></th>
<th><strong>Status</strong></th>
<th><strong>Acknowledged</strong></th>
<th><strong>Trigger Time</strong></th>
<th><strong>Latest Acknowledgement</strong></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th>
+ <strong>
+ Checkpointed Data Size
+ <i
+ class="header-icon"
+ nz-icon
+ nz-tooltip
+ nzTooltipTitle="This column shows the persisted data size during the sync and async phases of that checkpoint, the value could be different from full checkpoint data size if incremental checkpoint or changelog is enabled."
Review comment:
```suggestion
nzTooltipTitle="The size of data persisted during the sync and async phases of a checkpoint. It's usually smaller than Full checkpoint data size if incremental checkpoints or changelog is enabled"
```
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html
##########
@@ -288,7 +311,19 @@
<tr>
<th></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th>
+ <strong>
+ Checkpointed Data Size
+ <i
+ class="header-icon"
+ nz-icon
+ nz-tooltip
+ nzTooltipTitle="This column shows the persisted data size during the sync and async phases of that checkpoint, the value could be different from full checkpoint data size if incremental checkpoint or changelog is enabled."
Review comment:
```suggestion
nzTooltipTitle="The size of data persisted during the sync and async phases of a checkpoint. It's usually smaller than Full checkpoint data size if incremental checkpoints or changelog is enabled"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755)
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788679930
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Thanks!
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788763720
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
Or maybe we can explain the semantics as:
- For general incremental checkpoint, the incremental checkpoint data size means the delta data size comapred with last completed checkpoint.
- For changelog based checkpoint, the incremental checkpoint data size means the delta data size comapred with last triggered checkpoint.
Alothough it looks a bit complex, we can remove the first one if we can support avoiding re-upload for RocksDB incremental checkpoint in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 647c5b7e76e310ff363a31eb9de04c544f2effd9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852)
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788792675
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
> If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI.
Why do we need to know this? (we can infer by the reduced total size but why?)
I'd propose to derive the definition from the user needs, which I desribed above (maybe you can add or correct?).
If so, "incremental checkpoint size" should be the size of data uploaded during the async phase of the given checkpoint.
This should work for: Changelog, RocksDB with re-upload, RocksDB without re-upload, or any other incremental state backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789338610
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
> If so, "incremental checkpoint size" should be the size of data uploaded during the async phase of the given checkpoint.
I think "async checkpointed size" instead of "incremental checkpoint size" might be more suitable for your defination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r789407543
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
"Async Checkpointed Data Size" is fine for me.
But I think metrics/things from UI users can learn:
- Data Exploding
- When a checkpoint includes a new Materialization
- Materialization size
- changelog sizes from the last complete checkpoint (that can roughly infer restore time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d06095f451f4d1de35ad6271d9ba2dbaf0295500 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308)
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1012931948
@rkhachatryan , what do you mean "How does this work after up-scaling? Will state size be reported e.g. twice"? Checkpoint state size would only report during taking checkpoint.
After restore, we have two choices for incremental state size:
1. same as previous complete checkpoint reported.
2. reset as same as full checkpoint size.
It seems the 1st one is better, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r785523680
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java
##########
@@ -204,6 +203,21 @@ public long getStateSize() {
return size;
}
+ @Override
+ public long getIncrementalStateSize() {
+ long size = StateUtil.getStateSize(metaStateHandle);
+
+ for (StreamStateHandle sharedStateHandle : sharedState.values()) {
+ size += sharedStateHandle.getIncrementalStateSize();
Review comment:
I mean `IncrementalRemoteKeyedStateHandle` could have a final variable `incrementalSize` and return it from `getIncrementalStateSize`.
To create a handle:
- On TM, it would be computed by `RocksIncrementalSnapshotStrategy`.
- On JM, it would be read from checkpoint metadata by `MetadataV2V3SerializerBase#serializeKeyedStateHandle` (and written).
It shouldn't change after the creation of `IncrementalRemoteKeyedStateHandle`, right?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
Let me expand my concerns directly in this comment:
1. If sending handles fails (or aborted) then UI will show inconsistent sizes (missing aborted info).
2. It requires handles to properly implement hashCode/equals. Otherwise, wrapped backend or writer can construct a new handle pointing to the same files; I think this is currently the case
3. If two checkpoints finish out-of-order and call `buildSnapshotResult` then the size will be incorrect
4. Incremental size is computed in two places: inside `getIncrementalStateSize`/backend and in `buildSnapshotResult` which seems excessive and fragile
5. After materialization, incremental size will be much bigger and async duration smaller. Wouldn't it be confusing for the users?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SubtaskStateStats.java
##########
@@ -85,6 +88,7 @@
this.subtaskIndex = subtaskIndex;
checkArgument(stateSize >= 0, "Negative state size");
this.stateSize = stateSize;
+ this.incrementalStateSize = incrementalStateSize;
Review comment:
Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784658633
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalKeyedStateHandle.java
##########
@@ -38,5 +38,5 @@
* created.
*/
@Nonnull
- Set<StateHandleID> getSharedStateHandleIDs();
+ Map<StateHandleID, StreamStateHandle> getSharedStateHandleIDs();
Review comment:
Agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8d8e4393265d5b28db2881f6afc3c8048639ef Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954)
* b05c9a39758867364fb4d00a709696785a3f054f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
* e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790726690
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
I think it's fine to give such description in details. If so, I think we have reached the aggreement that we do not count pre-emptively uploaded changelogs as the checkpointed data size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790718434
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
I think you're right in the sense that pre-emptively uploaded changes do not affect sync/async phase.
But I'd say those changes **are** uploaded during that checkpoint.
I think confusion comes from the definition of "during that checkpoint". It can mean two things:
1. the time from triggering a checkpoint to its completion (i.e. sync+async phase in task context)
2. the time between two barriers
I think we should explicitly clarify that we mean 1st one the docs and UI:
"during that checkpoint" -> "during sync and async phases of that checkpoint".
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790659327
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html
##########
@@ -81,6 +82,9 @@
{{ checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] | humanizeDuration }}
</td>
<td *ngIf="checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] < 0">n/a</td>
+ <td>
+ {{ checkPointDetail['tasks'][vertex.id]['checkpointed_size'] | humanizeBytes }}
+ </td>
<td>{{ checkPointDetail['tasks'][vertex.id]['state_size'] | humanizeBytes }}</td>
Review comment:
I think adding a new column could make user aware of this new feature. Moreover, changing this would make me update all pictures again, which is really annoying and I have already done this twice...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc3707aa5f06fdf332a6da162927edc8a86ac6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059)
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a403ce534516301bbcd2c1088f60f343e3ae903 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
* d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce08a46add96553021558f2af3ecebc74a93e4e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788694656
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
I agree with 2, 3, 4, but still have concerns on 1, and more importantly 5.
> 1. even chk-10 failed finally, we would still get a large number of incremental state size
But the materialization state size will be lost, right?
To fix this, we'll have to track **sent** materialization sizes (by materialization and checkpoint ID); mark them as **reported** upon receiving checkpoint confirmation; and then cleanup once next materialization completes.
To me, this complexity seems unjustified.
> 5. We can add documentation in changelog part then.
I think that most users want to know two things regarding checkpoint size:
1. Total size on DFS (i.e. including materialized state)
2. If a checkpoint was slow, how much data was uploaded in the async phase - i.e. **without** materialization; including materialization size would actually do harm here, as it will complicate the reasoning about async phase duration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05c9a39758867364fb4d00a709696785a3f054f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974)
* e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788679764
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fecb367871725b9fa2a88158b1844453a7230ae Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764)
* 8a164a916b92d8329191b4fcd3f34d04a176eead Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797)
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29954",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
}, {
"hash" : "b05c9a39758867364fb4d00a709696785a3f054f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29974",
"triggerID" : "b05c9a39758867364fb4d00a709696785a3f054f",
"triggerType" : "PUSH"
}, {
"hash" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30034",
"triggerID" : "e3cf1d9fc4e3cdfab655c15732c07ab5ee9b7935",
"triggerType" : "PUSH"
}, {
"hash" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30059",
"triggerID" : "1fc3707aa5f06fdf332a6da162927edc8a86ac6a",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"triggerType" : "PUSH"
}, {
"hash" : "0f6b7186ac9005552c5e722bc320f8a2c3c8eb37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158",
"triggerID" : "1022317281",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0f6b7186ac9005552c5e722bc320f8a2c3c8eb37 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30158)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1022317281
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784309007
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
1. Conceptually, incremental state is only relevant to `CompositeStateHandle` (which defines `registerSharedStates` method). WDYT about moving this method there?
2. Then we could force non-default implementation
3. In javadoc, could you clarify what "incremental" means (as per [comment above](https://github.com/apache/flink/pull/18324#discussion_r784298983))
4. In javadoc, could you clarify the relation to channel state? Or maybe in some other place, like `OperatorSubtaskState`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
* 647c5b7e76e310ff363a31eb9de04c544f2effd9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852)
* e53e06e0f6e692252e3e87b4fe797fb306c297ae UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790249924
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
> "Checkpointed Data Size" still doesn't include materialization
Yes, does not include the materialization part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790604449
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
I'm afraid `incrementalSize` computed here doesn't take pre-emptively uploaded state into account; it only computes the size of the last portion before checkpoint.
And in most cases, most of the state should be pre-emptively uploaded.
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -205,7 +205,7 @@ public SequenceNumber lastAppendedSequenceNumber() {
SequenceNumberRange range = SequenceNumberRange.generic(from, activeSequenceNumber);
if (range.size() == readyToReturn.size()) {
checkState(toUpload.isEmpty());
- return completedFuture(buildHandle(keyGroupRange, readyToReturn));
+ return completedFuture(buildHandle(keyGroupRange, readyToReturn, 0L));
Review comment:
Similar problem: in case if **all** state was pre-uploaded, the reported size will be zero.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointStats.java
##########
@@ -63,6 +93,7 @@
* @param totalSubtaskCount Total number of subtasks for the checkpoint.
* @param taskStats Task stats for each involved operator.
* @param numAcknowledgedSubtasks Number of acknowledged subtasks.
+ * @param checkpointedSize Total checkpointed size over all subtasks.
* @param stateSize Total checkpoint state size over all subtasks.
Review comment:
I think it makes sense to update the javadoc for `stateSize` (ditto other stats classes).
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html
##########
@@ -81,6 +82,9 @@
{{ checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] | humanizeDuration }}
</td>
<td *ngIf="checkPointDetail['tasks'][vertex.id]['end_to_end_duration'] < 0">n/a</td>
+ <td>
+ {{ checkPointDetail['tasks'][vertex.id]['checkpointed_size'] | humanizeBytes }}
+ </td>
<td>{{ checkPointDetail['tasks'][vertex.id]['state_size'] | humanizeBytes }}</td>
Review comment:
I think instead of adding a new column, we could show both full and incremental size separated by `/` or in parenthesis. That would give a bit more space for future info, and **probably** be more readable.
WDYT?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CompositeStateHandle.java
##########
@@ -50,4 +50,14 @@
* @param stateRegistry The registry where shared states are registered.
*/
void registerSharedStates(SharedStateRegistry stateRegistry, long checkpointID);
+
+ /**
+ * Returns the persisted data size during checkpoint execution in bytes. If incremental
+ * checkpoint is enabled, this value represents the incremental persisted data size, and
+ * ussually smaller than {@link #getStateSize()}. If the size is unknown, this method would
Review comment:
typo: "ussually" -> "usually"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790666268
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
If we take the pre-emptively uploaded changelogs as checkpointed data size, the semantics would be broken as those changelogs does not uploaded in that checkpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790726690
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -302,14 +302,16 @@ public void reset(SequenceNumber from, SequenceNumber to) {
}
private static ChangelogStateHandleStreamImpl buildHandle(
- KeyGroupRange keyGroupRange, NavigableMap<SequenceNumber, UploadResult> results) {
+ KeyGroupRange keyGroupRange,
+ NavigableMap<SequenceNumber, UploadResult> results,
+ long incrementalSize) {
List<Tuple2<StreamStateHandle, Long>> tuples = new ArrayList<>();
long size = 0;
for (UploadResult uploadResult : results.values()) {
tuples.add(Tuple2.of(uploadResult.getStreamStateHandle(), uploadResult.getOffset()));
size += uploadResult.getSize();
}
- return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size);
+ return new ChangelogStateHandleStreamImpl(tuples, keyGroupRange, size, incrementalSize);
Review comment:
I think it's fine to give such description in details. If so, I think we have reached the aggreement that we do not count pre-emptively uploaded changelogs as the checkpointed data size, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r790799788
##########
File path: flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java
##########
@@ -205,7 +205,7 @@ public SequenceNumber lastAppendedSequenceNumber() {
SequenceNumberRange range = SequenceNumberRange.generic(from, activeSequenceNumber);
if (range.size() == readyToReturn.size()) {
checkState(toUpload.isEmpty());
- return completedFuture(buildHandle(keyGroupRange, readyToReturn));
+ return completedFuture(buildHandle(keyGroupRange, readyToReturn, 0L));
Review comment:
As discussed above, this is by design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
}, {
"hash" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8d8e4393265d5b28db2881f6afc3c8048639ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
* ee8d8e4393265d5b28db2881f6afc3c8048639ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29953)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
}, {
"hash" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8b51e96ed7d79f855118a2102c6b631d4298c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
* e8b51e96ed7d79f855118a2102c6b631d4298c40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka merged pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka merged pull request #18324:
URL: https://github.com/apache/flink/pull/18324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784301442
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -50,17 +50,28 @@
class ChangelogStateBackendHandleImpl implements ChangelogStateBackendHandle {
private static final long serialVersionUID = 1L;
+ private static final long undefinedIncrementalMaterializeSize = -1L;
Review comment:
Why don't we store this size in checkpoint metadata? So that we can get rid of unknown size and show the correct size after recovery?
Nit: `UNDEFINED_INCREMENTAL_MATERIALIZE_SIZE` ?
##########
File path: flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html
##########
@@ -45,12 +45,13 @@
>
<thead>
<tr>
- <th nzWidth="60px"></th>
+ <th nzWidth="70px"></th>
<th><strong>Name</strong></th>
<th><strong>Acknowledged</strong></th>
<th><strong>Latest Acknowledgment</strong></th>
<th><strong>End to End Duration</strong></th>
- <th><strong>Checkpointed Data Size</strong></th>
+ <th><strong>Incremental Checkpoint Data Size</strong></th>
+ <th><strong>Full Checkpoint Data Size</strong></th>
Review comment:
WDYT about adding a tooltip here and/or in other added tags? (maybe copying the javadoc).
I think `title` attribute should work.
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
Review comment:
:+1:
We should **not** count `prevDeltaCopy.getIncrementalStateSize()` in `incrementalMaterializeSize`.
Could you add a comment that it's inttentional?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -113,6 +124,19 @@ public long getStateSize() {
+ nonMaterialized.stream().mapToLong(StateObject::getStateSize).sum();
}
+ @Override
+ public long getIncrementalStateSize() {
+ long incrementalStateSize =
+ incrementalMaterializeSize == undefinedIncrementalMaterializeSize
+ ? materialized.stream()
+ .mapToLong(StateObject::getIncrementalStateSize)
+ .sum()
+ : incrementalMaterializeSize;
Review comment:
Depending on how we define "incremental state size", materialized part should be included or not:
1. if it's everything that was uploaded for **this** checkpoint, then it should
1. if it's the difference from the previous checkpoint, it should **not** be included
Right?
And it's problematic to find out what exactly was uploaded for **this** checkpoint because multiple checkpoints will likely include the same materialized state, and therefore report the same incremental state multiple times.
Besides that, the 2nd option seems more intuitive to me personally.
WDYT?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
(ditto 1st comment): I think we should NOT count materialized state size in incremental state size
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##########
@@ -113,6 +124,19 @@ public long getStateSize() {
+ nonMaterialized.stream().mapToLong(StateObject::getStateSize).sum();
}
+ @Override
+ public long getIncrementalStateSize() {
+ long incrementalStateSize =
+ incrementalMaterializeSize == undefinedIncrementalMaterializeSize
+ ? materialized.stream()
+ .mapToLong(StateObject::getIncrementalStateSize)
+ .sum()
+ : incrementalMaterializeSize;
+ incrementalStateSize +=
+ nonMaterialized.stream().mapToLong(StateObject::getIncrementalStateSize).sum();
+ return incrementalStateSize;
Review comment:
This method can be **potentially** called for every rendering (though I think currently it is cached on JM).
Maybe we should we compute the value once in constructor?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java
##########
@@ -204,6 +203,21 @@ public long getStateSize() {
return size;
}
+ @Override
+ public long getIncrementalStateSize() {
+ long size = StateUtil.getStateSize(metaStateHandle);
+
+ for (StreamStateHandle sharedStateHandle : sharedState.values()) {
+ size += sharedStateHandle.getIncrementalStateSize();
Review comment:
I guess this only works because `PlaceholderStreamStateHandle.getIncrementalStateSize` returns `0`.
But backend isn't requried to return placeholder; in fact, it currently doesn't - without FLINK-25395/ #18297 (In the future, the latter PR quite likely will be reverted I think).
WDYT about computing incremental state size externally (in `RocksIncrementalSnapshotStrategy`) and storing it in metadata?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SubtaskStateStats.java
##########
@@ -85,6 +88,7 @@
this.subtaskIndex = subtaskIndex;
checkArgument(stateSize >= 0, "Negative state size");
this.stateSize = stateSize;
+ this.incrementalStateSize = incrementalStateSize;
Review comment:
`checkState` similar to `stateSize`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
1. Conceptually, incremental state is only relevant to `CompositeStateHandle` (which defines `registerSharedStates` method). WDYT about moving this method there?
2. Then we could force non-default implementation
3. In javadoc, could you clarify what "incremental" means (please see 1st comment)
4. In javadoc, could you clarify the relation to channel state? Or maybe in some other place, like `OperatorSubtaskState`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalKeyedStateHandle.java
##########
@@ -38,5 +38,5 @@
* created.
*/
@Nonnull
- Set<StateHandleID> getSharedStateHandleIDs();
+ Map<StateHandleID, StreamStateHandle> getSharedStateHandleIDs();
Review comment:
I guess `ID` has to be removed from the method name now..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784642224
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java
##########
@@ -204,6 +203,21 @@ public long getStateSize() {
return size;
}
+ @Override
+ public long getIncrementalStateSize() {
+ long size = StateUtil.getStateSize(metaStateHandle);
+
+ for (StreamStateHandle sharedStateHandle : sharedState.values()) {
+ size += sharedStateHandle.getIncrementalStateSize();
Review comment:
What do you mean storing it in meatadata? Current checkpoint stats shown in UI depend on summarizing state size from state handle. For other state backend, they can return the real incremental state size in `StateHandle#getIncrementalStateSize` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784648887
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Even applying to `CompositeStateHandle`, I think we still need to add a default method of `getIncrementalStateSize`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r785532350
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Do we? Default implementation allows to forget to implement it, and that's probably the case with `SubtaskState`, isn't it?
The interface `CompositeStateHandle` is a bit broken I think because it combines:
a) marking composites and
b) registering shared states
Which are not directly related (so it should probably be split into two)
So to me the less invasive and safe change is to move the method to `CompositeStateHandle` but remove `default`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r785532350
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java
##########
@@ -63,4 +63,14 @@
* @return Size of the state in bytes.
*/
long getStateSize();
+
+ /**
+ * Returns the incremental state size in bytes. If the size is unknown, this method would return
+ * same result as {@link #getStateSize()}.
+ *
+ * @return Size of incremental state in bytes.
+ */
+ default long getIncrementalStateSize() {
+ return getStateSize();
+ }
Review comment:
Do we? Default implementation allows to forget to implement it, and that's probably the case with `SubtaskState`, isn't it?
The interface `CompositeStateHandle` is a bit broken I think because it combines:
a) marking composites and
b) registering shared states
Which are not directly related (so it should probably be split into two)
So less invasive and safe change is to move the method to `CompositeStateHandle` but remove `default`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r784653673
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
Review comment:
Do you mean add a comment to describe this? Sure, I will.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #18324:
URL: https://github.com/apache/flink/pull/18324#discussion_r788743464
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##########
@@ -368,19 +372,34 @@ public boolean deregisterKeySelectionListener(KeySelectionListener<K> listener)
// collections don't change once started and handles are immutable
List<ChangelogStateHandle> prevDeltaCopy =
new ArrayList<>(changelogStateBackendStateCopy.getRestoredNonMaterialized());
+ long incrementalMaterializeSize = 0L;
if (delta != null && delta.getStateSize() > 0) {
prevDeltaCopy.add(delta);
+ incrementalMaterializeSize += delta.getIncrementalStateSize();
}
if (prevDeltaCopy.isEmpty()
&& changelogStateBackendStateCopy.getMaterializedSnapshot().isEmpty()) {
return SnapshotResult.empty();
} else {
+ List<KeyedStateHandle> materializedSnapshot =
+ changelogStateBackendStateCopy.getMaterializedSnapshot();
+ for (KeyedStateHandle keyedStateHandle : materializedSnapshot) {
+ if (!lastCompletedHandles.contains(keyedStateHandle)) {
+ incrementalMaterializeSize += keyedStateHandle.getStateSize();
Review comment:
If we do not include the materialization part, we will do not know when the materialization completed on each task via the web UI. And we can explain the large incremental checkpoint size compared with short async duration in documentations of changelog state-backend.
I do think we need to think of it carefully how we define the incremental state size fo changelog state-backend.
If the materialization begins after `chk-9` and completes before `chk-11`. Will `chk-11` include that part?
Moreover, If we think the incremental checkpoint size is the incremental size compared with last completed checkpoint, what about the case that `chk-11` failed globally but `chk-12` succeed in the end, will `chk-12` include that materialized part?
Maybe we can change the definiation of incremental checkpoint size to "The incremental checkpoint size compared with last checkpoint (not the last completed one)", we can say only `chk-11` would include that materialization part no matter whether it succeed in the end.
By doing so, if we can avoid to reupload SST files in the future, we can also apply the semantics of incremental checkpoint size.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18324: [FLINK-25557][checkpoint] Introduce incremental/full checkpoint size stats
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18324:
URL: https://github.com/apache/flink/pull/18324#issuecomment-1009752905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29244",
"triggerID" : "1a403ce534516301bbcd2c1088f60f343e3ae903",
"triggerType" : "PUSH"
}, {
"hash" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29308",
"triggerID" : "d06095f451f4d1de35ad6271d9ba2dbaf0295500",
"triggerType" : "PUSH"
}, {
"hash" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29720",
"triggerID" : "ce08a46add96553021558f2af3ecebc74a93e4e4",
"triggerType" : "PUSH"
}, {
"hash" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29755",
"triggerID" : "d07846bf2cc1c1925b7cacc1f9d7f186ba58b0e0",
"triggerType" : "PUSH"
}, {
"hash" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29764",
"triggerID" : "6fecb367871725b9fa2a88158b1844453a7230ae",
"triggerType" : "PUSH"
}, {
"hash" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29797",
"triggerID" : "8a164a916b92d8329191b4fcd3f34d04a176eead",
"triggerType" : "PUSH"
}, {
"hash" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803",
"triggerID" : "3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d",
"triggerType" : "PUSH"
}, {
"hash" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852",
"triggerID" : "647c5b7e76e310ff363a31eb9de04c544f2effd9",
"triggerType" : "PUSH"
}, {
"hash" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865",
"triggerID" : "e53e06e0f6e692252e3e87b4fe797fb306c297ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5eb71b47d3ffe31a8ebe545604173c1f4f8f3d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29803)
* 647c5b7e76e310ff363a31eb9de04c544f2effd9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29852)
* e53e06e0f6e692252e3e87b4fe797fb306c297ae Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29865)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org