You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/27 03:30:50 UTC
[GitHub] [flink] rkhachatryan commented on a change in pull request #12332: [FLINK-17820][task][checkpointing] Respect fileSizeThreshold in FsCheckpointStateOutputStream.flush()
rkhachatryan commented on a change in pull request #12332:
URL: https://github.com/apache/flink/pull/12332#discussion_r430263134
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStreamFactory.java
##########
@@ -266,6 +264,16 @@ public void flush() throws IOException {
}
}
+ /**
+ * Flush buffers to file if their size is above {@link #localStateThreshold}.
+ */
+ @Override
+ public void flush() throws IOException {
+ if (pos > localStateThreshold) {
Review comment:
Makes sense, expanded the condition.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStreamFactory.java
##########
@@ -247,15 +247,13 @@ public long getPos() throws IOException {
return pos + (outStream == null ? 0 : outStream.getPos());
}
- @Override
- public void flush() throws IOException {
+ private void flushToFile() throws IOException {
Review comment:
Done.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorageTest.java
##########
@@ -165,8 +165,7 @@ public void testDirectoriesForExclusiveAndSharedState() throws Exception {
CheckpointStateOutputStream exclusiveStream =
storageLocation.createCheckpointStateOutputStream(CheckpointedStateScope.EXCLUSIVE);
- exclusiveStream.write(42);
- exclusiveStream.flush();
Review comment:
Good point!
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStreamFactory.java
##########
@@ -266,6 +264,16 @@ public void flush() throws IOException {
}
}
+ /**
+ * Flush buffers to file if their size is above {@link #localStateThreshold}.
+ */
+ @Override
+ public void flush() throws IOException {
+ if (pos > localStateThreshold) {
Review comment:
I must have reverted it while updating the tests. Fixed.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriterTest.java
##########
@@ -95,12 +118,12 @@ public void flush() throws IOException {
FlushRecorder dataStream = new FlushRecorder();
final ChannelStateCheckpointWriter writer = new ChannelStateCheckpointWriter(
- 1L,
- new ChannelStateWriteResult(),
- new ChannelStateSerializerImpl(),
- NO_OP_RUNNABLE,
- new MemoryCheckpointOutputStream(42),
- dataStream
+ 1L,
Review comment:
Yes, accidental change, reverted it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org