You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/02/02 22:22:25 UTC
[GitHub] [flink] rkhachatryan opened a new pull request #14838: [WIP][FLINK-XXXXX][state] Add StateChangelog API
rkhachatryan opened a new pull request #14838:
URL: https://github.com/apache/flink/pull/14838
This is a preview of StateChangelog API proposed in FLIP-158.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772050707
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573808224
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
Review comment:
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r577873641
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/StateChangelogWriterFactoryLoader.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.core.plugin.PluginManager;
+
+import java.util.Iterator;
+
+/** A thin wrapper around {@link PluginManager} to load {@link StateChangelogWriterFactory}. */
+@Internal
+public class StateChangelogWriterFactoryLoader {
+ private final PluginManager pluginManager;
+
+ public StateChangelogWriterFactoryLoader(PluginManager pluginManager) {
+ this.pluginManager = pluginManager;
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ public Iterator<StateChangelogWriterFactory> load() {
+ return pluginManager.load(StateChangelogWriterFactory.class);
+ }
+}
Review comment:
It should suggest users to rely on SPI/plugins instead of direct instantiation.
SPI part was missing which I guess confused you, sorry for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r569963528
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/changelog/StateChangelogClientTest.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.changelog.fs.StateChangeFormat;
+import org.apache.flink.changelog.inmemory.InMemoryStateChangelogClient;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateChange;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.util.function.Function.identity;
+import static java.util.stream.Collectors.toMap;
+import static java.util.stream.StreamSupport.stream;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+/** {@link InMemoryStateChangelogClient} test. */
+@RunWith(Parameterized.class)
+public class StateChangelogClientTest {
+
+ private final Random random = new Random();
+ private final LogClientType clientType;
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Object[] parameters() {
+ return LogClientType.values();
+ }
+
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ public StateChangelogClientTest(LogClientType logClientType) {
+ this.clientType = logClientType;
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testNoAppendAfterClose() throws IOException {
+ StateChangelogWriter<?> writer =
+ clientType.create(this).createWriter(new OperatorID(), KeyGroupRange.of(0, 0));
+ writer.close();
+ writer.append(0, new byte[0]);
+ }
+
+ @Test
+ public void testWriteAndRead() throws Exception {
+ KeyGroupRange kgRange = KeyGroupRange.of(0, 5);
+ Map<Integer, List<byte[]>> appendsByKeyGroup = generateAppends(kgRange, 10, 20);
+
+ try (StateChangelogClient<?> client = clientType.create(this);
+ StateChangelogWriter<?> writer = client.createWriter(new OperatorID(), kgRange)) {
+ SequenceNumber from = writer.lastAppendedSqn();
+ appendsByKeyGroup.forEach(
+ (group, appends) -> appends.forEach(bytes -> writer.append(group, bytes)));
+
+ StateChangelogHandle<?> handle = writer.persist(from).get();
+
+ assertByteMapsEqual(appendsByKeyGroup, extract(handle));
+ }
+ }
+
+ private void assertByteMapsEqual(
+ Map<Integer, List<byte[]>> expected, Map<Integer, List<byte[]>> actual) {
+ assertEquals(expected.size(), actual.size());
+ for (Map.Entry<Integer, List<byte[]>> e : expected.entrySet()) {
+ List<byte[]> expectedList = e.getValue();
+ List<byte[]> actualList = actual.get(e.getKey());
+ Iterator<byte[]> ite = expectedList.iterator(), ale = actualList.iterator();
+ while (ite.hasNext() && ale.hasNext()) {
+ assertArrayEquals(ite.next(), ale.next());
+ }
+ assertFalse(ite.hasNext());
+ assertFalse(ale.hasNext());
+ }
+ }
+
+ private Map<Integer, List<byte[]>> extract(StateChangelogHandle<?> handle) throws Exception {
+ Map<Integer, List<byte[]>> changes = new HashMap<>();
+ //noinspection unchecked
+ StateChangelogHandle<Object> objHandle = (StateChangelogHandle<Object>) handle;
+ try (CloseableIterator<StateChange> it = objHandle.getChanges(clientType.getContext())) {
+ while (it.hasNext()) {
+ StateChange change = it.next();
+ changes.computeIfAbsent(change.getKeyGroup(), k -> new ArrayList<>())
+ .add(change.getChange());
+ }
+ }
+ return changes;
+ }
+
+ private Map<Integer, List<byte[]>> generateAppends(
+ KeyGroupRange kgRange, int keyLen, int appendsPerGroup) {
+ return stream(kgRange.spliterator(), false)
+ .collect(toMap(identity(), unused -> generateData(appendsPerGroup, keyLen)));
+ }
+
+ private List<byte[]> generateData(int numAppends, int keyLen) {
+ return Stream.generate(() -> randomBytes(keyLen))
+ .limit(numAppends)
+ .collect(Collectors.toList());
+ }
+
+ private byte[] randomBytes(int len) {
+ byte[] bytes = new byte[len];
+ random.nextBytes(bytes);
+ return bytes;
+ }
+
+ private enum LogClientType {
+ MEMORY {
+ @Override
+ public StateChangelogClient<?> create(StateChangelogClientTest logClientTest) {
+ return new InMemoryStateChangelogClient();
+ }
+
+ @Override
+ public StateChangeFormat getContext() {
Review comment:
"StateChangeFormat" is missing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13baba5c0d15572ad8295d728fe2aedabff3342d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405)
* 6dd7135a501c0fcbe6bee4d08b561712bd3b3653 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773636933
1. > But this is where/the name I planned to put the WrappedStateBackend.
In my opinion, both backend and changelog should reside in the same module, at least for now.
2. I fixed the `StateChangeFormat` issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
}, {
"hash" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446",
"triggerID" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
* 000fa3e28def66c51ebf7b5975d8b610cccc6988 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773054677
1. The StateChangeLog is put under flink-state-backends,
under <module>flink-statebackend-changelog</module>
But this is where/the name I planned to put the WrappedStateBackend. What do you think @rkhachatryan
2. The code is missing "StateChangeFormat" right now. Would you make a workaround so that I can try it a bit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r574306144
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
+
+ /**
+ * Durably persist previously {@link #append(int, byte[]) appended} data. After this call, one
+ * of {@link #confirm(SequenceNumber, SequenceNumber) confirm}, {@link #reset(SequenceNumber,
+ * SequenceNumber) reset}, or {@link #truncate(SequenceNumber) truncate} eventually must be
+ * called for the corresponding change set. with reset/truncate/confirm methods?
+ *
+ * @param from inclusive
+ */
+ CompletableFuture<Handle> persist(SequenceNumber from) throws IOException;
Review comment:
Why is this "persist from"? From where until what? Shouldn't this be "persist until" or "persist up to"?
If not, then please explain this in the javadoc?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
Review comment:
`lastAppendedSqn` -> lastAppendedSequenceNumber`
##########
File path: flink-state-backends/pom.xml
##########
@@ -39,5 +39,6 @@ under the License.
<modules>
<module>flink-statebackend-rocksdb</module>
<module>flink-statebackend-heap-spillable</module>
+ <module>flink-statebackend-changelog</module>
Review comment:
I think the interfaces definitions should be in the `flink-runtime`, probably along the in memory implementation. We know there are other usecases where we might want to use it, furthermore there will be different implementations of the changelog.
Maybe file based implementation could be placed anywhere for the time being, but I would need to think about this structure a bit more.
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogClient.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+/**
+ * {@link StateChangelogWriter} factory. Scoped to a single entity (e.g. a SubTask or
+ * OperatorCoordinator).
+ */
+@Internal
+public interface StateChangelogClient<Handle extends StateChangelogHandle<?>>
Review comment:
Why is factory class named client? I had no idea what `StateChangelogClient` is doing looking at the name (I was more expecting it to be an internal class used by the changelog to perform the writes), and it's a bit funny that javadoc makes it easy to understand in the first sentence by calling it a `factory` :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r572644106
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
+
+ private transient SharedStateRegistry stateRegistry;
+
+ public StateChangelogHandleStreamImpl(
+ List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets, KeyGroupRange keyGroupRange) {
+ this.handlesAndOffsets = handlesAndOffsets;
+ this.keyGroupRange = keyGroupRange;
+ }
+
+ @Override
+ public void registerSharedStates(SharedStateRegistry stateRegistry) {
+ this.stateRegistry = stateRegistry;
+ handlesAndOffsets.forEach(
+ handleAndOffset ->
+ stateRegistry.registerReference(
+ getKey(handleAndOffset.f0), handleAndOffset.f0));
+ }
+
+ @Override
+ public KeyGroupRange getKeyGroupRange() {
+ return keyGroupRange;
+ }
+
+ @Nullable
+ @Override
+ public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+ KeyGroupRange offsets = keyGroupRange.getIntersection(keyGroupRange);
+ if (offsets.getNumberOfKeyGroups() == 0) {
+ return null;
+ }
+ return new StateChangelogHandleStreamImpl(handlesAndOffsets, offsets);
+ }
+
+ @Override
+ public CloseableIterator<StateChange> getChanges(StateChangeStreamReader reader) {
+ return new CloseableIterator<StateChange>() {
+ private final Iterator<Tuple2<StreamStateHandle, Long>> handleIterator =
+ handlesAndOffsets.iterator();
+
+ private CloseableIterator<StateChange> current = CloseableIterator.empty();
+
+ @Override
+ public boolean hasNext() {
+ advance();
+ return current.hasNext();
+ }
+
+ @Override
+ public StateChange next() {
+ advance();
+ return current.next();
+ }
+
+ private void advance() {
+ while (!current.hasNext() && handleIterator.hasNext()) {
+ Tuple2<StreamStateHandle, Long> tuple2 = handleIterator.next();
+ try {
+ current = reader.read(tuple2.f0, tuple2.f1);
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ current.close();
+ }
+ };
+ }
+
+ @Override
+ public void discardState() {
+ handlesAndOffsets.forEach(
+ handleAndOffset -> stateRegistry.unregisterReference(getKey(handleAndOffset.f0)));
+ }
+
+ @Override
+ public long getStateSize() {
+ return 0;
+ }
+
+ private static SharedStateRegistryKey getKey(StreamStateHandle stateHandle) {
Review comment:
I am not quite familiar with how sharing is exactly working, so forgive me if I am asking obvious questions:
1. Sharing is per state, or per task, or per state backend or multiple state backends? I understand it is at least per state changelog, and state changelog is mapping to each state backend. So it is probably per state backend or multiple state backends.
2. I understand "at least" sharable changes have to come from the same state backend, right? So what is the problem if using backend UUID? Or multiple backends bundles state changes altogether in a file?
If I can understand the above two questions, I think I can understand why state backend UUID can not be used as a key. And maybe rewrite the comment this way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r576239008
##########
File path: flink-state-backends/pom.xml
##########
@@ -39,5 +39,6 @@ under the License.
<modules>
<module>flink-statebackend-rocksdb</module>
<module>flink-statebackend-heap-spillable</module>
+ <module>flink-statebackend-changelog</module>
Review comment:
I'd say there **might** be use case where we could useit. But they are even close to design.
And it and **might** have different implementations.
I have concerns regarding both and whether MVP the right phase to expose API.
But I'm also fine with putting it under `flink-runtime` and marking everything `@Internal`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573788615
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
Review comment:
> What is the second field, offset to the file?
Yes, the 2nd field in `handlesAndOffsets` is offset (to file or other handle).
> Would it be more clear to make it a separate class, with each field a specific name and java doc/comment?
To me, replacing `Tuple2<...>` with `HandleAndOffset` class isn't much better; and introducing a new abstraction like `StatePointer` isn't necessarily more clear.
Using `long` and `Tuple2` in `StateHandle` implementations is common, so I don't think we should change it.
> The structure might also change if we want to use something else (DB) instead of a blob DFS?
In that case, a new `StateChangelogHandle` implementation should be added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r576236266
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
Review comment:
We've already agreed on this signature before but I can change it if you'd like.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c536632abeeef101cd3a89edcdf63f1e950eff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966)
* 38b7950b0c94efc6ed0dfc70828712e3b353ca65 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
}, {
"hash" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446",
"triggerID" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 000fa3e28def66c51ebf7b5975d8b610cccc6988 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-780789591
Thanks for reviewing @pnowojski .
I moved StateHandle classes to `org.apache.flink.runtime.state.changelog`. Other classes were already there.
I squashed the commits to remove confusing state with different packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r572613154
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChange.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+/** Change of state of a keyed operator. Used for generic incremental checkpoints. */
+@Internal
+public class StateChange {
Review comment:
Maybe rename this to `KeyedStateChange` since it relates changes to a key group?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
Review comment:
`Tuple2<StreamStateHandle, Long>` represents a series of state changes of previous persistent StateChangeLog checkpoints/snapshot I guess.
What is the second field, offset to the file?
Would it be more clear to make it a separate class, with each field a specific name and java doc/comment?
The structure might also change if we want to use something else (DB) instead of a blob DFS?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
+
+ private transient SharedStateRegistry stateRegistry;
+
+ public StateChangelogHandleStreamImpl(
+ List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets, KeyGroupRange keyGroupRange) {
+ this.handlesAndOffsets = handlesAndOffsets;
+ this.keyGroupRange = keyGroupRange;
+ }
+
+ @Override
+ public void registerSharedStates(SharedStateRegistry stateRegistry) {
+ this.stateRegistry = stateRegistry;
+ handlesAndOffsets.forEach(
+ handleAndOffset ->
+ stateRegistry.registerReference(
+ getKey(handleAndOffset.f0), handleAndOffset.f0));
Review comment:
see, confusing here; ".f0", same comment to make the Tuple2 -> Class
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
+
+ private transient SharedStateRegistry stateRegistry;
+
+ public StateChangelogHandleStreamImpl(
+ List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets, KeyGroupRange keyGroupRange) {
+ this.handlesAndOffsets = handlesAndOffsets;
+ this.keyGroupRange = keyGroupRange;
+ }
+
+ @Override
+ public void registerSharedStates(SharedStateRegistry stateRegistry) {
+ this.stateRegistry = stateRegistry;
+ handlesAndOffsets.forEach(
+ handleAndOffset ->
+ stateRegistry.registerReference(
+ getKey(handleAndOffset.f0), handleAndOffset.f0));
+ }
+
+ @Override
+ public KeyGroupRange getKeyGroupRange() {
+ return keyGroupRange;
+ }
+
+ @Nullable
+ @Override
+ public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+ KeyGroupRange offsets = keyGroupRange.getIntersection(keyGroupRange);
+ if (offsets.getNumberOfKeyGroups() == 0) {
+ return null;
+ }
+ return new StateChangelogHandleStreamImpl(handlesAndOffsets, offsets);
+ }
+
+ @Override
+ public CloseableIterator<StateChange> getChanges(StateChangeStreamReader reader) {
+ return new CloseableIterator<StateChange>() {
+ private final Iterator<Tuple2<StreamStateHandle, Long>> handleIterator =
+ handlesAndOffsets.iterator();
+
+ private CloseableIterator<StateChange> current = CloseableIterator.empty();
+
+ @Override
+ public boolean hasNext() {
+ advance();
+ return current.hasNext();
+ }
+
+ @Override
+ public StateChange next() {
+ advance();
+ return current.next();
+ }
+
+ private void advance() {
+ while (!current.hasNext() && handleIterator.hasNext()) {
+ Tuple2<StreamStateHandle, Long> tuple2 = handleIterator.next();
+ try {
+ current = reader.read(tuple2.f0, tuple2.f1);
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ current.close();
+ }
+ };
+ }
+
+ @Override
+ public void discardState() {
+ handlesAndOffsets.forEach(
+ handleAndOffset -> stateRegistry.unregisterReference(getKey(handleAndOffset.f0)));
+ }
+
+ @Override
+ public long getStateSize() {
+ return 0;
+ }
+
+ private static SharedStateRegistryKey getKey(StreamStateHandle stateHandle) {
Review comment:
I am not quite familiar with how sharing is exactly working, so forgive me if I am asking obvious questions:
1. Sharing is per state, or per task, or per state backend or multiple state backends? I understand it is at least per state changelog, and state changelog is mapping to each state backend. So it is per state backend or multiple state backends.
2. I understand "at least" changes from the same state backend would be sharable, right? So what is the problem if using backend UUID? Or multiple backends bundles state changes altogether in a file?
If I can understand the above two questions, I think I can understand why state backend UUID can not be used as a key. And maybe rewrite the comment this way?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/SequenceNumber.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A logical timestamp to draw a boundary between the materialized and non-materialized changes.
+ * Maintained by the state backend but implementations may choose to move its generation to {@link
Review comment:
"Maintained by the state backend", hmm. I was thinking about whether this info (last materialized sequence number) has to be maintained within the state backend.
If it is simply a boundary between materialized and non-materialized changes for state changelog, why we can not put it within the state changelog? Is it used in some other places?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
Review comment:
This is the method called from Statebackend when updating a state?
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
+
+ /**
+ * Durably persist previously {@link #append(int, byte[]) appended} data. After this call, one
+ * of {@link #confirm(SequenceNumber, SequenceNumber) confirm}, {@link #reset(SequenceNumber,
+ * SequenceNumber) reset}, or {@link #truncate(SequenceNumber) truncate} eventually must be
+ * called for the corresponding change set. with reset/truncate/confirm methods?
+ *
+ * @param from inclusive
+ */
+ CompletableFuture<Handle> persist(SequenceNumber from) throws IOException;
Review comment:
This is the method to call when snapshotting is called from the state backend?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r576243196
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogClient.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+/**
+ * {@link StateChangelogWriter} factory. Scoped to a single entity (e.g. a SubTask or
+ * OperatorCoordinator).
+ */
+@Internal
+public interface StateChangelogClient<Handle extends StateChangelogHandle<?>>
Review comment:
Thanks for pointing out. This is a leftover from the previous design which assumed a dedicated component.
I'll rename it to `StateChangelogWriterFactory`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773054677
The StateChangeLog is put under flink-state-backends,
under <module>flink-statebackend-changelog</module>
But this is where/the name I planned to put the WrappedStateBackend. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dd7135a501c0fcbe6bee4d08b561712bd3b3653 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410)
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38b7950b0c94efc6ed0dfc70828712e3b353ca65 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231)
* 1506a66da8f77069706d654fc43335e21fcf154e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1506a66da8f77069706d654fc43335e21fcf154e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573808464
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
+
+ /**
+ * Durably persist previously {@link #append(int, byte[]) appended} data. After this call, one
+ * of {@link #confirm(SequenceNumber, SequenceNumber) confirm}, {@link #reset(SequenceNumber,
+ * SequenceNumber) reset}, or {@link #truncate(SequenceNumber) truncate} eventually must be
+ * called for the corresponding change set. with reset/truncate/confirm methods?
+ *
+ * @param from inclusive
+ */
+ CompletableFuture<Handle> persist(SequenceNumber from) throws IOException;
Review comment:
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan merged pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan merged pull request #14838:
URL: https://github.com/apache/flink/pull/14838
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r577947431
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/StateChangelogWriterFactoryLoader.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.core.plugin.PluginManager;
+
+import java.util.Iterator;
+
+/** A thin wrapper around {@link PluginManager} to load {@link StateChangelogWriterFactory}. */
+@Internal
+public class StateChangelogWriterFactoryLoader {
+ private final PluginManager pluginManager;
+
+ public StateChangelogWriterFactoryLoader(PluginManager pluginManager) {
+ this.pluginManager = pluginManager;
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ public Iterator<StateChangelogWriterFactory> load() {
+ return pluginManager.load(StateChangelogWriterFactory.class);
+ }
+}
Review comment:
optional nit: in that case I would add a sentence that changelog factories should be loaded using plugin mechanism instead of directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r570577515
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/changelog/StateChangelogClientTest.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.changelog.fs.StateChangeFormat;
+import org.apache.flink.changelog.inmemory.InMemoryStateChangelogClient;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateChange;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.util.function.Function.identity;
+import static java.util.stream.Collectors.toMap;
+import static java.util.stream.StreamSupport.stream;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+/** {@link InMemoryStateChangelogClient} test. */
+@RunWith(Parameterized.class)
+public class StateChangelogClientTest {
+
+ private final Random random = new Random();
+ private final LogClientType clientType;
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Object[] parameters() {
+ return LogClientType.values();
+ }
+
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ public StateChangelogClientTest(LogClientType logClientType) {
+ this.clientType = logClientType;
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testNoAppendAfterClose() throws IOException {
+ StateChangelogWriter<?> writer =
+ clientType.create(this).createWriter(new OperatorID(), KeyGroupRange.of(0, 0));
+ writer.close();
+ writer.append(0, new byte[0]);
+ }
+
+ @Test
+ public void testWriteAndRead() throws Exception {
+ KeyGroupRange kgRange = KeyGroupRange.of(0, 5);
+ Map<Integer, List<byte[]>> appendsByKeyGroup = generateAppends(kgRange, 10, 20);
+
+ try (StateChangelogClient<?> client = clientType.create(this);
+ StateChangelogWriter<?> writer = client.createWriter(new OperatorID(), kgRange)) {
+ SequenceNumber from = writer.lastAppendedSqn();
+ appendsByKeyGroup.forEach(
+ (group, appends) -> appends.forEach(bytes -> writer.append(group, bytes)));
+
+ StateChangelogHandle<?> handle = writer.persist(from).get();
+
+ assertByteMapsEqual(appendsByKeyGroup, extract(handle));
+ }
+ }
+
+ private void assertByteMapsEqual(
+ Map<Integer, List<byte[]>> expected, Map<Integer, List<byte[]>> actual) {
+ assertEquals(expected.size(), actual.size());
+ for (Map.Entry<Integer, List<byte[]>> e : expected.entrySet()) {
+ List<byte[]> expectedList = e.getValue();
+ List<byte[]> actualList = actual.get(e.getKey());
+ Iterator<byte[]> ite = expectedList.iterator(), ale = actualList.iterator();
+ while (ite.hasNext() && ale.hasNext()) {
+ assertArrayEquals(ite.next(), ale.next());
+ }
+ assertFalse(ite.hasNext());
+ assertFalse(ale.hasNext());
+ }
+ }
+
+ private Map<Integer, List<byte[]>> extract(StateChangelogHandle<?> handle) throws Exception {
+ Map<Integer, List<byte[]>> changes = new HashMap<>();
+ //noinspection unchecked
+ StateChangelogHandle<Object> objHandle = (StateChangelogHandle<Object>) handle;
+ try (CloseableIterator<StateChange> it = objHandle.getChanges(clientType.getContext())) {
+ while (it.hasNext()) {
+ StateChange change = it.next();
+ changes.computeIfAbsent(change.getKeyGroup(), k -> new ArrayList<>())
+ .add(change.getChange());
+ }
+ }
+ return changes;
+ }
+
+ private Map<Integer, List<byte[]>> generateAppends(
+ KeyGroupRange kgRange, int keyLen, int appendsPerGroup) {
+ return stream(kgRange.spliterator(), false)
+ .collect(toMap(identity(), unused -> generateData(appendsPerGroup, keyLen)));
+ }
+
+ private List<byte[]> generateData(int numAppends, int keyLen) {
+ return Stream.generate(() -> randomBytes(keyLen))
+ .limit(numAppends)
+ .collect(Collectors.toList());
+ }
+
+ private byte[] randomBytes(int len) {
+ byte[] bytes = new byte[len];
+ random.nextBytes(bytes);
+ return bytes;
+ }
+
+ private enum LogClientType {
+ MEMORY {
+ @Override
+ public StateChangelogClient<?> create(StateChangelogClientTest logClientTest) {
+ return new InMemoryStateChangelogClient();
+ }
+
+ @Override
+ public StateChangeFormat getContext() {
Review comment:
This was a leftover after splitting the changes into two PRs, sorry about that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c536632abeeef101cd3a89edcdf63f1e950eff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966)
* 38b7950b0c94efc6ed0dfc70828712e3b353ca65 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1506a66da8f77069706d654fc43335e21fcf154e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349)
* 13baba5c0d15572ad8295d728fe2aedabff3342d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
}, {
"hash" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "000fa3e28def66c51ebf7b5975d8b610cccc6988",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
* 000fa3e28def66c51ebf7b5975d8b610cccc6988 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dd7135a501c0fcbe6bee4d08b561712bd3b3653 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dd7135a501c0fcbe6bee4d08b561712bd3b3653 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410)
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773636933
1. > But this is where/the name I planned to put the WrappedStateBackend.
In my opinion, both backend and changelog should reside in the same module, at least for now.
2. I fixed the `StateChangeFormat` issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r570577515
##########
File path: flink-state-backends/flink-statebackend-changelog/src/test/java/org/apache/flink/changelog/StateChangelogClientTest.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.changelog.fs.StateChangeFormat;
+import org.apache.flink.changelog.inmemory.InMemoryStateChangelogClient;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.StateChange;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static java.util.function.Function.identity;
+import static java.util.stream.Collectors.toMap;
+import static java.util.stream.StreamSupport.stream;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+/** {@link InMemoryStateChangelogClient} test. */
+@RunWith(Parameterized.class)
+public class StateChangelogClientTest {
+
+ private final Random random = new Random();
+ private final LogClientType clientType;
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Object[] parameters() {
+ return LogClientType.values();
+ }
+
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ public StateChangelogClientTest(LogClientType logClientType) {
+ this.clientType = logClientType;
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testNoAppendAfterClose() throws IOException {
+ StateChangelogWriter<?> writer =
+ clientType.create(this).createWriter(new OperatorID(), KeyGroupRange.of(0, 0));
+ writer.close();
+ writer.append(0, new byte[0]);
+ }
+
+ @Test
+ public void testWriteAndRead() throws Exception {
+ KeyGroupRange kgRange = KeyGroupRange.of(0, 5);
+ Map<Integer, List<byte[]>> appendsByKeyGroup = generateAppends(kgRange, 10, 20);
+
+ try (StateChangelogClient<?> client = clientType.create(this);
+ StateChangelogWriter<?> writer = client.createWriter(new OperatorID(), kgRange)) {
+ SequenceNumber from = writer.lastAppendedSqn();
+ appendsByKeyGroup.forEach(
+ (group, appends) -> appends.forEach(bytes -> writer.append(group, bytes)));
+
+ StateChangelogHandle<?> handle = writer.persist(from).get();
+
+ assertByteMapsEqual(appendsByKeyGroup, extract(handle));
+ }
+ }
+
+ private void assertByteMapsEqual(
+ Map<Integer, List<byte[]>> expected, Map<Integer, List<byte[]>> actual) {
+ assertEquals(expected.size(), actual.size());
+ for (Map.Entry<Integer, List<byte[]>> e : expected.entrySet()) {
+ List<byte[]> expectedList = e.getValue();
+ List<byte[]> actualList = actual.get(e.getKey());
+ Iterator<byte[]> ite = expectedList.iterator(), ale = actualList.iterator();
+ while (ite.hasNext() && ale.hasNext()) {
+ assertArrayEquals(ite.next(), ale.next());
+ }
+ assertFalse(ite.hasNext());
+ assertFalse(ale.hasNext());
+ }
+ }
+
+ private Map<Integer, List<byte[]>> extract(StateChangelogHandle<?> handle) throws Exception {
+ Map<Integer, List<byte[]>> changes = new HashMap<>();
+ //noinspection unchecked
+ StateChangelogHandle<Object> objHandle = (StateChangelogHandle<Object>) handle;
+ try (CloseableIterator<StateChange> it = objHandle.getChanges(clientType.getContext())) {
+ while (it.hasNext()) {
+ StateChange change = it.next();
+ changes.computeIfAbsent(change.getKeyGroup(), k -> new ArrayList<>())
+ .add(change.getChange());
+ }
+ }
+ return changes;
+ }
+
+ private Map<Integer, List<byte[]>> generateAppends(
+ KeyGroupRange kgRange, int keyLen, int appendsPerGroup) {
+ return stream(kgRange.spliterator(), false)
+ .collect(toMap(identity(), unused -> generateData(appendsPerGroup, keyLen)));
+ }
+
+ private List<byte[]> generateData(int numAppends, int keyLen) {
+ return Stream.generate(() -> randomBytes(keyLen))
+ .limit(numAppends)
+ .collect(Collectors.toList());
+ }
+
+ private byte[] randomBytes(int len) {
+ byte[] bytes = new byte[len];
+ random.nextBytes(bytes);
+ return bytes;
+ }
+
+ private enum LogClientType {
+ MEMORY {
+ @Override
+ public StateChangelogClient<?> create(StateChangelogClientTest logClientTest) {
+ return new InMemoryStateChangelogClient();
+ }
+
+ @Override
+ public StateChangeFormat getContext() {
Review comment:
This was a leftover after splitting the changes into two PRs, sorry about that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1c536632abeeef101cd3a89edcdf63f1e950eff Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
}, {
"hash" : "1506a66da8f77069706d654fc43335e21fcf154e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13349",
"triggerID" : "1506a66da8f77069706d654fc43335e21fcf154e",
"triggerType" : "PUSH"
}, {
"hash" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13405",
"triggerID" : "13baba5c0d15572ad8295d728fe2aedabff3342d",
"triggerType" : "PUSH"
}, {
"hash" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410",
"triggerID" : "6dd7135a501c0fcbe6bee4d08b561712bd3b3653",
"triggerType" : "PUSH"
}, {
"hash" : "f560722ab6bbf452340228a2c88e025ecc797274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f560722ab6bbf452340228a2c88e025ecc797274",
"triggerType" : "PUSH"
}, {
"hash" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "42dd6bd7e4e64b3ff0ecf5156cae0f860226d629",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dd7135a501c0fcbe6bee4d08b561712bd3b3653 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13410)
* f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
* 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573803391
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
+
+ private transient SharedStateRegistry stateRegistry;
+
+ public StateChangelogHandleStreamImpl(
+ List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets, KeyGroupRange keyGroupRange) {
+ this.handlesAndOffsets = handlesAndOffsets;
+ this.keyGroupRange = keyGroupRange;
+ }
+
+ @Override
+ public void registerSharedStates(SharedStateRegistry stateRegistry) {
+ this.stateRegistry = stateRegistry;
+ handlesAndOffsets.forEach(
+ handleAndOffset ->
+ stateRegistry.registerReference(
+ getKey(handleAndOffset.f0), handleAndOffset.f0));
+ }
+
+ @Override
+ public KeyGroupRange getKeyGroupRange() {
+ return keyGroupRange;
+ }
+
+ @Nullable
+ @Override
+ public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+ KeyGroupRange offsets = keyGroupRange.getIntersection(keyGroupRange);
+ if (offsets.getNumberOfKeyGroups() == 0) {
+ return null;
+ }
+ return new StateChangelogHandleStreamImpl(handlesAndOffsets, offsets);
+ }
+
+ @Override
+ public CloseableIterator<StateChange> getChanges(StateChangeStreamReader reader) {
+ return new CloseableIterator<StateChange>() {
+ private final Iterator<Tuple2<StreamStateHandle, Long>> handleIterator =
+ handlesAndOffsets.iterator();
+
+ private CloseableIterator<StateChange> current = CloseableIterator.empty();
+
+ @Override
+ public boolean hasNext() {
+ advance();
+ return current.hasNext();
+ }
+
+ @Override
+ public StateChange next() {
+ advance();
+ return current.next();
+ }
+
+ private void advance() {
+ while (!current.hasNext() && handleIterator.hasNext()) {
+ Tuple2<StreamStateHandle, Long> tuple2 = handleIterator.next();
+ try {
+ current = reader.read(tuple2.f0, tuple2.f1);
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ current.close();
+ }
+ };
+ }
+
+ @Override
+ public void discardState() {
+ handlesAndOffsets.forEach(
+ handleAndOffset -> stateRegistry.unregisterReference(getKey(handleAndOffset.f0)));
+ }
+
+ @Override
+ public long getStateSize() {
+ return 0;
+ }
+
+ private static SharedStateRegistryKey getKey(StreamStateHandle stateHandle) {
Review comment:
> Or multiple backends bundles state changes altogether in a file
Exactly. For a single checkpoint, all backends of a single TM will put their changes into the same file. Reference count for that file will be `number of backends` initially.
If we instead use backend.uuid as a key then refCount would be 1 and file would be deleted as soon as one of the backends dereferences it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-781481465
Test failure unrelated (FLINK-21277), merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r577688505
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/StateChangelogWriterFactoryLoader.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.core.plugin.PluginManager;
+
+import java.util.Iterator;
+
+/** A thin wrapper around {@link PluginManager} to load {@link StateChangelogWriterFactory}. */
+@Internal
+public class StateChangelogWriterFactoryLoader {
+ private final PluginManager pluginManager;
+
+ public StateChangelogWriterFactoryLoader(PluginManager pluginManager) {
+ this.pluginManager = pluginManager;
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ public Iterator<StateChangelogWriterFactory> load() {
+ return pluginManager.load(StateChangelogWriterFactory.class);
+ }
+}
Review comment:
What's the motivation/rationale behind this class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573775854
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChange.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+/** Change of state of a keyed operator. Used for generic incremental checkpoints. */
+@Internal
+public class StateChange {
Review comment:
This would suggest that there is `NonKeyedStateChange` somewhere. But currently there is none, and probably won't be. So it would be confusing to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772050707
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 000fa3e28def66c51ebf7b5975d8b610cccc6988 (Fri Feb 19 07:34:25 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
* 78a71bbedf9f0ee1e38bd8e94454b6560ab6c358 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
}, {
"hash" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231",
"triggerID" : "38b7950b0c94efc6ed0dfc70828712e3b353ca65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38b7950b0c94efc6ed0dfc70828712e3b353ca65 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13231)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r572644106
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/StateChangelogHandleStreamImpl.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.ExceptionUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/** {@link StateChangelogHandle} implementation based on {@link StreamStateHandle}. */
+@Internal
+public final class StateChangelogHandleStreamImpl
+ implements StateChangelogHandle<StateChangelogHandleStreamImpl.StateChangeStreamReader> {
+ private static final long serialVersionUID = -8070326169926626355L;
+
+ private final KeyGroupRange keyGroupRange;
+ /** NOTE: order is important as it reflects the order of changes. */
+ private final List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets;
+
+ private transient SharedStateRegistry stateRegistry;
+
+ public StateChangelogHandleStreamImpl(
+ List<Tuple2<StreamStateHandle, Long>> handlesAndOffsets, KeyGroupRange keyGroupRange) {
+ this.handlesAndOffsets = handlesAndOffsets;
+ this.keyGroupRange = keyGroupRange;
+ }
+
+ @Override
+ public void registerSharedStates(SharedStateRegistry stateRegistry) {
+ this.stateRegistry = stateRegistry;
+ handlesAndOffsets.forEach(
+ handleAndOffset ->
+ stateRegistry.registerReference(
+ getKey(handleAndOffset.f0), handleAndOffset.f0));
+ }
+
+ @Override
+ public KeyGroupRange getKeyGroupRange() {
+ return keyGroupRange;
+ }
+
+ @Nullable
+ @Override
+ public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+ KeyGroupRange offsets = keyGroupRange.getIntersection(keyGroupRange);
+ if (offsets.getNumberOfKeyGroups() == 0) {
+ return null;
+ }
+ return new StateChangelogHandleStreamImpl(handlesAndOffsets, offsets);
+ }
+
+ @Override
+ public CloseableIterator<StateChange> getChanges(StateChangeStreamReader reader) {
+ return new CloseableIterator<StateChange>() {
+ private final Iterator<Tuple2<StreamStateHandle, Long>> handleIterator =
+ handlesAndOffsets.iterator();
+
+ private CloseableIterator<StateChange> current = CloseableIterator.empty();
+
+ @Override
+ public boolean hasNext() {
+ advance();
+ return current.hasNext();
+ }
+
+ @Override
+ public StateChange next() {
+ advance();
+ return current.next();
+ }
+
+ private void advance() {
+ while (!current.hasNext() && handleIterator.hasNext()) {
+ Tuple2<StreamStateHandle, Long> tuple2 = handleIterator.next();
+ try {
+ current = reader.read(tuple2.f0, tuple2.f1);
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(e);
+ }
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ current.close();
+ }
+ };
+ }
+
+ @Override
+ public void discardState() {
+ handlesAndOffsets.forEach(
+ handleAndOffset -> stateRegistry.unregisterReference(getKey(handleAndOffset.f0)));
+ }
+
+ @Override
+ public long getStateSize() {
+ return 0;
+ }
+
+ private static SharedStateRegistryKey getKey(StreamStateHandle stateHandle) {
Review comment:
I am not quite familiar with how sharing is exactly working, so forgive me if I am asking obvious questions:
1. Sharing is per state, or per task, or per state backend or multiple state backends? I understand it is at least per state changelog, and state changelog is mapping to each state backend. So it is probably per state backend or multiple state backends.
2. I understand "at least" changes from the same state backend would be sharable, right? So what is the problem if using backend UUID? Or multiple backends bundles state changes altogether in a file?
If I can understand the above two questions, I think I can understand why state backend UUID can not be used as a key. And maybe rewrite the comment this way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r573807868
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/SequenceNumber.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A logical timestamp to draw a boundary between the materialized and non-materialized changes.
+ * Maintained by the state backend but implementations may choose to move its generation to {@link
Review comment:
We can't put it into the state changelog because it is not aware of state materialization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773054677
The StateChangeLog is put under flink-state-backends,
under <module>flink-statebackend-changelog</module>
But this is where/the name I planned to put the WrappedStateBackend. What do you think @rkhachatryan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773810436
> 1. > But this is where/the name I planned to put the WrappedStateBackend.
>
> In my opinion, both backend and changelog should reside in the same module, at least for now.
I have no objection to putting them together.
>
> 1. I fixed the `StateChangeFormat` issue
Thanks for fixing this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
* 78a71bbedf9f0ee1e38bd8e94454b6560ab6c358 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78a71bbedf9f0ee1e38bd8e94454b6560ab6c358 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964)
* c1c536632abeeef101cd3a89edcdf63f1e950eff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773810436
> 1. > But this is where/the name I planned to put the WrappedStateBackend.
>
> In my opinion, both backend and changelog should reside in the same module, at least for now.
I have no objection to putting them together.
>
> 1. I fixed the `StateChangeFormat` issue
Thanks for fixing this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772050707
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 9687e242de6c8c55221ed01b679844560ded1ab4 (Tue Feb 02 22:25:27 UTC 2021)
**Warnings:**
* **2 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r576239008
##########
File path: flink-state-backends/pom.xml
##########
@@ -39,5 +39,6 @@ under the License.
<modules>
<module>flink-statebackend-rocksdb</module>
<module>flink-statebackend-heap-spillable</module>
+ <module>flink-statebackend-changelog</module>
Review comment:
I'd say there **might** be use case where we could use. They are even close to design.
And it and **might** have different implementations.
I have concerns regarding both, but I don't mind putting the API part into the `flink-runtime`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773054677
1. The StateChangeLog is put under flink-state-backends,
under <module>flink-statebackend-changelog</module>
But this is where/the name I planned to put the WrappedStateBackend. What do you think @rkhachatryan
2. The code is missing "StateChangeFormat" right now. Would you make a workaround so that I can try it a bit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
* 78a71bbedf9f0ee1e38bd8e94454b6560ab6c358 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964)
* c1c536632abeeef101cd3a89edcdf63f1e950eff UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817",
"triggerID" : "9687e242de6c8c55221ed01b679844560ded1ab4",
"triggerType" : "PUSH"
}, {
"hash" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964",
"triggerID" : "78a71bbedf9f0ee1e38bd8e94454b6560ab6c358",
"triggerType" : "PUSH"
}, {
"hash" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966",
"triggerID" : "c1c536632abeeef101cd3a89edcdf63f1e950eff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9687e242de6c8c55221ed01b679844560ded1ab4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12817)
* 78a71bbedf9f0ee1e38bd8e94454b6560ab6c358 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12964)
* c1c536632abeeef101cd3a89edcdf63f1e950eff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12966)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14838: [FLINK-19503][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14838:
URL: https://github.com/apache/flink/pull/14838#discussion_r576234098
##########
File path: flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/changelog/StateChangelogWriter.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateChangelogHandle;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+
+/** Allows to write data to the log. Scoped to a single writer (e.g. state backend). */
+@Internal
+public interface StateChangelogWriter<Handle extends StateChangelogHandle<?>>
+ extends AutoCloseable {
+
+ /**
+ * Get {@link SequenceNumber} of the last element added by {@link #append(int, byte[]) append}.
+ */
+ SequenceNumber lastAppendedSqn();
+
+ /** Appends the provided data to this log. No persistency guarantees. */
+ void append(int keyGroup, byte[] value);
+
+ /**
+ * Durably persist previously {@link #append(int, byte[]) appended} data. After this call, one
+ * of {@link #confirm(SequenceNumber, SequenceNumber) confirm}, {@link #reset(SequenceNumber,
+ * SequenceNumber) reset}, or {@link #truncate(SequenceNumber) truncate} eventually must be
+ * called for the corresponding change set. with reset/truncate/confirm methods?
+ *
+ * @param from inclusive
+ */
+ CompletableFuture<Handle> persist(SequenceNumber from) throws IOException;
Review comment:
It's from the gitven `from` until the last change already added.
I'll update the javadoc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] curcur edited a comment on pull request #14838: [WIP][state] Add StateChangelog API
Posted by GitBox <gi...@apache.org>.
curcur edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-773054677
The StateChangeLog is put under flink-state-backends,
under <module>flink-statebackend-changelog</module>
But this is where/the name I planned to put the WrappedStateBackend. What do you think
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org