You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/19 11:55:00 UTC
[GitHub] [flink] anonymouscodeholic opened a new pull request #13688: Arena block size sanity
anonymouscodeholic opened a new pull request #13688:
URL: https://github.com/apache/flink/pull/13688
This PR replaces the PR https://github.com/apache/flink/pull/13393. I'm the author of that PR, but have lost access to the GitHub account owning the branch for the PR.
Logs a WARN if the case arena block size is too high.
## What is the purpose of the change
This change adds a WARN message in the case RocksDB will very likely (100% sure?) perform very poorly because of the way the managed memory is configured.
There's an associated JIRA ticket [FLINK-19238](https://issues.apache.org/jira/browse/FLINK-19238).
## Brief change log
`RocksDBMemoryControllerUtils` has new methods to calculate different values as RocksDB calculates those.
The actual logging is in `RocksDBOperationUtils.sanityCheckArenaBlockSize`. The arena block size is not known before this. Most of the code changes is caused by the need to pass `writeBufferManagerCapacity` to the check.
## Verifying this change
If you run any properly performing Flink application (on a RocksDB state backend), there won't be any change caused by this PR.
If you setup a simple Flink app and run it with `StreamExecutionEnvironment.createLocalEnvironment` and high parallelism (let's say 16), you should see the new WARN messages
This change added tests and can be verified as follows:
* `RocksDBOperationsUtilsTest.testSanityCheckArenaBlockSize`
## Does this pull request potentially affect one of the following parts:
* Dependencies (does it add or upgrade a dependency): / no
* The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
* The serializers: no
* The runtime per-record code paths (performance sensitive): no
* Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
* The S3 file system connector: no
## Documentation
* Does this pull request introduce a new feature? yes/no. The only user visible change is a new WARN message. Not really a feature...
* If yes, how is the feature documented? JavaDocs
Logs a WARN if the case arena block size is too high.
## What is the purpose of the change
This change adds a WARN message in the case RocksDB will very likely (100% sure?) perform very poorly because of the way the managed memory is configured.
There's an associated JIRA ticket [FLINK-19238](https://issues.apache.org/jira/browse/FLINK-19238).
## Brief change log
`RocksDBMemoryControllerUtils` has new methods to calculate different values as RocksDB calculates those.
The actual logging is in `RocksDBOperationUtils.sanityCheckArenaBlockSize`. The arena block size is not known before this. Most of the code changes is caused by the need to pass `writeBufferManagerCapacity` to the check.
## Verifying this change
If you run any properly performing Flink application (on a RocksDB state backend), there won't be any change caused by this PR.
If you setup a simple Flink app and run it with `StreamExecutionEnvironment.createLocalEnvironment` and high parallelism (let's say 16), you should see the new WARN messages
This change added tests and can be verified as follows:
`RocksDBOperationsUtilsTest.testSanityCheckArenaBlockSize`
## Does this pull request potentially affect one of the following parts:
* Dependencies (does it add or upgrade a dependency): / no
* The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
* The serializers: no
* The runtime per-record code paths (performance sensitive): no
* Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
* The S3 file system connector: no
## Documentation
* Does this pull request introduce a new feature? yes/no. The only user visible change is a new WARN message. Not really a feature...
* If yes, how is the feature documented? JavaDocs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 5fed292da3f551884829a2ccc562ec51d0ecae2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712817061
> Merged master to this branch and still having strange test failures....
`mvn verify`
```
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.169 s <<< FAILURE! - in org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheRecoveryITCase
[ERROR] testRecoveryRegisterAndDownload(org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheRecoveryITCase) Time elapsed: 0.169 s <<< ERROR!
java.lang.NoSuchMethodError: 'java.lang.ClassLoader org.apache.flink.runtime.execution.librarycache.LibraryCacheManager$ClassLoaderLease.getOrResolveClassLoader(java.util.Collection, java.util.Collection)'
at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheRecoveryITCase.testRecoveryRegisterAndDownload(BlobLibraryCacheRecoveryITCase.java:113)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
```
Trying with `mvn clean verify`.... Though mvn should be clever enough to compile all that has changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712666788
Thanks for continue working on this one @anonymouscodeholic . Checking the Azure test failure it's caused by FLINK-19619 and not related to changes here. I will review the latest commits asap.
Just a heads up, that the feature freeze of 1.12.0 is approaching and let's try to get this one in before that. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-723457021
@Myasuka could you take a look at c700e3a and let me know whether it addresses your comments well? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 83600840ab86ba77f2045f3eed61e2b7c86effe2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511598641
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBOperationsUtilsTest.java
##########
@@ -78,6 +81,60 @@ public void testPathExceptionOnWindows() throws Exception {
}
}
+ @Test
+ public void testSanityCheckArenaBlockSize() {
+ long testWriteBufferSize = 56 * 1024 * 1024L;
+ long testDefaultArenaSize = testWriteBufferSize / 8;
+ long testValidArenaSize = testWriteBufferSize / 7;
+ long testInvalidArenaSize = testWriteBufferSize / 7 - 8L;
+ List<TestData> tests = Arrays.asList(
+ new TestData(testWriteBufferSize, 0, testInvalidArenaSize, false),
+ new TestData(testWriteBufferSize, testDefaultArenaSize, testInvalidArenaSize, false),
+ new TestData(testWriteBufferSize, 0, testValidArenaSize, true),
+ new TestData(testWriteBufferSize, testDefaultArenaSize, testValidArenaSize, true)
+ );
+
+ for (TestData test : tests) {
+ long writeBufferSize = test.getWriteBufferSize();
+ long arenaBlockSizeConfigured = test.getArenaBlockSizeConfigured();
+ long writeBufferManagerCapacity = test.getWriteBufferManagerCapacity();
+ boolean expected = test.getExpectedResult();
+
+ boolean sanityCheckResult = RocksDBOperationUtils.sanityCheckArenaBlockSize(writeBufferSize, arenaBlockSizeConfigured, writeBufferManagerCapacity);
+ assertThat(sanityCheckResult, is(expected));
+ }
+ }
+
+ private static class TestData {
+ private final long writeBufferSize;
+ private final long arenaBlockSizeConfigured;
+ private final long writeBufferManagerCapacity;
+ private final boolean expectedResult;
+
+ public TestData(long writeBufferSize, long arenaBlockSizeConfigured, long writeBufferManagerCapacity, boolean expectedResult) {
+ this.writeBufferSize = writeBufferSize;
+ this.arenaBlockSizeConfigured = arenaBlockSizeConfigured;
+ this.writeBufferManagerCapacity = writeBufferManagerCapacity;
+ this.expectedResult = expectedResult;
+ }
+
+ public long getWriteBufferSize() {
+ return writeBufferSize;
+ }
+
+ public long getArenaBlockSizeConfigured() {
+ return arenaBlockSizeConfigured;
+ }
+
+ public long getWriteBufferManagerCapacity() {
+ return writeBufferManagerCapacity;
+ }
+
+ public boolean getExpectedResult() {
+ return expectedResult;
+ }
+ }
Review comment:
I changed the test `testSanityCheckArenaBlockSize` to the suggested version and added one more `assertThat`. Also, minor tweaks in the error messages in asserts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-714953778
> From the CI report on Azure all tests passed with [5fed292](https://github.com/apache/flink/commit/5fed292da3f551884829a2ccc562ec51d0ecae2a), so I guess there might be some environment issues during your local test which is irrelative to the changes of this PR.
Yes, I have all kinds of trouble running tests locally. Those issues seem to be unrelated to this PR.
Is there still something missing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-723455636
Thanks for the review @Myasuka , I will resolve these minor issues when merging this PR, if no objections.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511778174
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,33 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long writeBufferSize = 64 * 1024 * 1024;
+ long expectArenaBlockSize = writeBufferSize / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+ }
+
+ @Test
+ public void testCalculateRocksDBMutableLimit() {
+ long bufferSize = 64 * 1024 * 1024;
+ long limit = bufferSize * 7 / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBMutableLimit(bufferSize), is(limit));
+ }
+
+ @Test
+ public void testValidateArenaBlockSize() {
+ long arenaBlockSize = 8 * 1024 * 1024;
+ assertFalse(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 0.5)));
+ assertTrue(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 1.5)));
}
Review comment:
ok, I'm fine to keep these tests. And +1 for adding the alignment logic, please check my inline comments. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
}, {
"hash" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 84825efe429d1074ad6389422bf258ec4e50d040 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245)
* 57a875919abcac55e0b779a7b3aa2b016e716a05 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
Myasuka commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r518198122
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBResourceContainer.java
##########
@@ -103,6 +103,19 @@ public DBOptions getDbOptions() {
return opt;
}
+ /**
+ * Gets write buffer manager capacity.
+ *
+ * @return the capacity, or null if write buffer managed is not enabled (not using managed memory).
Review comment:
This doc is not so accurate, we could not use managed memory but still enable write buffer manager, e.g. configuring `state.backend.rocksdb.memory.fixed-per-slot` and set `state.backend.rocksdb.memory.managed` as false.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBOperationsUtilsTest.java
##########
@@ -78,6 +79,25 @@ public void testPathExceptionOnWindows() throws Exception {
}
}
+ @Test
+ public void testSanityCheckArenaBlockSize() {
+ long testWriteBufferSize = 56 * 1024 * 1024L;
+ long testDefaultArenaSize = RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(testWriteBufferSize);
+ long testWriteBufferCapacityBoundary = testDefaultArenaSize * 8 / 7;
+ assertThat("The sanity check result is incorrect with default arena block size",
+ RocksDBOperationUtils.sanityCheckArenaBlockSize(testWriteBufferSize, 0, testWriteBufferCapacityBoundary),
+ is(true));
+ assertThat("The sanity check result is incorrect with default arena block size given as argument",
Review comment:
The same as above.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
##########
@@ -557,8 +557,7 @@ public CheckpointStorage createCheckpointStorage(JobID jobId) throws IOException
metricGroup,
stateHandles,
keyGroupCompressionDecorator,
- cancelStreamRegistry
- )
+ cancelStreamRegistry)
Review comment:
This change seems not so necessary in this PR.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBOperationsUtilsTest.java
##########
@@ -78,6 +79,25 @@ public void testPathExceptionOnWindows() throws Exception {
}
}
+ @Test
+ public void testSanityCheckArenaBlockSize() {
+ long testWriteBufferSize = 56 * 1024 * 1024L;
+ long testDefaultArenaSize = RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(testWriteBufferSize);
+ long testWriteBufferCapacityBoundary = testDefaultArenaSize * 8 / 7;
+ assertThat("The sanity check result is incorrect with default arena block size",
Review comment:
I think the message should be `should pass` instead of `is incorrect`.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtils.java
##########
@@ -95,4 +95,51 @@ static Cache createCache(long cacheCapacity, double highPriorityPoolRatio) {
static WriteBufferManager createWriteBufferManager(long writeBufferManagerCapacity, Cache cache) {
return new WriteBufferManager(writeBufferManagerCapacity, cache);
}
+
+ /**
+ * Calculate the default arena block size as RocksDB calculates it in
+ * <a href="https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196-L201">
Review comment:
We should still use current `github.com/dataArtisans/frocksdb` link here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
}, {
"hash" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8367",
"triggerID" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 57a875919abcac55e0b779a7b3aa2b016e716a05 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8367)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 83600840ab86ba77f2045f3eed61e2b7c86effe2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222)
* 84825efe429d1074ad6389422bf258ec4e50d040 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 closed pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 closed pull request #13688:
URL: https://github.com/apache/flink/pull/13688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
Myasuka commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-722343531
Sorry for late reply, I'll take a look tonight @anonymouscodeholic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511004271
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOperationUtils.java
##########
@@ -156,9 +160,49 @@ public static ColumnFamilyDescriptor createColumnFamilyDescriptor(
Preconditions.checkState(!Arrays.equals(RocksDB.DEFAULT_COLUMN_FAMILY, nameBytes),
"The chosen state name 'default' collides with the name of the default column family!");
+ if (writeBufferManagerCapacity != null) {
+ // It'd be great to perform the check earlier, e.g. when creating write buffer manager.
+ // Unfortunately the check needs write buffer size that was just calculated.
+ sanityCheckArenaBlockSize(options.writeBufferSize(), options.arenaBlockSize(), writeBufferManagerCapacity);
+ }
+
return new ColumnFamilyDescriptor(nameBytes, options);
}
+ /**
+ * Logs a warning ff the arena block size is too high causing RocksDB to flush constantly.
+ * Essentially, the condition here
+ * <a href="https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/include/rocksdb/write_buffer_manager.h#L47"/>
+ * will always be true.
Review comment:
```suggestion
* Essentially, the condition
* <a href="https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/include/rocksdb/write_buffer_manager.h#L47">
* here</a> will always be true.
```
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,33 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long writeBufferSize = 64 * 1024 * 1024;
+ long expectArenaBlockSize = writeBufferSize / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+ }
+
+ @Test
+ public void testCalculateRocksDBMutableLimit() {
+ long bufferSize = 64 * 1024 * 1024;
+ long limit = bufferSize * 7 / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBMutableLimit(bufferSize), is(limit));
+ }
+
+ @Test
+ public void testValidateArenaBlockSize() {
+ long arenaBlockSize = 8 * 1024 * 1024;
+ assertFalse(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 0.5)));
+ assertTrue(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 1.5)));
}
Review comment:
I'm hesitating on adding these tests since they're testing against the implementation instead of any contract. Once the calculation formula changes, these tests will fail and need to be adjusted accordingly.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBOperationsUtilsTest.java
##########
@@ -78,6 +81,60 @@ public void testPathExceptionOnWindows() throws Exception {
}
}
+ @Test
+ public void testSanityCheckArenaBlockSize() {
+ long testWriteBufferSize = 56 * 1024 * 1024L;
+ long testDefaultArenaSize = testWriteBufferSize / 8;
+ long testValidArenaSize = testWriteBufferSize / 7;
+ long testInvalidArenaSize = testWriteBufferSize / 7 - 8L;
+ List<TestData> tests = Arrays.asList(
+ new TestData(testWriteBufferSize, 0, testInvalidArenaSize, false),
+ new TestData(testWriteBufferSize, testDefaultArenaSize, testInvalidArenaSize, false),
+ new TestData(testWriteBufferSize, 0, testValidArenaSize, true),
+ new TestData(testWriteBufferSize, testDefaultArenaSize, testValidArenaSize, true)
+ );
+
+ for (TestData test : tests) {
+ long writeBufferSize = test.getWriteBufferSize();
+ long arenaBlockSizeConfigured = test.getArenaBlockSizeConfigured();
+ long writeBufferManagerCapacity = test.getWriteBufferManagerCapacity();
+ boolean expected = test.getExpectedResult();
+
+ boolean sanityCheckResult = RocksDBOperationUtils.sanityCheckArenaBlockSize(writeBufferSize, arenaBlockSizeConfigured, writeBufferManagerCapacity);
+ assertThat(sanityCheckResult, is(expected));
+ }
+ }
+
+ private static class TestData {
+ private final long writeBufferSize;
+ private final long arenaBlockSizeConfigured;
+ private final long writeBufferManagerCapacity;
+ private final boolean expectedResult;
+
+ public TestData(long writeBufferSize, long arenaBlockSizeConfigured, long writeBufferManagerCapacity, boolean expectedResult) {
+ this.writeBufferSize = writeBufferSize;
+ this.arenaBlockSizeConfigured = arenaBlockSizeConfigured;
+ this.writeBufferManagerCapacity = writeBufferManagerCapacity;
+ this.expectedResult = expectedResult;
+ }
+
+ public long getWriteBufferSize() {
+ return writeBufferSize;
+ }
+
+ public long getArenaBlockSizeConfigured() {
+ return arenaBlockSizeConfigured;
+ }
+
+ public long getWriteBufferManagerCapacity() {
+ return writeBufferManagerCapacity;
+ }
+
+ public boolean getExpectedResult() {
+ return expectedResult;
+ }
+ }
Review comment:
```suggestion
public void testSanityCheckArenaBlockSize() {
long testWriteBufferSize = 56 * 1024 * 1024L;
long testDefaultArenaSize = RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(testWriteBufferSize);
long testWriteBufferCapacityBoundary = testDefaultArenaSize * 8 / 7;
assertThat("The sanity check result is incorrect with default arena block size",
RocksDBOperationUtils.sanityCheckArenaBlockSize(testWriteBufferSize, 0, testWriteBufferCapacityBoundary),
is(true));
assertThat("The sanity check should pass when the configured arena block size is small enough.",
RocksDBOperationUtils.sanityCheckArenaBlockSize(testWriteBufferSize, testDefaultArenaSize - 1, testWriteBufferCapacityBoundary),
is(true));
assertThat("The sanity check should fail when the configured arena block size is too big.",
RocksDBOperationUtils.sanityCheckArenaBlockSize(testWriteBufferSize, testDefaultArenaSize + 1, testWriteBufferCapacityBoundary),
is(false));
}
```
Sorry but I'm still not satisfied with this test and suggest to further simplify it and adding some hints if any of the test fails.
And although this test is also implementation bounded, it checks/guards the result of multiple calculations, so I think we should keep it.
We will also need to remove useless imports if the suggestion is accepted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511387433
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,33 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long writeBufferSize = 64 * 1024 * 1024;
+ long expectArenaBlockSize = writeBufferSize / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+ }
+
+ @Test
+ public void testCalculateRocksDBMutableLimit() {
+ long bufferSize = 64 * 1024 * 1024;
+ long limit = bufferSize * 7 / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBMutableLimit(bufferSize), is(limit));
+ }
+
+ @Test
+ public void testValidateArenaBlockSize() {
+ long arenaBlockSize = 8 * 1024 * 1024;
+ assertFalse(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 0.5)));
+ assertTrue(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 1.5)));
}
Review comment:
The contract for e.g. `calculateRocksDBDefaultArenaBlockSize` would be what the JavaDoc says? What a test for `calculateRocksDBDefaultArenaBlockSize` should do IMHO is to test that the method follows its documentation. If the formula changes, the JavaDoc should change and thus the test method would change too. Does this make sense?
BTW, now when I look at this closer, I think there's a minor issue. In https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L200 there's the aligning that is missing from the method. Also the test should check a case when alignment is needed and where it's not needed. I'll fix this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511598512
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,33 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long writeBufferSize = 64 * 1024 * 1024;
+ long expectArenaBlockSize = writeBufferSize / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+ }
+
+ @Test
+ public void testCalculateRocksDBMutableLimit() {
+ long bufferSize = 64 * 1024 * 1024;
+ long limit = bufferSize * 7 / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBMutableLimit(bufferSize), is(limit));
+ }
+
+ @Test
+ public void testValidateArenaBlockSize() {
+ long arenaBlockSize = 8 * 1024 * 1024;
+ assertFalse(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 0.5)));
+ assertTrue(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 1.5)));
}
Review comment:
I added testing the alignment to `testCalculateRocksDBDefaultArenaBlockSize`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867)
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 5fed292da3f551884829a2ccc562ec51d0ecae2a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712802795
Merged master to this branch and still having strange test failures....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 5fed292da3f551884829a2ccc562ec51d0ecae2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941)
* 83600840ab86ba77f2045f3eed61e2b7c86effe2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 83600840ab86ba77f2045f3eed61e2b7c86effe2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222)
* 84825efe429d1074ad6389422bf258ec4e50d040 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712802434
> Some quick comments:
>
> It seems the [comments](https://github.com/apache/flink/pull/13393#discussion_r502241882) on the javadoc of `RocksDBMemoryControllerUtils` (fix on the link style) were neglected, could you double check it @anonymouscodeholic ?
>
> And maybe also the [comment](https://github.com/apache/flink/pull/13393#discussion_r502243922) on `RocksDBOperationsUtilsTest`, which I think could make the test case more readable.
>
> Thanks.
Hey, things got complicated, because I couldn't approve the changes in the original PR and had to hand-pick them in this PR. I think I just missed those few comments.
Just pushed the latest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867)
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712106574
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 728efeb026459830372935d684b1e189869f26cb (Mon Oct 19 11:57:12 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511759563
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtils.java
##########
@@ -95,4 +95,50 @@ static Cache createCache(long cacheCapacity, double highPriorityPoolRatio) {
static WriteBufferManager createWriteBufferManager(long writeBufferManagerCapacity, Cache cache) {
return new WriteBufferManager(writeBufferManagerCapacity, cache);
}
+
+ /**
+ * Calculate the default arena block size as RocksDB calculates it in
+ * <a href="https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196">
+ * here</a>.
+ *
+ * @return the default arena block size
+ * @param writeBufferSize the write buffer size (bytes)
+ */
+ static long calculateRocksDBDefaultArenaBlockSize(long writeBufferSize) {
+ long arenaBlockSize = writeBufferSize / 8;
+
+ long align = 4 * 1024;
+ return ((arenaBlockSize + align - 1) / align) * align;
Review comment:
```suggestion
// Align up to 4k
final long align = 4 * 1024;
return ((arenaBlockSize + align - 1) / align) * align;
```
Marking the primitive variable as `final` would make it a constant variable and help the compiler to do optimization, please check [here](https://stackoverflow.com/questions/62603729/java-compiler-optimizations-with-final-local-variables) for more details. We could also make it as a class-level static final field, but keeping it here directly maps to the RocksDB code.
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBOperationsUtilsTest.java
##########
@@ -78,6 +79,24 @@ public void testPathExceptionOnWindows() throws Exception {
}
}
+ public void testSanityCheckArenaBlockSize() {
Review comment:
```suggestion
@Test
public void testSanityCheckArenaBlockSize() {
```
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,45 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long align = 4 * 1024;
+
+ long writeBufferSize = 64 * 1024 * 1024;
+
+ long expectArenaBlockSize = writeBufferSize / 8;
+ long expectArenaBlockSize2 = expectArenaBlockSize + align;
+ long expectArenaBlockSize3 = expectArenaBlockSize + 2 * align;
+
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+
+ // Alignment tests
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize + 7), is(expectArenaBlockSize));
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize + 8), is(expectArenaBlockSize2));
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize + 8 * align + 7), is(expectArenaBlockSize2));
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize + 8 * align + 8), is(expectArenaBlockSize3));
Review comment:
```suggestion
final long align = 4 * 1024;
final long writeBufferSize = 64 * 1024 * 1024;
final long expectArenaBlockSize = writeBufferSize / 8;
// Normal case test
assertThat("Arena block size calculation error for normal case",
RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
// Alignment tests
assertThat("Arena block size calculation error for alignment case",
RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize - 1), is(expectArenaBlockSize));
assertThat("Arena block size calculation error for alignment case2",
RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize + 8), is(expectArenaBlockSize + align));
```
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtils.java
##########
@@ -95,4 +95,50 @@ static Cache createCache(long cacheCapacity, double highPriorityPoolRatio) {
static WriteBufferManager createWriteBufferManager(long writeBufferManagerCapacity, Cache cache) {
return new WriteBufferManager(writeBufferManagerCapacity, cache);
}
+
+ /**
+ * Calculate the default arena block size as RocksDB calculates it in
+ * <a href="https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196">
Review comment:
```suggestion
* <a href="https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/column_family.cc#L196-L201">
```
It's better to also include the source of alignment logic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-722297475
@Myasuka Can you have a look at this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13688: Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712672608
Some quick comments:
It seems the [comments](https://github.com/apache/flink/pull/13393#discussion_r502241882) on the javadoc of `RocksDBMemoryControllerUtils` (fix on the link style) were neglected, could you double check it @anonymouscodeholic ?
And maybe also the [comment](https://github.com/apache/flink/pull/13393#discussion_r502243922) on `RocksDBOperationsUtilsTest`, which I think could make the test case more readable.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
carp84 commented on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-713412374
> Hey, things got complicated, because I couldn't approve the changes in the original PR and had to hand-pick them in this PR. I think I just missed those few comments.
>
> Just pushed the latest.
No problem, thanks for the follow up actions.
> Merged master to this branch and still having strange test failures....
From the CI report on Azure all tests passed with 5fed292, so I guess there might be some environment issues during your local test which is irrelative to the changes of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] anonymouscodeholic commented on a change in pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
anonymouscodeholic commented on a change in pull request #13688:
URL: https://github.com/apache/flink/pull/13688#discussion_r511598827
##########
File path: flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBMemoryControllerUtilsTest.java
##########
@@ -91,10 +93,33 @@ public void testCreateSharedResourcesWithExpectedCapacity() {
long totalMemorySize = 2048L;
double writeBufferRatio = 0.5;
double highPriPoolRatio = 0.1;
- RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
+ RocksDBSharedResources rocksDBSharedResources = RocksDBMemoryControllerUtils.allocateRocksDBSharedResources(totalMemorySize, writeBufferRatio, highPriPoolRatio);
long expectedCacheCapacity = RocksDBMemoryControllerUtils.calculateActualCacheCapacity(totalMemorySize, writeBufferRatio);
long expectedWbmCapacity = RocksDBMemoryControllerUtils.calculateWriteBufferManagerCapacity(totalMemorySize, writeBufferRatio);
+
assertThat(actualCacheCapacity.get(), is(expectedCacheCapacity));
assertThat(actualWbmCapacity.get(), is(expectedWbmCapacity));
+ assertThat(rocksDBSharedResources.getWriteBufferManagerCapacity(), is(expectedWbmCapacity));
+ }
+
+ @Test
+ public void testCalculateRocksDBDefaultArenaBlockSize() {
+ long writeBufferSize = 64 * 1024 * 1024;
+ long expectArenaBlockSize = writeBufferSize / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBDefaultArenaBlockSize(writeBufferSize), is(expectArenaBlockSize));
+ }
+
+ @Test
+ public void testCalculateRocksDBMutableLimit() {
+ long bufferSize = 64 * 1024 * 1024;
+ long limit = bufferSize * 7 / 8;
+ assertThat(RocksDBMemoryControllerUtils.calculateRocksDBMutableLimit(bufferSize), is(limit));
+ }
+
+ @Test
+ public void testValidateArenaBlockSize() {
+ long arenaBlockSize = 8 * 1024 * 1024;
+ assertFalse(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 0.5)));
+ assertTrue(RocksDBMemoryControllerUtils.validateArenaBlockSize(arenaBlockSize, (long) (arenaBlockSize * 1.5)));
}
Review comment:
What do you think now @carp84 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 73be3eb3e74fb5424ac1217f01d1900616b98bb1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867)
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 5fed292da3f551884829a2ccc562ec51d0ecae2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 5fed292da3f551884829a2ccc562ec51d0ecae2a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941)
* 83600840ab86ba77f2045f3eed61e2b7c86effe2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
}, {
"hash" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8367",
"triggerID" : "57a875919abcac55e0b779a7b3aa2b016e716a05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 84825efe429d1074ad6389422bf258ec4e50d040 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245)
* 57a875919abcac55e0b779a7b3aa2b016e716a05 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8367)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13688: [FLINK-19238] Arena block size sanity
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13688:
URL: https://github.com/apache/flink/pull/13688#issuecomment-712121175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "728efeb026459830372935d684b1e189869f26cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "728efeb026459830372935d684b1e189869f26cb",
"triggerType" : "PUSH"
}, {
"hash" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7867",
"triggerID" : "73be3eb3e74fb5424ac1217f01d1900616b98bb1",
"triggerType" : "PUSH"
}, {
"hash" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cef297fd3cb1534ca3c33ecd66dc6e73c57380d",
"triggerType" : "PUSH"
}, {
"hash" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7941",
"triggerID" : "5fed292da3f551884829a2ccc562ec51d0ecae2a",
"triggerType" : "PUSH"
}, {
"hash" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8222",
"triggerID" : "83600840ab86ba77f2045f3eed61e2b7c86effe2",
"triggerType" : "PUSH"
}, {
"hash" : "84825efe429d1074ad6389422bf258ec4e50d040",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245",
"triggerID" : "84825efe429d1074ad6389422bf258ec4e50d040",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 728efeb026459830372935d684b1e189869f26cb UNKNOWN
* 7cef297fd3cb1534ca3c33ecd66dc6e73c57380d UNKNOWN
* 84825efe429d1074ad6389422bf258ec4e50d040 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8245)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org