You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2017/01/23 15:27:21 UTC
[GitHub] flink pull request #3194: [FLINK-5615][query] execute the QueryableStateITCa...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/3194
[FLINK-5615][query] execute the QueryableStateITCase for all three state back-ends
This extends the `QueryableStateITCase` so that it is able to run with any selected state backend. Some optimisations reduce the total runtime of the test cases so that we are able to run the tests with all three currently available backends, i.e. `MemoryStateBackend`, `FsStateBackend`, and `RocksDBStateBackend`, with little extra costs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-5615
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3194.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3194
----
commit 9c1a247ada0015ff1b97c6017e8c1de874ba6d17
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-17T13:26:16Z
[FLINK-5613][query] querying a non-existing key is inconsistent among state backends
Querying for a non-existing key for a state that has a default value set
currently results in an UnknownKeyOrNamespace exception when the
MemoryStateBackend or FsStateBackend is used but results in the default value
if RocksDBStateBackend is set.
This removes the special handling from the RocksDBStateBackend and makes it
consistent with the other two back-ends, i.e. returning null which results
in the mentioned UnknownKeyOrNamespace exception.
commit cccbc5f30e09f5ccbc61a75fc9519dacf91c5522
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-23T14:17:46Z
[FLINK-5615][query] improve testQueryableStateWithTaskManagerFailure test duration
This is based on the following points:
* slow down QueryableStateITCase#TestKeyRangeSource for the rest of the
program to make more progress (we do not need a full-speed source here!)
* reduce the checkpointing interval
* reduce the amount of progress before starting our own evaluation
* reduce the number of checkpoints to wait for before killing a TM
* reduce the thread waiting time when asking how many checkpoints exist
Note that by slowing down QueryableStateITCase#TestKeyRangeSource, the other
tests should only be affected positively, too, since they also did not really
need a full-speed source and thus have more CPU cycles for their own tasks.
This, among with
commit 3dd506dd65f6f9f9e8879e6bf6df0261435d5317
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-23T14:47:40Z
[FLINK-5615][query] speed up some more tests in QueryableStateITCase
This is based on reducing the number of keys the source generates. We do not
really need 1024 different keys for the tests - go with 256 now.
commit ee6b78eb3a91f39386148e3ef1c55a0f3824843f
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-23T14:57:09Z
[FLINK-5615][query] execute the QueryableStateITCase for all state back-ends
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3194: [FLINK-5615][query] execute the QueryableStateITCase for ...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/3194
Thanks for addressing the comments, merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3194: [FLINK-5615][query] execute the QueryableStateITCa...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/3194#discussion_r97500347
--- Diff: flink-tests/src/test/java/org/apache/flink/test/query/QueryableStateITCaseFsBackend.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.query;
+
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+/**
+ * Several integration tests for queryable state using the {@link FsStateBackend}.
+ */
+public class QueryableStateITCaseFsBackend extends AbstractQueryableStateITCase {
+
--- End diff --
empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3194: [FLINK-5615][query] execute the QueryableStateITCa...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/3194#discussion_r97500318
--- Diff: flink-tests/src/test/java/org/apache/flink/test/query/QueryableStateITCaseRocksDBBackend.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.query;
+
+import org.apache.flink.contrib.streaming.state.RocksDBStateBackend;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+/**
+ * Several integration tests for queryable state using the {@link RocksDBStateBackend}.
+ */
+public class QueryableStateITCaseRocksDBBackend extends AbstractQueryableStateITCase {
+
--- End diff --
empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3194: [FLINK-5615][query] execute the QueryableStateITCa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3194
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---