You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by xt...@apache.org on 2023/08/02 07:43:53 UTC
[flink] branch master updated (b18bde9bb44 -> 8db8119138b)
This is an automated email from the ASF dual-hosted git repository.
xtsong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from b18bde9bb44 [FLINK-32675][doc] Add document for the tiered storage of hybrid shuffle
new d012b8c981b [FLINK-32559] Do not generate config option docs for Deprecated / ExcludeFromDocumentation classes.
new 8db8119138b [FLINK-32559] Deprecate Queryable State
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../docs/concepts/stateful-stream-processing.md | 2 -
.../docs/connectors/datastream/overview.md | 5 -
docs/content.zh/docs/deployment/config.md | 10 -
.../resource-providers/standalone/docker.md | 4 +-
.../resource-providers/standalone/kubernetes.md | 34 ---
.../docs/deployment/security/security-ssl.md | 6 -
.../datastream/fault-tolerance/queryable_state.md | 268 ------------------
docs/content.zh/docs/learn-flink/etl.md | 1 -
.../docs/concepts/stateful-stream-processing.md | 2 -
.../content/docs/connectors/datastream/overview.md | 9 -
docs/content/docs/deployment/config.md | 10 -
.../resource-providers/standalone/docker.md | 4 +-
.../resource-providers/standalone/kubernetes.md | 32 ---
.../docs/deployment/security/security-ssl.md | 6 -
.../datastream/fault-tolerance/queryable_state.md | 314 ---------------------
docs/content/docs/learn-flink/etl.md | 1 -
.../generated/queryable_state_configuration.html | 60 ----
.../flink/api/common/state/StateDescriptor.java | 9 +
.../flink/configuration/QueryableStateOptions.java | 8 +-
.../docs/util/ConfigurationOptionLocator.java | 10 +-
.../client/QueryableStateClient.java | 4 +
.../streaming/api/datastream/KeyedStream.java | 9 +
.../api/datastream/QueryableStateStream.java | 3 +
23 files changed, 45 insertions(+), 766 deletions(-)
delete mode 100644 docs/content.zh/docs/dev/datastream/fault-tolerance/queryable_state.md
delete mode 100644 docs/content/docs/dev/datastream/fault-tolerance/queryable_state.md
delete mode 100644 docs/layouts/shortcodes/generated/queryable_state_configuration.html
[flink] 02/02: [FLINK-32559] Deprecate Queryable State
Posted by xt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xtsong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 8db8119138b492e16969fd363577efa082102538
Author: Xintong Song <to...@gmail.com>
AuthorDate: Tue Aug 1 16:12:46 2023 +0800
[FLINK-32559] Deprecate Queryable State
This closes #23113
---
.../docs/concepts/stateful-stream-processing.md | 2 -
.../docs/connectors/datastream/overview.md | 5 -
docs/content.zh/docs/deployment/config.md | 10 -
.../resource-providers/standalone/docker.md | 4 +-
.../resource-providers/standalone/kubernetes.md | 34 ---
.../docs/deployment/security/security-ssl.md | 6 -
.../datastream/fault-tolerance/queryable_state.md | 268 ------------------
docs/content.zh/docs/learn-flink/etl.md | 1 -
.../docs/concepts/stateful-stream-processing.md | 2 -
.../content/docs/connectors/datastream/overview.md | 9 -
docs/content/docs/deployment/config.md | 10 -
.../resource-providers/standalone/docker.md | 4 +-
.../resource-providers/standalone/kubernetes.md | 32 ---
.../docs/deployment/security/security-ssl.md | 6 -
.../datastream/fault-tolerance/queryable_state.md | 314 ---------------------
docs/content/docs/learn-flink/etl.md | 1 -
.../generated/queryable_state_configuration.html | 60 ----
.../flink/api/common/state/StateDescriptor.java | 9 +
.../flink/configuration/QueryableStateOptions.java | 8 +-
.../client/QueryableStateClient.java | 4 +
.../streaming/api/datastream/KeyedStream.java | 9 +
.../api/datastream/QueryableStateStream.java | 3 +
22 files changed, 36 insertions(+), 765 deletions(-)
diff --git a/docs/content.zh/docs/concepts/stateful-stream-processing.md b/docs/content.zh/docs/concepts/stateful-stream-processing.md
index 676391f7efe..0f16378d449 100644
--- a/docs/content.zh/docs/concepts/stateful-stream-processing.md
+++ b/docs/content.zh/docs/concepts/stateful-stream-processing.md
@@ -49,8 +49,6 @@ and [savepoints]({{< ref "docs/ops/state/savepoints" >}}).
Knowledge about the state also allows for rescaling Flink applications, meaning
that Flink takes care of redistributing state across parallel instances.
-[Queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) allows you to access state from outside of Flink during runtime.
-
When working with state, it might also be useful to read about [Flink's state
backends]({{< ref "docs/ops/state/state_backends" >}}). Flink
provides different state backends that specify how and where state is stored.
diff --git a/docs/content.zh/docs/connectors/datastream/overview.md b/docs/content.zh/docs/connectors/datastream/overview.md
index fe6b451d949..26694e174f3 100644
--- a/docs/content.zh/docs/connectors/datastream/overview.md
+++ b/docs/content.zh/docs/connectors/datastream/overview.md
@@ -75,9 +75,4 @@ Flink 还有些一些额外的连接器通过 [Apache Bahir](https://bahir.apach
一种常见的模式是从外部数据库或者 Web 服务查询数据得到初始数据流,然后通过 `Map` 或者 `FlatMap` 对初始数据流进行丰富和增强。
Flink 提供了[异步 I/O]({{< ref "docs/dev/datastream/operators/asyncio" >}}) API 来让这个过程更加简单、高效和稳定。
-### 可查询状态
-
-当 Flink 应用程序需要向外部存储推送大量数据时会导致 I/O 瓶颈问题出现。在这种场景下,如果对数据的读操作远少于写操作,那么让外部应用从 Flink 拉取所需的数据会是一种更好的方式。
-[可查询状态]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) 接口可以实现这个功能,该接口允许被 Flink 托管的状态可以被按需查询。
-
{{< top >}}
diff --git a/docs/content.zh/docs/deployment/config.md b/docs/content.zh/docs/deployment/config.md
index 7512997bb76..fd61b26ea13 100644
--- a/docs/content.zh/docs/deployment/config.md
+++ b/docs/content.zh/docs/deployment/config.md
@@ -298,16 +298,6 @@ See the [History Server Docs]({{< ref "docs/deployment/advanced/historyserver" >
*Options for experimental features in Flink.*
-### Queryable State
-
-*Queryable State* is an experimental features that gives lets you access Flink's internal state like a key/value store.
-See the [Queryable State Docs]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) for details.
-
-{{< generated/queryable_state_configuration >}}
-
-----
-----
-
# Client
{{< generated/client_configuration >}}
diff --git a/docs/content.zh/docs/deployment/resource-providers/standalone/docker.md b/docs/content.zh/docs/deployment/resource-providers/standalone/docker.md
index 9e5aa5eff15..ee0127e8932 100644
--- a/docs/content.zh/docs/deployment/resource-providers/standalone/docker.md
+++ b/docs/content.zh/docs/deployment/resource-providers/standalone/docker.md
@@ -574,7 +574,7 @@ You can customize the Flink image in several ways:
$ echo "
# enable an optional library
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/
+ ln -fs /opt/flink/opt/flink-sql-gateway-*.jar /opt/flink/lib/
# enable a custom library
ln -fs /mnt/custom_lib.jar /opt/flink/lib/
@@ -607,7 +607,7 @@ You can customize the Flink image in several ways:
ADD /host/path/to/flink-conf.yaml /container/local/path/to/custom/conf/flink-conf.yaml
ADD /host/path/to/log4j.properties /container/local/path/to/custom/conf/log4j.properties
- RUN ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/.
+ RUN ln -fs /opt/flink/opt/flink-sql-gateway-*.jar /opt/flink/lib/.
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
diff --git a/docs/content.zh/docs/deployment/resource-providers/standalone/kubernetes.md b/docs/content.zh/docs/deployment/resource-providers/standalone/kubernetes.md
index a44358b48ed..26e96eef3b0 100644
--- a/docs/content.zh/docs/deployment/resource-providers/standalone/kubernetes.md
+++ b/docs/content.zh/docs/deployment/resource-providers/standalone/kubernetes.md
@@ -247,15 +247,6 @@ data:
通常,只启动一个 JobManager pod 就足够了,因为一旦 pod 崩溃,Kubernetes 就会重新启动它。如果要实现更快的恢复,需要将 `jobmanager-session-deployment-ha.yaml` 中的 `replicas` 配置 或 `jobmanager-application-ha.yaml` 中的 `parallelism` 配置设定为大于 `1` 的整型值来启动 Standby JobManagers。
-<a name="enabling-queryable-state"></a>
-
-### 启用 Queryable State
-
-如果你为 TaskManager 创建了 `NodePort` service,那么你就可以访问 TaskManager 的 Queryable State 服务:
-
- 1. 运行 `kubectl create -f taskmanager-query-state-service.yaml` 来为 `taskmanager` pod 创建 `NodePort` service。`taskmanager-query-state-service.yaml` 的示例文件可以从[附录](#common-cluster-resource-definitions)中找到。
- 2. 运行 `kubectl get svc flink-taskmanager-query-state` 来查询 service 对应 node-port 的端口号。然后你就可以创建 [QueryableStateClient(<public-node-ip>, <node-port>]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}#querying-state) 来提交状态查询。
-
<a name="using-standalone-kubernetes-with-reactive-mode"></a>
### 在 Reactive 模式下使用 Standalone Kubernetes
@@ -292,7 +283,6 @@ data:
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
- queryable-state.proxy.ports: 6125
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
parallelism.default: 2
@@ -360,7 +350,6 @@ data:
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
- queryable-state.proxy.ports: 6125
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
parallelism.default: 2
@@ -454,25 +443,6 @@ spec:
component: jobmanager
```
-`taskmanager-query-state-service.yaml`。可选的 service,该 service 将 TaskManager 的端口暴露为公共 Kubernetes node 的节点端口,通过该端口来访问 queryable state 服务。
-
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: flink-taskmanager-query-state
-spec:
- type: NodePort
- ports:
- - name: query-state
- port: 6125
- targetPort: 6125
- nodePort: 30025
- selector:
- app: flink
- component: taskmanager
-```
-
<a name="session-cluster-resource-definitions"></a>
### Session 集群资源定义
@@ -610,8 +580,6 @@ spec:
ports:
- containerPort: 6122
name: rpc
- - containerPort: 6125
- name: query-state
livenessProbe:
tcpSocket:
port: 6122
@@ -775,8 +743,6 @@ spec:
ports:
- containerPort: 6122
name: rpc
- - containerPort: 6125
- name: query-state
livenessProbe:
tcpSocket:
port: 6122
diff --git a/docs/content.zh/docs/deployment/security/security-ssl.md b/docs/content.zh/docs/deployment/security/security-ssl.md
index 78f3a155152..49350ffae88 100644
--- a/docs/content.zh/docs/deployment/security/security-ssl.md
+++ b/docs/content.zh/docs/deployment/security/security-ssl.md
@@ -87,12 +87,6 @@ Examples for proxies that Flink users have deployed are [Envoy Proxy](https://ww
The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety of authentication options and thus better integration into existing infrastructures.
-
-### Queryable State
-
-Connections to the queryable state endpoints is currently not authenticated or encrypted.
-
-
## Configuring SSL
SSL can be enabled separately for *internal* and *external* connectivity:
diff --git a/docs/content.zh/docs/dev/datastream/fault-tolerance/queryable_state.md b/docs/content.zh/docs/dev/datastream/fault-tolerance/queryable_state.md
deleted file mode 100644
index 4a9a90845a6..00000000000
--- a/docs/content.zh/docs/dev/datastream/fault-tolerance/queryable_state.md
+++ /dev/null
@@ -1,268 +0,0 @@
----
-title: "Queryable State"
-weight: 5
-type: docs
-aliases:
- - /zh/dev/stream/state/queryable_state.html
-is_beta: true
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Queryable State
-
-{{< hint warning >}}
-目前 querable state 的客户端 API 还在不断演进,<strong>不保证</strong>现有接口的稳定性。在后续的 Flink 版本中有可能发生 API 变化。
-{{< /hint >}}
-
-简而言之, 这个特性将 Flink 的 managed keyed (partitioned) state
-(参考 [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}})) 暴露给外部,从而用户可以在 Flink 外部查询作业 state。
-在某些场景中,Queryable State 消除了对外部系统的分布式操作以及事务的需求,比如 KV 存储系统,而这些外部系统往往会成为瓶颈。除此之外,这个特性对于调试作业非常有用。
-
-{{< hint warning >}}
- <strong>注意:</strong> 进行查询时,state 会在并发线程中被访问,但 state 不会进行同步和拷贝。这种设计是为了避免同步和拷贝带来的作业延时。对于使用 Java 堆内存的 state backend,
- <i>比如</i> <code>MemoryStateBackend</code> 或者 <code>FsStateBackend</code>,它们获取状态时不会进行拷贝,而是直接引用状态对象,所以对状态的 read-modify-write 是不安全的,并且可能会因为并发修改导致查询失败。但 <code>RocksDBStateBackend</code> 是安全的,不会遇到上述问题。
-{{< /hint >}}
-
-## 架构
-
-在展示如何使用 Queryable State 之前,先简单描述一下该特性的组成部分,主要包括以下三部分:
-
- 1. `QueryableStateClient`,默认运行在 Flink 集群外部,负责提交用户的查询请求;
- 2. `QueryableStateClientProxy`,运行在每个 `TaskManager` 上(*即* Flink 集群内部),负责接收客户端的查询请求,从所负责的 Task Manager 获取请求的 state,并返回给客户端;
- 3. `QueryableStateServer`, 运行在 `TaskManager` 上,负责服务本地存储的 state。
-
-客户端连接到一个代理,并发送请求获取特定 `k` 对应的 state。 如 [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}) 所述,keyed state 按照
-*Key Groups* 进行划分,每个 `TaskManager` 会分配其中的一些 key groups。代理会询问 `JobManager` 以找到 `k` 所属 key group 的 TaskManager。根据返回的结果, 代理将会向运行在 `TaskManager` 上的 `QueryableStateServer` 查询 `k` 对应的 state, 并将结果返回给客户端。
-
-## 激活 Queryable State
-
-为了在 Flink 集群上使用 queryable state,需要进行以下操作:
-
- 1. 将 `flink-queryable-state-runtime-{{< version >}}.jar`
-从 [Flink distribution]({{< downloads >}} "Apache Flink: Downloads") 的 `opt/` 目录拷贝到 `lib/` 目录;
- 2. 将参数 `queryable-state.enable` 设置为 `true`。详细信息以及其它配置可参考文档 [Configuration]({{< ref "docs/deployment/config" >}}#queryable-state)。
-
-为了验证集群的 queryable state 已经被激活,可以检查任意 task manager 的日志中是否包含 "Started the Queryable State Proxy Server @ ..."。
-
-## 将 state 设置为可查询的
-
-激活集群的 queryable state 功能后,还要将 state 设置为可查询的才能对外可见,可以通过以下两种方式进行设置:
-
-* 创建 `QueryableStateStream`,它会作为一个 sink,并将输入数据转化为 queryable state;
-* 通过 `stateDescriptor.setQueryable(String queryableStateName)` 将 state 描述符所表示的 keyed state 设置成可查询的。
-
-接下来的部分将详细解释这两种方式。
-
-### Queryable State Stream
-
-在 `KeyedStream` 上调用 `.asQueryableState(stateName, stateDescriptor)` 将会返回一个 `QueryableStateStream`, 它会将流数据转化为 queryable state。
-对应不同的 state 类型,`asQueryableState()` 有以下一些方法变体:
-
-```java
-// ValueState
-QueryableStateStream asQueryableState(
- String queryableStateName,
- ValueStateDescriptor stateDescriptor)
-
-// Shortcut for explicit ValueStateDescriptor variant
-QueryableStateStream asQueryableState(String queryableStateName)
-
-// ReducingState
-QueryableStateStream asQueryableState(
- String queryableStateName,
- ReducingStateDescriptor stateDescriptor)
-```
-
-
-{{< hint info >}}
-**注意:** 没有可查询的 `ListState` sink,因为这种情况下 list 会不断增长,并且可能不会被清理,最终会消耗大量的内存。
-{{< /hint >}}
-
-返回的 `QueryableStateStream` 可以被视作一个sink,而且**不能再**被进一步转换。在内部实现上,一个 `QueryableStateStream` 被转换成一个 operator,使用输入的数据来更新 queryable state。state 如何更新是由 `asQueryableState` 提供的 `StateDescriptor` 来决定的。在下面的代码中, keyed stream 的所有数据将会通过 `ValueState.update(value)` 来更新状态:
-
-```java
-stream.keyBy(value -> value.f0).asQueryableState("query-name");
-```
-
-这个行为类似于 Scala API 中的 `flatMapWithState`。
-
-### Managed Keyed State
-
-operator 中的 Managed keyed state
-(参考 [Using Managed Keyed State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#using-managed-keyed-state))
-可以通过 `StateDescriptor.setQueryable(String queryableStateName)` 将 state descriptor 设置成可查询的,从而使 state 可查询,如下所示:
-
-```java
-ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
-descriptor.setQueryable("query-name"); // queryable state name
-```
-
-{{< hint info >}}
-**注意:** 参数 `queryableStateName` 可以任意选取,并且只被用来进行查询,它可以和 state 的名称不同。
-{{< /hint >}}
-
-这种方式不会限制 state 类型,即任意的 `ValueState`、`ReduceState`、`ListState`、`MapState`、`AggregatingState` 以及已弃用的 `FoldingState`
-均可作为 queryable state。
-
-## 查询 state
-
-目前为止,你已经激活了集群的 queryable state 功能,并且将一些 state 设置成了可查询的,接下来将会展示如何进行查询。
-
-为了进行查询,可以使用辅助类 `QueryableStateClient`,这个类位于 `flink-queryable-state-client` 的 jar 中,在项目的 `pom.xml` 需要显示添加对 `flink-queryable-state-client` 和 `flink-core` 的依赖, 如下所示:
-
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>{{< version >}}</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-queryable-state-client-java</artifactId>
- <version>{{< version >}}</version>
-</dependency>
-```
-
-关于依赖的更多信息, 可以参考如何[配置 Flink 项目]({{< ref "docs/dev/configuration/overview" >}})。
-
-`QueryableStateClient` 将提交你的请求到内部代理,代理会处理请求并返回结果。客户端的初始化只需要提供一个有效的 `TaskManager` 主机名
-(每个 task manager 上都运行着一个 queryable state 代理),以及代理监听的端口号。关于如何配置代理以及端口号可以参考 [Configuration Section](#configuration).
-
-```java
-QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort);
-```
-
-客户端就绪后,为了查询类型为 `K` 的 key,以及类型为 `V` 的state,可以使用如下方法:
-
-```java
-CompletableFuture<S> getKvState(
- JobID jobId,
- String queryableStateName,
- K key,
- TypeInformation<K> keyTypeInfo,
- StateDescriptor<S, V> stateDescriptor)
-```
-
-该方法会返回一个最终将包含 state 的 queryable state 实例,该实例可通过 JobID 和 queryableStateName 识别。在方法参数中,`key` 用来指定所要查询的状态所属的 key。
-`keyTypeInfo` 告诉 Flink 如何对 key 进行序列化和反序列化。`stateDescriptor` 包含了所请求 state 的必要信息,即 state 类型(`Value`,`Reduce` 等等),
-以及如何对其进行序列化和反序列。
-
-细心的读者会注意到返回的 future 包含类型为 `S` 的值,*即*一个存储实际值的 `State` 对象。它可以是Flink支持的任何类型的 state:`ValueState`、`ReduceState`、
-`ListState`、`MapState`、`AggregatingState` 以及弃用的 `FoldingState`。
-
-{{< hint info >}}
-**注意:** 这些 state 对象不允许对其中的 state 进行修改。你可以通过 `valueState.get()` 获取实际的 state,
-或者通过 `mapState.entries()` 遍历所有 `<K, V>`,但是不能修改它们。举例来说,对返回的 list state 调用 `add()`
- 方法将会导致 `UnsupportedOperationException`。
-{{< /hint >}}
-
-{{< hint info >}}
-**注意:** 客户端是异步的,并且可能被多个线程共享。客户端不再使用后需要通过 `QueryableStateClient.shutdown()`
- 来终止,从而释放资源。
-{{< /hint >}}
-
-### 示例
-
-下面的例子扩展自 `CountWindowAverage`
-(参考 [Using Managed Keyed State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#using-managed-keyed-state)),
-将其中的 state 设置成可查询的,并展示了如何进行查询:
-
-```java
-public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>> {
-
- private transient ValueState<Tuple2<Long, Long>> sum; // a tuple containing the count and the sum
-
- @Override
- public void flatMap(Tuple2<Long, Long> input, Collector<Tuple2<Long, Long>> out) throws Exception {
- Tuple2<Long, Long> currentSum = sum.value();
- currentSum.f0 += 1;
- currentSum.f1 += input.f1;
- sum.update(currentSum);
-
- if (currentSum.f0 >= 2) {
- out.collect(new Tuple2<>(input.f0, currentSum.f1 / currentSum.f0));
- sum.clear();
- }
- }
-
- @Override
- public void open(Configuration config) {
- ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
- descriptor.setQueryable("query-name");
- sum = getRuntimeContext().getState(descriptor);
- }
-}
-```
-
-上面的代码作为作业运行后,可以获取作业的 ID,然后可以通过下面的方式查询任何 key 下的 state。
-
-```java
-QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort);
-
-// the state descriptor of the state to be fetched.
-ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average",
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}));
-
-CompletableFuture<ValueState<Tuple2<Long, Long>>> resultFuture =
- client.getKvState(jobId, "query-name", key, BasicTypeInfo.LONG_TYPE_INFO, descriptor);
-
-// now handle the returned value
-resultFuture.thenAccept(response -> {
- try {
- Tuple2<Long, Long> res = response.get();
- } catch (Exception e) {
- e.printStackTrace();
- }
-});
-```
-
-## Configuration
-
-下面的配置会影响 queryable state 服务器端和客户端的行为,它们定义在 `QueryableStateOptions`。
-
-### State Server
-* `queryable-state.server.ports`: 服务器端口范围,如果同一台机器上运行了多个 task manager,可以避免端口冲突。指定的可以是一个具体的端口号,如 "9123",
- 可以是一个端口范围,如 "50100-50200",或者可以是端口范围以及端口号的组合,如 "50100-50200,50300-50400,51234"。默认端口号是 9067。
-* `queryable-state.server.network-threads`: 服务器端 network (event loop) thread 的数量,用来接收查询请求 (如果设置为0,则线程数为 slot 数)。
-* `queryable-state.server.query-threads`: 服务器端处理查询请求的线程数 (如果设置为0,则线程数为 slot 数)。
-
-
-### Proxy
-* `queryable-state.proxy.ports`: 代理的服务端口范围。如果同一台机器上运行了多个 task manager,可以避免端口冲突。指定的可以是一个具体的端口号,如 "9123",
- 可以是一个端口范围,如"50100-50200",或者可以是端口范围以及端口号的组合,如 "50100-50200,50300-50400,51234"。默认端口号是 9069。
-* `queryable-state.proxy.network-threads`: 代理上 network (event loop) thread 的数量,用来接收查询请求 (如果设置为0,则线程数为 slot 数)。
-* `queryable-state.proxy.query-threads`: 代理上处理查询请求的线程数 (如果设置为0,则线程数为 slot 数)。
-
-## 限制
-
-* queryable state 的生命周期受限于作业的生命周期,*比如* tasks 在启动时注册可查询状态,并在退出时注销。在后续版本中,希望能够将其解耦
-从而允许 task 结束后依然能够查询 state,并且通过 state 备份来加速恢复。
-* 目前是通过 tell 来通知可用的 KvState。将来会使用 asks 和 acknowledgements 来提升稳定性。
-* 服务器端和客户端会记录请求的统计信息。因为统计信息目前不会暴露给外部,所以这个功能默认没有开启。如果将来支持通过 Metrics 系统发布这些数据,将开启统计功能。
-
-{{< top >}}
diff --git a/docs/content.zh/docs/learn-flink/etl.md b/docs/content.zh/docs/learn-flink/etl.md
index 30812fd42be..591d6899ef9 100644
--- a/docs/content.zh/docs/learn-flink/etl.md
+++ b/docs/content.zh/docs/learn-flink/etl.md
@@ -227,7 +227,6 @@ minutesByStartCell
* **持久性**: Flink 状态是容错的,例如,它可以自动按一定的时间间隔产生 checkpoint,并且在任务失败后进行恢复
* **纵向可扩展性**: Flink 状态可以存储在集成的 RocksDB 实例中,这种方式下可以通过增加本地磁盘来扩展空间
* **横向可扩展性**: Flink 状态可以随着集群的扩缩容重新分布
-* **可查询性**: Flink 状态可以通过使用 [状态查询 API]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) 从外部进行查询。
在本节中你将学习如何使用 Flink 的 API 来管理 keyed state。
diff --git a/docs/content/docs/concepts/stateful-stream-processing.md b/docs/content/docs/concepts/stateful-stream-processing.md
index 7f139b021bb..fccddfcd3d6 100644
--- a/docs/content/docs/concepts/stateful-stream-processing.md
+++ b/docs/content/docs/concepts/stateful-stream-processing.md
@@ -49,8 +49,6 @@ and [savepoints]({{< ref "docs/ops/state/savepoints" >}}).
Knowledge about the state also allows for rescaling Flink applications, meaning
that Flink takes care of redistributing state across parallel instances.
-[Queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) allows you to access state from outside of Flink during runtime.
-
When working with state, it might also be useful to read about [Flink's state
backends]({{< ref "docs/ops/state/state_backends" >}}). Flink
provides different state backends that specify how and where state is stored.
diff --git a/docs/content/docs/connectors/datastream/overview.md b/docs/content/docs/connectors/datastream/overview.md
index 4891b9f79f1..a1b2cb9ccb5 100644
--- a/docs/content/docs/connectors/datastream/overview.md
+++ b/docs/content/docs/connectors/datastream/overview.md
@@ -82,14 +82,5 @@ in order to enrich the primary datastream.
Flink offers an API for [Asynchronous I/O]({{< ref "docs/dev/datastream/operators/asyncio" >}})
to make it easier to do this kind of enrichment efficiently and robustly.
-### Queryable State
-
-When a Flink application pushes a lot of data to an external data store, this
-can become an I/O bottleneck.
-If the data involved has many fewer reads than writes, a better approach can be
-for an external application to pull from Flink the data it needs.
-The [Queryable State]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) interface
-enables this by allowing the state being managed by Flink to be queried on demand.
-
{{< top >}}
diff --git a/docs/content/docs/deployment/config.md b/docs/content/docs/deployment/config.md
index 2f9780e140d..76883d686c3 100644
--- a/docs/content/docs/deployment/config.md
+++ b/docs/content/docs/deployment/config.md
@@ -300,16 +300,6 @@ See the [History Server Docs]({{< ref "docs/deployment/advanced/historyserver" >
*Options for experimental features in Flink.*
-### Queryable State
-
-*Queryable State* is an experimental features that gives lets you access Flink's internal state like a key/value store.
-See the [Queryable State Docs]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}) for details.
-
-{{< generated/queryable_state_configuration >}}
-
-----
-----
-
# Client
{{< generated/client_configuration >}}
diff --git a/docs/content/docs/deployment/resource-providers/standalone/docker.md b/docs/content/docs/deployment/resource-providers/standalone/docker.md
index b941b8f834c..b69be010673 100644
--- a/docs/content/docs/deployment/resource-providers/standalone/docker.md
+++ b/docs/content/docs/deployment/resource-providers/standalone/docker.md
@@ -573,7 +573,7 @@ You can customize the Flink image in several ways:
$ echo "
# enable an optional library
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/
+ ln -fs /opt/flink/opt/flink-sql-gateway-*.jar /opt/flink/lib/
# enable a custom library
ln -fs /mnt/custom_lib.jar /opt/flink/lib/
@@ -606,7 +606,7 @@ You can customize the Flink image in several ways:
ADD /host/path/to/flink-conf.yaml /container/local/path/to/custom/conf/flink-conf.yaml
ADD /host/path/to/log4j.properties /container/local/path/to/custom/conf/log4j.properties
- RUN ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/.
+ RUN ln -fs /opt/flink/opt/flink-sql-gateway-*.jar /opt/flink/lib/.
RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
diff --git a/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md b/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md
index 6044530a675..d14d0b5003c 100644
--- a/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md
+++ b/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md
@@ -236,12 +236,6 @@ Refer to the [appendix](#appendix) for full configuration.
Usually, it is enough to only start a single JobManager pod, because Kubernetes will restart it once the pod crashes.
If you want to achieve faster recovery, configure the `replicas` in `jobmanager-session-deployment-ha.yaml` or `parallelism` in `jobmanager-application-ha.yaml` to a value greater than `1` to start standby JobManagers.
-### Enabling Queryable State
-
-You can access the queryable state of TaskManager if you create a `NodePort` service for it:
- 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service for the `taskmanager` pod. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
- 2. Run `kubectl get svc flink-taskmanager-query-state` to get the `<node-port>` of this service. Then you can create the [QueryableStateClient(<public-node-ip>, <node-port>]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}#querying-state) to submit state queries.
-
### Using Standalone Kubernetes with Reactive Mode
[Reactive Mode]({{< ref "docs/deployment/elastic_scaling" >}}#reactive-mode) allows to run Flink in a mode, where the *Application Cluster* is always adjusting the job parallelism to the available resources. In combination with Kubernetes, the replica count of the TaskManager deployment determines the available resources. Increasing the replica count will scale up the job, reducing it will trigger a scale down. This can also be done automatically by using a [Horizontal Pod Autoscaler](ht [...]
@@ -282,7 +276,6 @@ data:
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
- queryable-state.proxy.ports: 6125
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
parallelism.default: 2
@@ -350,7 +343,6 @@ data:
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
- queryable-state.proxy.ports: 6125
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
parallelism.default: 2
@@ -441,24 +433,6 @@ spec:
component: jobmanager
```
-`taskmanager-query-state-service.yaml`. Optional service, that exposes the TaskManager port to access the queryable state as a public Kubernetes node's port.
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: flink-taskmanager-query-state
-spec:
- type: NodePort
- ports:
- - name: query-state
- port: 6125
- targetPort: 6125
- nodePort: 30025
- selector:
- app: flink
- component: taskmanager
-```
-
### Session cluster resource definitions
`jobmanager-session-deployment-non-ha.yaml`
@@ -594,8 +568,6 @@ spec:
ports:
- containerPort: 6122
name: rpc
- - containerPort: 6125
- name: query-state
livenessProbe:
tcpSocket:
port: 6122
@@ -757,8 +729,6 @@ spec:
ports:
- containerPort: 6122
name: rpc
- - containerPort: 6125
- name: query-state
livenessProbe:
tcpSocket:
port: 6122
@@ -846,8 +816,6 @@ spec:
ports:
- containerPort: 6122
name: rpc
- - containerPort: 6125
- name: query-state
- containerPort: 6121
name: metrics
livenessProbe:
diff --git a/docs/content/docs/deployment/security/security-ssl.md b/docs/content/docs/deployment/security/security-ssl.md
index 674614b83d8..64ba8f8d210 100644
--- a/docs/content/docs/deployment/security/security-ssl.md
+++ b/docs/content/docs/deployment/security/security-ssl.md
@@ -87,12 +87,6 @@ Examples for proxies that Flink users have deployed are [Envoy Proxy](https://ww
The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety of authentication options and thus better integration into existing infrastructures.
-
-### Queryable State
-
-Connections to the queryable state endpoints is currently not authenticated or encrypted.
-
-
## Configuring SSL
SSL can be enabled separately for *internal* and *external* connectivity:
diff --git a/docs/content/docs/dev/datastream/fault-tolerance/queryable_state.md b/docs/content/docs/dev/datastream/fault-tolerance/queryable_state.md
deleted file mode 100644
index 2a9edc3ffa6..00000000000
--- a/docs/content/docs/dev/datastream/fault-tolerance/queryable_state.md
+++ /dev/null
@@ -1,314 +0,0 @@
----
-title: "Queryable State"
-weight: 5
-type: docs
-aliases:
- - /dev/stream/state/queryable_state.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Queryable State
-
-{{< hint warning >}}
- The client APIs for queryable state are currently in an evolving state and
- there are **no guarantees** made about stability of the provided interfaces. It is
- likely that there will be breaking API changes on the client side in the upcoming Flink versions.
-{{< /hint >}}
-
-In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
-(see [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}})) to the outside world and
-allows the user to query a job's state from outside Flink. For some scenarios, queryable state
-eliminates the need for distributed operations/transactions with external systems such as key-value
-stores which are often the bottleneck in practice. In addition, this feature may be particularly
-useful for debugging purposes.
-
-{{< hint warning >}}
- When querying a state object, that object is accessed from a concurrent
- thread without any synchronization or copying. This is a design choice, as any of the above would lead
- to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
- <i>e.g.</i> <code>HashMapStateBackend</code>, does not work
- with copies when retrieving values but instead directly references the stored values, read-modify-write
- patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
- The <code>EmbeddedRocksDBStateBackend</code> is safe from these issues.
-{{< /hint >}}
-
-## Architecture
-
-Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
-The Queryable State feature consists of three main entities:
-
- 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
- 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
- for receiving the client's queries, fetching the requested state from the responsible Task Manager on his behalf, and
- returning it to the client, and
- 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
-
-The client connects to one of the proxies and sends a request for the state associated with a specific
-key, `k`. As stated in [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}), keyed state is organized in
-*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
-responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
-then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the
-response back to the client.
-
-## Activating Queryable State
-
-To enable queryable state on your Flink cluster, you need to do the following:
-
- 1. copy the `flink-queryable-state-runtime-{{< version >}}.jar`
-from the `opt/` folder of your [Flink distribution]({{< downloads >}} "Apache Flink: Downloads"),
-to the `lib/` folder.
- 2. set the property `queryable-state.enable` to `true`. See the [Configuration]({{< ref "docs/deployment/config" >}}#queryable-state) documentation for details and additional parameters.
-
-To verify that your cluster is running with queryable state enabled, check the logs of any
-task manager for the line: `"Started the Queryable State Proxy Server @ ..."`.
-
-## Making State Queryable
-
-Now that you have activated queryable state on your cluster, it is time to see how to use it. In order for a state to
-be visible to the outside world, it needs to be explicitly made queryable by using:
-
-* either a `QueryableStateStream`, a convenience object which acts as a sink and offers its incoming values as queryable
-state, or
-* the `stateDescriptor.setQueryable(String queryableStateName)` method, which makes the keyed state represented by the
- state descriptor, queryable.
-
-The following sections explain the use of these two approaches.
-
-### Queryable State Stream
-
-Calling `.asQueryableState(stateName, stateDescriptor)` on a `KeyedStream` returns a `QueryableStateStream` which offers
-its values as queryable state. Depending on the type of state, there are the following variants of the `asQueryableState()`
-method:
-
-```java
-// ValueState
-QueryableStateStream asQueryableState(
- String queryableStateName,
- ValueStateDescriptor stateDescriptor)
-
-// Shortcut for explicit ValueStateDescriptor variant
-QueryableStateStream asQueryableState(String queryableStateName)
-
-// ReducingState
-QueryableStateStream asQueryableState(
- String queryableStateName,
- ReducingStateDescriptor stateDescriptor)
-```
-
-
-{{< hint info >}}
-**Note:** There is no queryable `ListState` sink as it would result in an ever-growing
-list which may not be cleaned up and thus will eventually consume too much memory.
-{{< /hint >}}
-
-The returned `QueryableStateStream` can be seen as a sink and **cannot** be further transformed. Internally, a
-`QueryableStateStream` gets translated to an operator which uses all incoming records to update the queryable state
-instance. The updating logic is implied by the type of the `StateDescriptor` provided in the `asQueryableState` call.
-In a program like the following, all records of the keyed stream will be used to update the state instance via the
-`ValueState.update(value)`:
-
-```java
-stream.keyBy(value -> value.f0).asQueryableState("query-name");
-```
-
-This acts like the Scala API's `flatMapWithState`.
-
-### Managed Keyed State
-
-Managed keyed state of an operator
-(see [Using Managed Keyed State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#using-managed-keyed-state))
-can be made queryable by making the appropriate state descriptor queryable via
-`StateDescriptor.setQueryable(String queryableStateName)`, as in the example below:
-```java
-ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
-descriptor.setQueryable("query-name"); // queryable state name
-```
-
-{{< hint info >}}
-**Note:** The `queryableStateName` parameter may be chosen arbitrarily and is only
-used for queries. It does not have to be identical to the state's own name.
-{{< /hint >}}
-
-This variant has no limitations as to which type of state can be made queryable. This means that this can be used for
-any `ValueState`, `ReduceState`, `ListState`, `MapState`, and `AggregatingState`.
-
-## Querying State
-
-So far, you have set up your cluster to run with queryable state and you have declared (some of) your state as
-queryable. Now it is time to see how to query this state.
-
-For this you can use the `QueryableStateClient` helper class. This is available in the `flink-queryable-state-client`
-jar which must be explicitly included as a dependency in the `pom.xml` of your project along with `flink-core`, as shown below:
-
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>{{< version >}}</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-queryable-state-client-java</artifactId>
- <version>{{< version >}}</version>
-</dependency>
-```
-
-For more on this, you can check how to [set up a Flink program]({{< ref "docs/dev/configuration/overview" >}}).
-
-The `QueryableStateClient` will submit your query to the internal proxy, which will then process your query and return
-the final result. The only requirement to initialize the client is to provide a valid `TaskManager` hostname (remember
-that there is a queryable state proxy running on each task manager) and the port where the proxy listens. More on how
-to configure the proxy and state server port(s) in the [Configuration Section](#configuration).
-
-```java
-QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort);
-```
-
-With the client ready, to query a state of type `V`, associated with a key of type `K`, you can use the method:
-
-```java
-CompletableFuture<S> getKvState(
- JobID jobId,
- String queryableStateName,
- K key,
- TypeInformation<K> keyTypeInfo,
- StateDescriptor<S, V> stateDescriptor)
-```
-
-The above returns a `CompletableFuture` eventually holding the state value for the queryable state instance identified
-by `queryableStateName` of the job with ID `jobID`. The `key` is the key whose state you are interested in and the
-`keyTypeInfo` will tell Flink how to serialize/deserialize it. Finally, the `stateDescriptor` contains the necessary
-information about the requested state, namely its type (`Value`, `Reduce`, etc) and the necessary information on how
-to serialize/deserialize it.
-
-The careful reader will notice that the returned future contains a value of type `S`, *i.e.* a `State` object containing
-the actual value. This can be any of the state types supported by Flink: `ValueState`, `ReduceState`, `ListState`, `MapState`,
-and `AggregatingState`.
-
-{{< hint info >}}
-**Note:** These state objects do not allow modifications to the contained state. You can use them to get
-the actual value of the state, *e.g.* using `valueState.get()`, or iterate over
-the contained `<K, V>` entries, *e.g.* using the `mapState.entries()`, but you cannot
-modify them. As an example, calling the `add()` method on a returned list state will throw an
-`UnsupportedOperationException`.
-{{< /hint >}}
-
-{{< hint info >}}
-**Note:** The client is asynchronous and can be shared by multiple threads. It needs
-to be shutdown via <code>QueryableStateClient.shutdown()</code> when unused in order to free
-resources.
-{{< /hint >}}
-
-### Example
-
-The following example extends the `CountWindowAverage` example
-(see [Using Managed Keyed State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#using-managed-keyed-state))
-by making it queryable and shows how to query this value:
-
-```java
-public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>> {
-
- private transient ValueState<Tuple2<Long, Long>> sum; // a tuple containing the count and the sum
-
- @Override
- public void flatMap(Tuple2<Long, Long> input, Collector<Tuple2<Long, Long>> out) throws Exception {
- Tuple2<Long, Long> currentSum = sum.value();
- currentSum.f0 += 1;
- currentSum.f1 += input.f1;
- sum.update(currentSum);
-
- if (currentSum.f0 >= 2) {
- out.collect(new Tuple2<>(input.f0, currentSum.f1 / currentSum.f0));
- sum.clear();
- }
- }
-
- @Override
- public void open(Configuration config) {
- ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {})); // type information
- descriptor.setQueryable("query-name");
- sum = getRuntimeContext().getState(descriptor);
- }
-}
-```
-
-Once used in a job, you can retrieve the job ID and then query any key's current state from this operator:
-
-```java
-QueryableStateClient client = new QueryableStateClient(tmHostname, proxyPort);
-
-// the state descriptor of the state to be fetched.
-ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average",
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}));
-
-CompletableFuture<ValueState<Tuple2<Long, Long>>> resultFuture =
- client.getKvState(jobId, "query-name", key, BasicTypeInfo.LONG_TYPE_INFO, descriptor);
-
-// now handle the returned value
-resultFuture.thenAccept(response -> {
- try {
- Tuple2<Long, Long> res = response.get();
- } catch (Exception e) {
- e.printStackTrace();
- }
-});
-```
-
-## Configuration
-
-The following configuration parameters influence the behaviour of the queryable state server and client.
-They are defined in `QueryableStateOptions`.
-
-### State Server
-* `queryable-state.server.ports`: the server port range of the queryable state server. This is useful to avoid port clashes if more
- than 1 task managers run on the same machine. The specified range can be: a port: "9123", a range of ports: "50100-50200",
- or a list of ranges and or points: "50100-50200,50300-50400,51234". The default port is 9067.
-* `queryable-state.server.network-threads`: number of network (event loop) threads receiving incoming requests for the state server (0 => #slots)
-* `queryable-state.server.query-threads`: number of threads handling/serving incoming requests for the state server (0 => #slots).
-
-
-### Proxy
-* `queryable-state.proxy.ports`: the server port range of the queryable state proxy. This is useful to avoid port clashes if more
- than 1 task managers run on the same machine. The specified range can be: a port: "9123", a range of ports: "50100-50200",
- or a list of ranges and or points: "50100-50200,50300-50400,51234". The default port is 9069.
-* `queryable-state.proxy.network-threads`: number of network (event loop) threads receiving incoming requests for the client proxy (0 => #slots)
-* `queryable-state.proxy.query-threads`: number of threads handling/serving incoming requests for the client proxy (0 => #slots).
-
-## Limitations
-
-* The queryable state life-cycle is bound to the life-cycle of the job, *e.g.* tasks register
-queryable state on startup and unregister it on disposal. In future versions, it is desirable to
-decouple this in order to allow queries after a task finishes, and to speed up recovery via state
-replication.
-* Notifications about available KvState happen via a simple tell. In the future this should be improved to be
-more robust with asks and acknowledgements.
-* The server and client keep track of statistics for queries. These are currently disabled by
-default as they would not be exposed anywhere. As soon as there is better support to publish these
-numbers via the Metrics system, we should enable the stats.
-
-{{< top >}}
diff --git a/docs/content/docs/learn-flink/etl.md b/docs/content/docs/learn-flink/etl.md
index 2a969327896..aa1423edb35 100644
--- a/docs/content/docs/learn-flink/etl.md
+++ b/docs/content/docs/learn-flink/etl.md
@@ -262,7 +262,6 @@ Your applications are certainly capable of using state without getting Flink inv
* **durable**: Flink state is fault-tolerant, i.e., it is automatically checkpointed at regular intervals, and is restored upon failure
* **vertically scalable**: Flink state can be kept in embedded RocksDB instances that scale by adding more local disk
* **horizontally scalable**: Flink state is redistributed as your cluster grows and shrinks
-* **queryable**: Flink state can be queried externally via the [Queryable State API]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}}).
In this section you will learn how to work with Flink's APIs that manage keyed state.
diff --git a/docs/layouts/shortcodes/generated/queryable_state_configuration.html b/docs/layouts/shortcodes/generated/queryable_state_configuration.html
deleted file mode 100644
index 0c6d4cb1a9f..00000000000
--- a/docs/layouts/shortcodes/generated/queryable_state_configuration.html
+++ /dev/null
@@ -1,60 +0,0 @@
-<table class="configuration table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>queryable-state.client.network-threads</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>Number of network (Netty's event loop) Threads for queryable state client.</td>
- </tr>
- <tr>
- <td><h5>queryable-state.enable</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>Option whether the queryable state proxy and server should be enabled where possible and configurable.</td>
- </tr>
- <tr>
- <td><h5>queryable-state.proxy.network-threads</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>Number of network (Netty's event loop) Threads for queryable state proxy.</td>
- </tr>
- <tr>
- <td><h5>queryable-state.proxy.ports</h5></td>
- <td style="word-wrap: break-word;">"9069"</td>
- <td>String</td>
- <td>The port range of the queryable state proxy. The specified range can be a single port: "9123", a range of ports: "50100-50200", or a list of ranges and ports: "50100-50200,50300-50400,51234".</td>
- </tr>
- <tr>
- <td><h5>queryable-state.proxy.query-threads</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>Number of query Threads for queryable state proxy. Uses the number of slots if set to 0.</td>
- </tr>
- <tr>
- <td><h5>queryable-state.server.network-threads</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>Number of network (Netty's event loop) Threads for queryable state server.</td>
- </tr>
- <tr>
- <td><h5>queryable-state.server.ports</h5></td>
- <td style="word-wrap: break-word;">"9067"</td>
- <td>String</td>
- <td>The port range of the queryable state server. The specified range can be a single port: "9123", a range of ports: "50100-50200", or a list of ranges and ports: "50100-50200,50300-50400,51234".</td>
- </tr>
- <tr>
- <td><h5>queryable-state.server.query-threads</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>Number of query Threads for queryable state server. Uses the number of slots if set to 0.</td>
- </tr>
- </tbody>
-</table>
diff --git a/flink-core/src/main/java/org/apache/flink/api/common/state/StateDescriptor.java b/flink-core/src/main/java/org/apache/flink/api/common/state/StateDescriptor.java
index 309abfb3919..b38461d5b34 100644
--- a/flink-core/src/main/java/org/apache/flink/api/common/state/StateDescriptor.java
+++ b/flink-core/src/main/java/org/apache/flink/api/common/state/StateDescriptor.java
@@ -227,7 +227,10 @@ public abstract class StateDescriptor<S extends State, T> implements Serializabl
*
* @param queryableStateName State name for queries (unique name per job)
* @throws IllegalStateException If queryable state name already set
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
+ @Deprecated
public void setQueryable(String queryableStateName) {
Preconditions.checkArgument(
ttlConfig.getUpdateType() == StateTtlConfig.UpdateType.Disabled,
@@ -244,8 +247,11 @@ public abstract class StateDescriptor<S extends State, T> implements Serializabl
* Returns the queryable state name.
*
* @return Queryable state name or <code>null</code> if not set.
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
@Nullable
+ @Deprecated
public String getQueryableStateName() {
return queryableStateName;
}
@@ -254,7 +260,10 @@ public abstract class StateDescriptor<S extends State, T> implements Serializabl
* Returns whether the state created from this descriptor is queryable.
*
* @return <code>true</code> if state is queryable, <code>false</code> otherwise.
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
+ @Deprecated
public boolean isQueryable() {
return queryableStateName != null;
}
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/QueryableStateOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/QueryableStateOptions.java
index c5712be9959..e41c0d8f51e 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/QueryableStateOptions.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/QueryableStateOptions.java
@@ -22,8 +22,14 @@ import org.apache.flink.annotation.PublicEvolving;
import static org.apache.flink.configuration.ConfigOptions.key;
-/** The set of configuration options relating to Queryable State. */
+/**
+ * The set of configuration options relating to Queryable State.
+ *
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed in a
+ * future Flink major version.
+ */
@PublicEvolving
+@Deprecated
public class QueryableStateOptions {
// ------------------------------------------------------------------------
diff --git a/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java b/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java
index 0a959bf209b..8da88cda282 100644
--- a/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java
+++ b/flink-queryable-state/flink-queryable-state-client-java/src/main/java/org/apache/flink/queryablestate/client/QueryableStateClient.java
@@ -71,8 +71,12 @@ import java.util.stream.Stream;
* Resolved locations are cached. When the server address of the requested KvState instance is
* determined, the client sends out a request to the server. The returned final answer is then
* forwarded to the Client.
+ *
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed in a
+ * future Flink major version.
*/
@PublicEvolving
+@Deprecated
public class QueryableStateClient {
private static final Logger LOG = LoggerFactory.getLogger(QueryableStateClient.class);
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/KeyedStream.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/KeyedStream.java
index 504c357d608..6b3f2d1139c 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/KeyedStream.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/KeyedStream.java
@@ -1055,8 +1055,11 @@ public class KeyedStream<T, KEY> extends DataStream<T> {
*
* @param queryableStateName Name under which to the publish the queryable state instance
* @return Queryable state instance
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
@PublicEvolving
+ @Deprecated
public QueryableStateStream<KEY, T> asQueryableState(String queryableStateName) {
ValueStateDescriptor<T> valueStateDescriptor =
new ValueStateDescriptor<>(UUID.randomUUID().toString(), getType());
@@ -1070,8 +1073,11 @@ public class KeyedStream<T, KEY> extends DataStream<T> {
* @param queryableStateName Name under which to the publish the queryable state instance
* @param stateDescriptor State descriptor to create state instance from
* @return Queryable state instance
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
@PublicEvolving
+ @Deprecated
public QueryableStateStream<KEY, T> asQueryableState(
String queryableStateName, ValueStateDescriptor<T> stateDescriptor) {
@@ -1094,8 +1100,11 @@ public class KeyedStream<T, KEY> extends DataStream<T> {
* @param queryableStateName Name under which to the publish the queryable state instance
* @param stateDescriptor State descriptor to create state instance from
* @return Queryable state instance
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed
+ * in a future Flink major version.
*/
@PublicEvolving
+ @Deprecated
public QueryableStateStream<KEY, T> asQueryableState(
String queryableStateName, ReducingStateDescriptor<T> stateDescriptor) {
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/QueryableStateStream.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/QueryableStateStream.java
index ef9c1172416..e876f82babd 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/QueryableStateStream.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/QueryableStateStream.java
@@ -28,8 +28,11 @@ import org.apache.flink.util.Preconditions;
*
* @param <K> State key type
* @param <V> State value type
+ * @deprecated The Queryable State feature is deprecated since Flink 1.18, and will be removed in a
+ * future Flink major version.
*/
@PublicEvolving
+@Deprecated
public class QueryableStateStream<K, V> {
/** Name under which the state is queryable. */
[flink] 01/02: [FLINK-32559] Do not generate config option docs for Deprecated / ExcludeFromDocumentation classes.
Posted by xt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xtsong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit d012b8c981bccb178827a06627a08c21ef6ee380
Author: Xintong Song <to...@gmail.com>
AuthorDate: Wed Aug 2 09:43:21 2023 +0800
[FLINK-32559] Do not generate config option docs for Deprecated / ExcludeFromDocumentation classes.
---
.../org/apache/flink/docs/util/ConfigurationOptionLocator.java | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/flink-docs/src/main/java/org/apache/flink/docs/util/ConfigurationOptionLocator.java b/flink-docs/src/main/java/org/apache/flink/docs/util/ConfigurationOptionLocator.java
index 54c9666524c..81f5713d055 100644
--- a/flink-docs/src/main/java/org/apache/flink/docs/util/ConfigurationOptionLocator.java
+++ b/flink-docs/src/main/java/org/apache/flink/docs/util/ConfigurationOptionLocator.java
@@ -19,6 +19,7 @@
package org.apache.flink.docs.util;
import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.annotation.docs.Documentation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.util.function.BiConsumerWithException;
@@ -161,7 +162,9 @@ public class ConfigurationOptionLocator {
if (!EXCLUSIONS.contains(className)) {
Class<?> optionsClass = Class.forName(className);
- optionClasses.add(optionsClass);
+ if (shouldBeDocumented(optionsClass)) {
+ optionClasses.add(optionsClass);
+ }
}
}
}
@@ -169,6 +172,11 @@ public class ConfigurationOptionLocator {
return optionClasses;
}
+ private static boolean shouldBeDocumented(Class<?> clazz) {
+ return clazz.getAnnotation(Deprecated.class) == null
+ && clazz.getAnnotation(Documentation.ExcludeFromDocumentation.class) == null;
+ }
+
@VisibleForTesting
public static List<OptionWithMetaInfo> extractConfigOptions(Class<?> clazz) {
try {