You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by kl0u <gi...@git.apache.org> on 2017/11/07 11:06:03 UTC
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/4966
[FLINK-7822][FLINK-7823] Adds documentation and fixes configuration of QS.
## What is the purpose of the change
*This PR adds documentation for the new queryable state.
R @aljoscha
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink qs-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4966.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4966
----
commit 91c7082158e534ffc30894e9f23e6ea6c94fab07
Author: kkloudas <kk...@gmail.com>
Date: 2017-11-06T11:43:18Z
[FLINK-7823][QS] Update Queryable State configuration parameters.
commit 2a076d6119eb01a4b4bc48925a37dde1b439dd54
Author: kkloudas <kk...@gmail.com>
Date: 2017-11-06T16:21:45Z
[FLINK-7822][QS][doc] Update Queryable State docs.
----
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337260
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
--- End diff --
nit: "The Queryable State feature"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337715
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
+`flink-queryable-state-runtime{{ site.scala_version_suffix }}-{{site.version }}.jar`
+from the `opt/` folder of your [Flink distribution](https://flink.apache.org/downloads.html "Apache Flink: Downloads"),
+to the `lib/` folder. In other case, the queryable state feature is not enabled.
--- End diff --
nit: "Otherwise, ..."
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337397
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his behalf, and returning it to the client, and
--- End diff --
maybe "fetching the requested state from the responsible TaskManager"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337649
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
--- End diff --
nit: omit "the"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149337820
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
+`flink-queryable-state-runtime{{ site.scala_version_suffix }}-{{site.version }}.jar`
+from the `opt/` folder of your [Flink distribution](https://flink.apache.org/downloads.html "Apache Flink: Downloads"),
+to the `lib/` folder. In other case, the queryable state feature is not enabled.
+
+To verify that your cluster is running with queryable state enabled, check the logs of any
+task manager for the line: `"Started the Queryable State Proxy Server @ ..."`.
+
## Making State Queryable
-In order to make state queryable, the queryable state server first needs to be enabled globally
-by setting the `query.server.enable` configuration parameter to `true` (current default).
-Then appropriate state needs to be made queryable by using either
+Now that you have activated queryable state on your cluster, it is time to see how to use it. In order for a state to
--- End diff --
"state to be visible"
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/4966#discussion_r149340569
--- Diff: docs/dev/stream/state/queryable_state.md ---
@@ -32,38 +32,67 @@ under the License.
likely that there will be breaking API changes on the client side in the upcoming Flink versions.
</div>
-In a nutshell, this feature allows users to query Flink's managed partitioned state
-(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) from outside of
-Flink. For some scenarios, queryable state thus eliminates the need for distributed
-operations/transactions with external systems such as key-value stores which are often the
-bottleneck in practice.
+In a nutshell, this feature exposes Flink's managed keyed (partitioned) state
+(see [Working with State]({{ site.baseurl }}/dev/stream/state/state.html)) to the outside world and
+allows the user to query a job's state from outside Flink. For some scenarios, queryable state
+eliminates the need for distributed operations/transactions with external systems such as key-value
+stores which are often the bottleneck in practice. In addition, this feature may be particularly
+useful for debugging purposes.
<div class="alert alert-warning">
- <strong>Attention:</strong> Queryable state accesses keyed state from a concurrent thread rather
- than synchronizing with the operator and potentially blocking its operation. Since any state
- backend using Java heap space, e.g. MemoryStateBackend or
- FsStateBackend, does not work with copies when retrieving values but instead directly
- references the stored values, read-modify-write patterns are unsafe and may cause the
- queryable state server to fail due to concurrent modifications.
- The RocksDBStateBackend is safe from these issues.
+ <strong>Attention:</strong> When querying a state object, that object is accessed from a concurrent
+ thread without any synchronization or copying. This is a design choice, as any of the above would lead
+ to increased job latency, which we wanted to avoid. Since any state backend using Java heap space,
+ <i>e.g.</i> <code>MemoryStateBackend</code> or <code>FsStateBackend</code>, does not work
+ with copies when retrieving values but instead directly references the stored values, read-modify-write
+ patterns are unsafe and may cause the queryable state server to fail due to concurrent modifications.
+ The <code>RocksDBStateBackend</code> is safe from these issues.
</div>
+## Architecture
+
+Before showing how to use the Queryable State, it is useful to briefly describe the entities that compose it.
+The Queryable State consists of three main entities:
+
+ 1. the `QueryableStateClient`, which (potentially) runs outside the Flink cluster and submits the user queries,
+ 2. the `QueryableStateClientProxy`, which runs on each `TaskManager` (*i.e.* inside the Flink cluster) and is responsible
+ for receiving the client's queries, fetching the requested state on his behalf, and returning it to the client, and
+ 3. the `QueryableStateServer` which runs on each `TaskManager` and is responsible for serving the locally stored state.
+
+In a nutshell, the client will connect to one of the proxies and send a request for the state associated with a specific
+key, `k`. As stated in [Working with State]({{ site.baseurl }}/dev/stream/state/state.html), keyed state is organized in
+*Key Groups*, and each `TaskManager` is assigned a number of these key groups. To discover which `TaskManager` is
+responsible for the key group holding `k`, the proxy will ask the `JobManager`. Based on the answer, the proxy will
+then query the `QueryableStateServer` running on that `TaskManager` for the state associated with `k`, and forward the
+response back to the client.
+
+## Activating Queryable State
+
+To enable queryable state on your Flink cluster, you just have to copy the
--- End diff --
You mean make it: "To enable **the** queryable state on your Flink cluster..."?
---
[GitHub] flink pull request #4966: [FLINK-7822][FLINK-7823] Adds documentation and fi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4966
---