You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tz...@apache.org on 2020/03/26 14:00:25 UTC
[flink-statefun] 07/13: [FLINK-16758][docs] Port Operations content
This is an automated email from the ASF dual-hosted git repository.
tzulitai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-statefun.git
commit ad8251e9746f6d7a9bc3b2927f5b94376be479f7
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Tue Mar 24 18:02:42 2020 -0500
[FLINK-16758][docs] Port Operations content
---
docs/deployment-and-operations/configurations.md | 65 +++++++++++++++
docs/deployment-and-operations/index.md | 31 +++++++
docs/deployment-and-operations/metrics.md | 100 +++++++++++++++++++++++
docs/deployment-and-operations/packaging.md | 71 ++++++++++++++++
docs/getting-started/walkthrough.md | 8 +-
5 files changed, 271 insertions(+), 4 deletions(-)
diff --git a/docs/deployment-and-operations/configurations.md b/docs/deployment-and-operations/configurations.md
new file mode 100644
index 0000000..d09ed25
--- /dev/null
+++ b/docs/deployment-and-operations/configurations.md
@@ -0,0 +1,65 @@
+---
+title: Configurations
+nav-id: configurations
+nav-pos: 2
+nav-title: Configurations
+nav-parent_id: deployment-and-ops
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Stateful Functions includes a small number of SDK specific configurations.
+These may be set through your job's ``flink-conf.yaml``.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>statefun.module.global-config.<KEY></h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Adds the given key/value pair to the Stateful Functions global configuration.</td>
+ </tr>
+ <tr>
+ <td><h5>statefun.message.serializer</h5></td>
+ <td style="word-wrap: break-word;">WITH_PROTOBUF_PAYLOADS</td>
+ <td>Message Serializer</td>
+ <td>The serializer to use for on the wire messages. Options are WITH_PROTOBUF_PAYLOADS, WITH_KRYO_PAYLOADS, WITH_RAW_PAYLOADS.</td>
+ </tr>
+ <tr>
+ <td><h5>statefun.flink-job-name</h5></td>
+ <td style="word-wrap: break-word;">StatefulFunctions</td>
+ <td>String</td>
+ <td>The name to display in the Flink-UI.</td>
+ </tr>
+ <tr>
+ <td><h5>statefun.feedback.memory.size</h5></td>
+ <td style="word-wrap: break-word;">32 MB</td>
+ <td>Memory</td>
+ <td>TThe number of bytes to use for in memory buffering of the feedback channel, before spilling to disk.</td>
+ </tr>
+ </tbody>
+</table>
\ No newline at end of file
diff --git a/docs/deployment-and-operations/index.md b/docs/deployment-and-operations/index.md
new file mode 100644
index 0000000..d922a85
--- /dev/null
+++ b/docs/deployment-and-operations/index.md
@@ -0,0 +1,31 @@
+---
+title: Deployment and Operations
+nav-id: deployment-and-ops
+nav-pos: 5
+nav-title: 'Deployment and Operations'
+nav-parent_id: root
+nav-show_overview: true
+permalink: /deployment-and-operations/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Stateful Functions is a framework built on top of the Apache Flink runtime, which means it inherits Flink's deployment and operations model, and there are no new concepts you need to learn.
+Read through the official [Apache Flink documentation](https://ci.apache.org/projects/flink/flink-docs-stable/) to learn how to run and maintain an application in production.
+The following pages outline Stateful Functions' specific configurations and metrics.
\ No newline at end of file
diff --git a/docs/deployment-and-operations/metrics.md b/docs/deployment-and-operations/metrics.md
new file mode 100644
index 0000000..e566ad3
--- /dev/null
+++ b/docs/deployment-and-operations/metrics.md
@@ -0,0 +1,100 @@
+---
+title: Metrics
+nav-id: metrics
+nav-pos: 3
+nav-title: Metrics
+nav-parent_id: deployment-and-ops
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Stateful Functions includes a number of SDK specific metrics.
+Along with the [standard metric scopes](https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#system-scope), Stateful Functions supports ``Function Scope`` which one level below operator scope.
+
+``metrics.scope.function``
+* Default: <host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>.<function_namespace>.<function_name>
+* Applied to all metrics that were scoped to a function.
+
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Metrics</th>
+ <th class="text-left" style="width: 15%">Scope</th>
+ <th class="text-left" style="width: 15%">Description</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>in</h5></td>
+ <td>Function</td>
+ <td>The number of incoming messages.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td><h5>inRate</h5></td>
+ <td>Function</td>
+ <td>The average number of incoming messages per second.</td>
+ <td>Meter</td>
+ </tr>
+ <tr>
+ <td><h5>out-local</h5></td>
+ <td>Function</td>
+ <td>The number of messages sent to a function on the same task slot.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td><h5>out-localRate</h5></td>
+ <td>Function</td>
+ <td>The average number of messages sent to a function on the same task slot per second.</td>
+ <td>Meter</td>
+ </tr>
+ <tr>
+ <td><h5>out-remote</h5></td>
+ <td>Function</td>
+ <td>The number of messages sent to a function on a different task slot.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td><h5>out-remoteRate</h5></td>
+ <td>Function</td>
+ <td>The average number of messages sent to a function on a different task slot per second.</td>
+ <td>Meter</td>
+ </tr>
+ <tr>
+ <td><h5>out-egress</h5></td>
+ <td>Function</td>
+ <td>The number of messages sent to an egress.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td><h5>feedback.produced</h5></td>
+ <td>Operator</td>
+ <td>The number of messages read from the feedback channel.</td>
+ <td>Meter</td>
+ </tr>
+ <tr>
+ <td><h5>feedback.producedRate</h5></td>
+ <td>Operator</td>
+ <td>The average number of messages read from the feedback channel per second.</td>
+ <td>Meter</td>
+ </tr>
+ </tbody>
+</table>
\ No newline at end of file
diff --git a/docs/deployment-and-operations/packaging.md b/docs/deployment-and-operations/packaging.md
new file mode 100644
index 0000000..5507666
--- /dev/null
+++ b/docs/deployment-and-operations/packaging.md
@@ -0,0 +1,71 @@
+---
+title: Packaging For Deployment
+nav-id: packaging
+nav-pos: 1
+nav-title: Packaging For Deployment
+nav-parent_id: deployment-and-ops
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Stateful Functions applications can be packaged as either standalone applications or Flink jobs that can be submitted to a cluster.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Images
+
+The recommended deployment mode for Stateful Functions applications is to build a Docker image.
+This way, user code does not need to package any Apache Flink components.
+The provided base image allows teams to package their applications with all the necessary runtime dependencies quickly.
+
+Below is an example Dockerfile for building a Stateful Functions image with both an [embedded module]({{ site.baseurl }}/sdk/modules.html#embedded-module) and a [remote module]({{ site.baseurl }}/sdk/modules.html#remote-module) for an application called ``statefun-example``.
+
+{% highlight dockerfile %}
+FROM statefun
+
+RUN mkdir -p /opt/statefun/modules/statefun-example
+RUN mkdir -p /opt/statefun/modules/remote
+
+COPY target/statefun-example*jar /opt/statefun/modules/statefun-example/
+COPY module.yaml /opt/statefun/modules/remote/module.yaml
+{% endhighlight %}
+
+## Flink Jar
+
+If you prefer to package your job to submit to an existing Flink cluster, simply include ``statefun-flink-distribution`` as a dependency to your application.
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>statefun-flink-distribution</artifactId>
+ <version>{{ site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+It includes all of Stateful Functions' runtime dependencies and configures the application's main entry-point.
+You do not need to take any action beyond adding the dependency to your POM.
+
+<div class="alert alert-info">
+ <strong>Attention:</strong> The distribution must be bundled in your application fat JAR so that it is on Flink's <a href="https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order">user code class loader</a>
+</div>
+
+{% highlight bash %}
+./bin/flink run ./statefun-example.jar
+{% endhighlight %}
diff --git a/docs/getting-started/walkthrough.md b/docs/getting-started/walkthrough.md
index 862c53c..4c895bd 100644
--- a/docs/getting-started/walkthrough.md
+++ b/docs/getting-started/walkthrough.md
@@ -48,7 +48,7 @@ message GreetResponse {
}
{% endhighlight %}
-Under the hood, messages are processed using :ref:`stateful functions <java>`, by definition any class that implements the ``StatefulFunction`` interface.
+Under the hood, messages are processed using [stateful functions]({{ site.baseurl }}/sdk/java.html), by definition any class that implements the ``StatefulFunction`` interface.
{% highlight java %}
package org.apache.flink.statefun.examples.greeter;
@@ -73,7 +73,7 @@ public final class GreetFunction implements StatefulFunction {
{% endhighlight %}
-This function takes in a request and sends a response to an external system (or :ref:`egress <egress>`).
+This function takes in a request and sends a response to an external system (or [egress]({{ site.baseurl }}/io-module/index.html#egress)).
While this is nice, it does not show off the real power of stateful functions: handling state.
## A Stateful Hello
@@ -103,7 +103,7 @@ Speaking in general terms, the simplest solution would be to create one function
However, stateful functions are virtual and do not consume any CPU or memory when not actively being invoked.
That means your application can create as many functions as necessary — in this case, users — without worrying about resource consumption.
-Whenever data is consumed from an external system (or :ref:`ingress <ingress>`), it is routed to a specific function based on a given function type and identifier.
+Whenever data is consumed from an external system (or [ingress]({{ site.baseurl }}/io-module/index.html#ingress)), it is routed to a specific function based on a given function type and identifier.
The function type represents the Class of function to be invoked, such as the Greeter function, while the identifier (``GreetRequest#getWho``) scopes the call to a specific virtual instance based on some key.
{% highlight java %}
@@ -126,7 +126,7 @@ In case there is a following message for a user named Jane, a new instance of th
## Persistence
-:ref:`Persisted value <persisted-value>` is a special data type that enables stateful functions to maintain fault-tolerant state scoped to their identifiers, so that each instance of a function can track state independently.
+[Persisted value]({{ site.baseurl }}/sdk/java.html#persistence) is a special data type that enables stateful functions to maintain fault-tolerant state scoped to their identifiers, so that each instance of a function can track state independently.
To “remember” information across multiple greeting messages, you then need to associate a persisted value field (``count``) to the Greet function. For each user, functions can now track how many times they have been seen.
{% highlight java %}