You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tz...@apache.org on 2020/03/26 14:21:15 UTC
[flink-statefun] 03/14: [FLINK-16758][docs] Port getting started
material
This is an automated email from the ASF dual-hosted git repository.
tzulitai pushed a commit to branch release-2.0
in repository https://gitbox.apache.org/repos/asf/flink-statefun.git
commit 053aa9911b4a6e60887982ef44607fc0b0db8a22
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Tue Mar 24 10:58:30 2020 -0500
[FLINK-16758][docs] Port getting started material
---
docs/fig/greeter-function.gif | Bin 0 -> 63845 bytes
docs/fig/stateful_functions.png | Bin 0 -> 66340 bytes
docs/getting-started/index.md | 28 +++++
docs/getting-started/project-setup.md | 86 +++++++++++++++
docs/getting-started/walkthrough.md | 200 ++++++++++++++++++++++++++++++++++
docs/index.md | 57 +++-------
6 files changed, 332 insertions(+), 39 deletions(-)
diff --git a/docs/fig/greeter-function.gif b/docs/fig/greeter-function.gif
new file mode 100644
index 0000000..c7055ee
Binary files /dev/null and b/docs/fig/greeter-function.gif differ
diff --git a/docs/fig/stateful_functions.png b/docs/fig/stateful_functions.png
new file mode 100755
index 0000000..85b4f0d
Binary files /dev/null and b/docs/fig/stateful_functions.png differ
diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md
new file mode 100644
index 0000000..6ad8ec4
--- /dev/null
+++ b/docs/getting-started/index.md
@@ -0,0 +1,28 @@
+---
+title: "Getting Started"
+nav-id: getting-started
+nav-title: '<i class="fa fa-rocket title appetizer" aria-hidden="true"></i> Getting Started'
+nav-parent_id: root
+section-break: true
+nav-show_overview: false
+nav-pos: 1
+always-expand: true
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
diff --git a/docs/getting-started/project-setup.md b/docs/getting-started/project-setup.md
new file mode 100644
index 0000000..b670de1
--- /dev/null
+++ b/docs/getting-started/project-setup.md
@@ -0,0 +1,86 @@
+---
+title: "Project Setup"
+nav-id: projectsetup
+nav-title: 'Project Setup'
+nav-parent_id: getting-started
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Dependency
+
+You can quickly get started building a Stateful Functions applications by adding the ``statefun-sdk`` to an existing project or using the provided maven archetype.
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>statefun-sdk</artifactId>
+ <version>{{ site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+## Maven Archetype
+
+{% highlight bash %}
+$ mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=statefun-quickstart \
+ -DarchetypeVersion={{ site.version }}
+{% endhighlight %}
+
+This allows you to name your newly created project.
+It will interactively ask you for the groupId, artifactId, and package name.
+There will be a new directory with the same name as your artifact id.
+
+{% highlight bash %}
+$ tree statefun-quickstart/
+ statefun-quickstart/
+ ├── Dockerfile
+ ├── pom.xml
+ └── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── apache
+ | └── flink
+ │ └── statefun
+ │ └── Module.java
+ └── resources
+ └── META-INF
+ └── services
+ └── org.apache.flink.statefun.sdk.spi.StatefulFunctionModule
+{% endhighlight %}
+
+The project contains four files:
+
+* ``pom.xml``: A pom file with the basic dependencies to start building a Stateful Functions application.
+* ``Module``: The entry point for the application.
+* ``org.apache.flink.statefun.sdk.spi.StatefulFunctionModule``: A service entry for the runtime to find the module.
+* ``Dockerfile``: A Dockerfile to quickly build a Stateful Functions image ready to deploy.
+
+We recommend you import this project into your IDE to develop and test it.
+IntelliJ IDEA supports Maven projects out of the box.
+If you use Eclipse, the m2e plugin allows to import Maven projects.
+Some Eclipse bundles include that plugin by default, others require you to install it manually.
+
+## Build Project
+
+If you want to build/package your project, go to your project directory and run the ``mvn clean package`` command.
+You will find a JAR file that contains your application, plus any libraries that you may have added as dependencies to the application: ``target/<artifact-id>-<version>.jar``.
diff --git a/docs/getting-started/walkthrough.md b/docs/getting-started/walkthrough.md
new file mode 100644
index 0000000..862c53c
--- /dev/null
+++ b/docs/getting-started/walkthrough.md
@@ -0,0 +1,200 @@
+---
+title: "Walkthrough"
+nav-id: walkthrough
+nav-title: 'Walkthrough'
+nav-parent_id: getting-started
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Like all great introductions in software, this walkthrough will start at the beginning: saying hello.
+The application will run a simple function that accepts a request and responds with a greeting.
+It will not attempt to cover all the complexities of application development, but instead focus on building a stateful function — which is where you will implement your business logic.
+
+* This will be replaced by the TOC
+{:toc}
+
+## A Basic Hello
+
+Greeting actions are triggered by consuming, routing and passing messages that are defined using ProtoBuf.
+
+{% highlight proto %}
+syntax = "proto3";
+
+message GreetRequest {
+ string who = 1;
+}
+
+message GreetResponse {
+ string who = 1;
+ string greeting = 2;
+}
+{% endhighlight %}
+
+Under the hood, messages are processed using :ref:`stateful functions <java>`, by definition any class that implements the ``StatefulFunction`` interface.
+
+{% highlight java %}
+package org.apache.flink.statefun.examples.greeter;
+
+import org.apache.flink.statefun.sdk.Context;
+import org.apache.flink.statefun.sdk.StatefulFunction;
+
+public final class GreetFunction implements StatefulFunction {
+
+ @Override
+ public void invoke(Context context, Object input) {
+ GreetRequest greetMessage = (GreetRequest) input;
+
+ GreetResponse response = GreetResponse.newBuilder()
+ .setWho(greetMessage.getWho())
+ .setGreeting("Hello " + greetMessage.getWho())
+ .build();
+
+ context.send(GreetingConstants.GREETING_EGRESS_ID, response);
+ }
+}
+{% endhighlight %}
+
+
+This function takes in a request and sends a response to an external system (or :ref:`egress <egress>`).
+While this is nice, it does not show off the real power of stateful functions: handling state.
+
+## A Stateful Hello
+
+Suppose you want to generate a personalized response for each user depending on how many times they have sent a request.
+
+{% highlight java %}
+private static String greetText(String name, int seen) {
+ switch (seen) {
+ case 0:
+ return String.format("Hello %s !", name);
+ case 1:
+ return String.format("Hello again %s !", name);
+ case 2:
+ return String.format("Third times the charm! %s!", name);
+ case 3:
+ return String.format("Happy to see you once again %s !", name);
+ default:
+ return String.format("Hello at the %d-th time %s", seen + 1, name);
+}
+{% endhighlight %}
+
+## Routing Messages
+
+To send a user a personalized greeting, the system needs to keep track of how many times it has seen each user so far.
+Speaking in general terms, the simplest solution would be to create one function for every user and independently track the number of times they have been seen. Using most frameworks, this would be prohibitively expensive.
+However, stateful functions are virtual and do not consume any CPU or memory when not actively being invoked.
+That means your application can create as many functions as necessary — in this case, users — without worrying about resource consumption.
+
+Whenever data is consumed from an external system (or :ref:`ingress <ingress>`), it is routed to a specific function based on a given function type and identifier.
+The function type represents the Class of function to be invoked, such as the Greeter function, while the identifier (``GreetRequest#getWho``) scopes the call to a specific virtual instance based on some key.
+
+{% highlight java %}
+package org.apache.flink.statefun.examples.greeter;
+
+import org.apache.flink.statefun.examples.kafka.generated.GreetRequest;
+import org.apache.flink.statefun.sdk.io.Router;
+
+final class GreetRouter implements Router<GreetRequest> {
+
+ @Override
+ public void route(GreetRequest message, Downstream<GreetRequest> downstream) {
+ downstream.forward(GreetingConstants.GREETER_FUNCTION_TYPE, message.getWho(), message);
+ }
+}
+{% endhighlight %}
+
+So, if a message for a user named John comes in, it will be shipped to John’s dedicated Greeter function.
+In case there is a following message for a user named Jane, a new instance of the Greeter function will be spawned.
+
+## Persistence
+
+:ref:`Persisted value <persisted-value>` is a special data type that enables stateful functions to maintain fault-tolerant state scoped to their identifiers, so that each instance of a function can track state independently.
+To “remember” information across multiple greeting messages, you then need to associate a persisted value field (``count``) to the Greet function. For each user, functions can now track how many times they have been seen.
+
+{% highlight java %}
+package org.apache.flink.statefun.examples.greeter;
+
+import org.apache.flink.statefun.sdk.Context;
+import org.apache.flink.statefun.sdk.StatefulFunction;
+import org.apache.flink.statefun.sdk.annotations.Persisted;
+import org.apache.flink.statefun.sdk.state.PersistedValue;
+
+public final class GreetFunction implements StatefulFunction {
+
+ @Persisted
+ private final PersistedValue<Integer> count = PersistedValue.of("count", Integer.class);
+
+ @Override
+ public void invoke(Context context, Object input) {
+ GreetRequest greetMessage = (GreetRequest) input;
+
+ GreetResponse response = computePersonalizedGreeting(greetMessage);
+
+ context.send(GreetingConstants.GREETING_EGRESS_ID, response);
+ }
+
+ private GreetResponse computePersonalizedGreeting(GreetRequest greetMessage) {
+ final String name = greetMessage.getWho();
+ final int seen = count.getOrDefault(0);
+ count.set(seen + 1);
+
+ String greeting = greetText(name, seen);
+
+ return GreetResponse.newBuilder()
+ .setWho(name)
+ .setGreeting(greeting)
+ .build();
+ }
+}
+{% endhighlight %}
+
+Each time a message is processed, the function computes a personalized message for that user.
+It reads and updates the number of times that user has been seen and sends a greeting to the egress.
+
+You can check the full code for the application described in this walkthrough `here <{examples}/statefun-greeter-example>`.
+In particular, take a look at the module ``GreetingModule``, which is the main entry point for the full application, to see how everything gets tied together.
+You can run this example locally using the provided Docker setup.
+
+{% highlight bash %}
+$ docker-compose build
+$ docker-compose up
+{% endhighlight %}
+
+Then, send some messages to the topic "names", and observe what comes out of "greetings".
+
+{% highlight bash %}
+$ docker-compose exec kafka-broker kafka-console-producer.sh \
+ --broker-list localhost:9092 \
+ --topic names
+
+$ docker-compose exec kafka-broker kafka-console-consumer.sh \
+ --bootstrap-server localhost:9092 \
+ --topic greetings
+{% endhighlight %}
+
+<p class="text-center">
+ <img width="80%" src="{{ site.baseurl }}/fig/greeter-function.gif"/>
+</p>
+
+## Want To Go Further?
+
+This Greeter never forgets a user.
+Try and modify the function so that it will reset the ``count`` for any user that spends more than 60 seconds without interacting with the system.
diff --git a/docs/index.md b/docs/index.md
index 9837395..6ca7616 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,5 +1,5 @@
---
-title: "Apache Flink Documentation"
+title: "Stateful Functions - An Apache Flink® Project"
nav-pos: 0
nav-title: '<i class="fa fa-home title" aria-hidden="true"></i> Home'
nav-parent_id: root
@@ -23,52 +23,31 @@ specific language governing permissions and limitations
under the License.
-->
-This documentation is for Apache Flink version {{ site.version_title }}. These pages were built at: {% build_time %}.
+## A Framework for Stateful Distributed Applications
-Apache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams. Flink builds batch processing on top of the streaming engine, overlaying native iteration support, managed memory, and program optimization.
+**Stateful Functions** is an open source framework that reduces the complexity of building and orchestrating distributed stateful applications at scale.
+It brings together the benefits of stream processing with Apache Flink® and Function-as-a-Service (FaaS) to provide a powerful abstraction for the next generation of event-driven architectures.
-## First Steps
+<p class="text-center">
+ <img width="80%" src="{{ site.baseurl }}/fig/stateful_functions.png"/>
+</p>
-* **Code Walkthroughs**: Follow step-by-step guides and implement a simple application or query in one of Flink's APIs.
- * [Implement a DataStream application](./getting-started/walkthroughs/datastream_api.html)
- * [Write a Table API query](./getting-started/walkthroughs/table_api.html)
+There are many ways to get started with Stateful Functions.
+Which one is the best for you depends on your goals and prior experience.
+Whether you prefer a more theoretical or a practical approach, we hope you’ll find this section helpful.
-* **Docker Playgrounds**: Set up a sandboxed Flink environment in just a few minutes to explore and play with Flink.
- * [Run and manage Flink streaming applications](./getting-started/docker-playgrounds/flink-operations-playground.html)
+## Learn By Doing
-* **Concepts**: Learn about Flink's basic concepts to better understand the documentation.
- * [Dataflow Programming Model](concepts/programming-model.html)
- * [Distributed Runtime](concepts/runtime.html)
- * [Glossary](concepts/glossary.html)
+If you prefer to learn by doing, start with our code :ref:`walkthrough <walkthrough>`.
+It provides a step by step introduction to the API and guides you through real applications.
-## API References
+## Learn Concepts Step By Step
-The API references list and explain all features of Flink's APIs.
+If you prefer to learn concepts step by step, start with our guide to :ref:`main concepts <concepts>`.
+It will walk you through all the API's and concepts to build advanced stateful systems.
-* [Basic API Concepts](dev/api_concepts.html)
-* [DataStream API](dev/datastream_api.html)
-* [DataSet API](dev/batch/index.html)
-* [Table API & SQL](dev/table/index.html)
-## Deployment
+## Start A New Project
-Before putting your Flink job into production, read the [Production Readiness Checklist](ops/production_ready.html).
+The :ref:`project setup <project_setup>` instructions show you how to create a project for a new Stateful Functions application in just a few steps.
-## Release Notes
-
-Release notes cover important changes between Flink versions. Please carefully read these notes if you plan to upgrade your Flink setup to a later version.
-
-* [Release notes for Flink 1.10](release-notes/flink-1.10.html).
-* [Release notes for Flink 1.9](release-notes/flink-1.9.html).
-* [Release notes for Flink 1.8](release-notes/flink-1.8.html).
-* [Release notes for Flink 1.7](release-notes/flink-1.7.html).
-* [Release notes for Flink 1.6](release-notes/flink-1.6.html).
-* [Release notes for Flink 1.5](release-notes/flink-1.5.html).
-
-## External Resources
-
-- **Flink Forward**: Talks from past conferences are available at the [Flink Forward](http://flink-forward.org/) website and on [YouTube](https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA). [Robust Stream Processing with Apache Flink](http://2016.flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/) is a good place to start.
-
-- **Training**: The [training materials](https://training.ververica.com/) from Ververica include slides, exercises, and sample solutions.
-
-- **Blogs**: The [Apache Flink](https://flink.apache.org/blog/) and [Ververica](https://www.ververica.com/blog) blogs publish frequent, in-depth technical articles about Flink.