You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tz...@apache.org on 2020/03/12 02:45:50 UTC
[flink-statefun] 05/08: [FLINK-16515][docs] Consolidate Java SDK
documentation
This is an automated email from the ASF dual-hosted git repository.
tzulitai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-statefun.git
commit 25b94f9738c6d267d166734cd0fe065ba00a471e
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Wed Mar 4 19:05:54 2020 -0600
[FLINK-16515][docs] Consolidate Java SDK documentation
---
statefun-docs/docs/getting_started/walkthrough.rst | 4 +-
statefun-docs/docs/sdk/index.rst | 6 +-
.../sdk/{stateful_functions.rst => java/index.rst} | 123 ++++++++++++++-------
.../docs/sdk/{ => java}/match_functions.rst | 6 +-
statefun-docs/docs/sdk/persistence.rst | 81 --------------
.../flink/statefun/docs/BasicFunctionModule.java | 2 +-
.../org/apache/flink/statefun/docs/FnCaller.java | 4 +-
.../docs/{FnCaller.java => Identifiers.java} | 14 +--
8 files changed, 94 insertions(+), 146 deletions(-)
diff --git a/statefun-docs/docs/getting_started/walkthrough.rst b/statefun-docs/docs/getting_started/walkthrough.rst
index 3d7f7a1..0604eeb 100644
--- a/statefun-docs/docs/getting_started/walkthrough.rst
+++ b/statefun-docs/docs/getting_started/walkthrough.rst
@@ -44,7 +44,7 @@ Greeting actions are triggered by consuming, routing and passing messages that a
}
-Under the hood, messages are processed using :ref:`stateful functions <statefun>`, by definition any class that implements the ``StatefulFunction`` interface.
+Under the hood, messages are processed using :ref:`stateful functions <java>`, by definition any class that implements the ``StatefulFunction`` interface.
.. code-block:: java
@@ -168,7 +168,7 @@ Each time a message is processed, the function computes a personalized message f
It reads and updates the number of times that user has been seen and sends a greeting to the egress.
You can check the full code for the application described in this walkthrough `here <{examples}/statefun-greeter-example>`_.
-In particular, take a look at the :ref:`module <module>` GreetingModule, which is the main entry point for the full application, to see how everything gets tied together.
+In particular, take a look at the module GreetingModule, which is the main entry point for the full application, to see how everything gets tied together.
You can run this example locally using the provided Docker setup.
.. code-block:: bash
diff --git a/statefun-docs/docs/sdk/index.rst b/statefun-docs/docs/sdk/index.rst
index 3b18316..7080487 100644
--- a/statefun-docs/docs/sdk/index.rst
+++ b/statefun-docs/docs/sdk/index.rst
@@ -21,10 +21,8 @@ SDK
.. toctree::
:hidden:
-
- stateful_functions
- match_functions
- persistence
+
+ java/index
Stateful Functions applications are a collection of virtual stateful functions that can send arbitrary messages between each other and external systems.
The execution can happen on a local JVM, or clusters of many machines.
diff --git a/statefun-docs/docs/sdk/stateful_functions.rst b/statefun-docs/docs/sdk/java/index.rst
similarity index 59%
rename from statefun-docs/docs/sdk/stateful_functions.rst
rename to statefun-docs/docs/sdk/java/index.rst
index 8682f25..8dd14cf 100644
--- a/statefun-docs/docs/sdk/stateful_functions.rst
+++ b/statefun-docs/docs/sdk/java/index.rst
@@ -13,16 +13,32 @@
specific language governing permissions and limitations
under the License.
-.. _statefun:
+.. _java:
-##################
-Stateful Functions
-##################
+####
+Java
+####
+
+.. toctree::
+ :hidden:
+
+ match_functions
Stateful functions are the building blocks of applications; they are atomic units of isolation, distribution, and persistence.
As objects, they encapsulate the state of a single entity (e.g., a specific user, device, or session) and encode its behavior.
Stateful functions can interact with each other, and external systems, through message passing.
+To get started, add the Java SDK as a dependency to your application.
+
+.. code-block:: xml
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>statefun-sdk</artifactId>
+ <version>{version}</version>
+ <scope>provided</scope>
+ </dependency>
+
.. contents:: :local:
Defining A Stateful Function
@@ -30,7 +46,7 @@ Defining A Stateful Function
A stateful function is any class that implements the ``StatefulFunction`` interface. The following is an example of a simple hello world function.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/FnHelloWorld.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/FnHelloWorld.java
:language: java
:lines: 18-
@@ -40,44 +56,24 @@ Input's are untyped and passed through the system as a ``java.lang.Object`` so o
The ``Context`` provides metadata about the current message and function, and is how you can call other functions or external systems.
Functions are invoked based on a function type and unique identifier.
-Function Type's and Identifiers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-In a local environment, the address of an object is the same as a reference to it.
-But in a distributed system, objects may be spread across multiple machines and may or may not be active at any given moment.
-
-In Stateful Functions, function types and identifiers are used to reference specific stateful functions in the system.
-A function type is similar to a class in an object-oriented language; it declares what sort of function the address references.
-The id is a primary key and scopes the function call to a specific instance of the function type.
+Function Types and Messaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Suppose a Stateful Functions application was tracking metadata about each user account on a website.
-The system would contain a user stateful function that accepts and responds to inputs about users and tracks relevant information.
-Stateful Functions will create one virtual instance of this stateful function for every user.
-Other functions can call the function for any particular user by the user function type and using the current user id as the instance identifier.
+In Java, function types are defined as a stringly typed reference containing a namespace and name.
+The type is bound to the implementing class in the :ref:`module <java_module>` definition.
+Below is an example function type for the hello world function.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/FnUser.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/Identifiers.java
:language: java
- :emphasize-lines: 10
:lines: 18-
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/FnCaller.java
+This type can then be referenced from other functions to create an address and message a particular instance.
+
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/FnCaller.java
:language: java
- :emphasize-lines: 14
+ :emphasize-lines: 12
:lines: 18-
-Virtual Functions
-^^^^^^^^^^^^^^^^^
-
-Functions are virtual, which means the system can support an infinite number of active functions while only requiring a static number of physical objects on the JVM heap.
-Any function can call any other without ever triggering an allocation.
-The system will make it appear as if functions are always available in-memory.
-
-Stateful Functions applications deploy on Apache Flink's horizontally parallel runtime.
-If the user function, seen above, is run on a Flink cluster with a parallelism of 10, then only ten objects will ever be allocated.
-Even if the application creates a billion user functions for a billion different users, memory usage will be stable.
-Those billion virtual functions will be evenly partitioned and run by the ten underlying objects.
-New object creation only occurs the first time a function of that type, regardless of id, is needed.
-
Sending Delayed Messages
^^^^^^^^^^^^^^^^^^^^^^^^
@@ -85,7 +81,7 @@ Functions are able to send messages on a delay so that they will arrive after so
Functions may even send themselves delayed messages that can serve as a callback.
The delayed message is non-blocking so functions will continue to process records between the time a delayed message is sent and received.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/delay/FnDelayedMessage.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/delay/FnDelayedMessage.java
:language: java
:lines: 18-
@@ -114,16 +110,59 @@ Unknown
The stateful function was restarted, possibly on a different machine, before the ``CompletableFuture`` was completed, therefore it is unknown what is the status of the asynchronous operation.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/async/EnrichmentFunction.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/async/EnrichmentFunction.java
+ :language: java
+ :lines: 18-
+
+.. _persisted-value:
+
+Persistence
+^^^^^^^^^^^
+Stateful Functions treats state as a first class citizen and so all stateful functions can easily define state that is automatically made fault tolerant by the runtime.
+All stateful functions may contain state by merely defining one or more persisted fields.
+
+The simplest way to get started is with a ``PersistedValue``, which is defined by its name and the class of the type that it stores.
+The data is always scoped to a specific function type and identifier.
+Below is a stateful function that greets users based on the number of times they have been seen.
+
+.. warning::
+
+ All ``PersistedValue``, ``PersistedTable``, and ``PersistedAppendingBuffer`` fields must be marked with an ``@Persisted`` annotation or they will not be made fault tolerant by the runtime.
+
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/FnUserGreeter.java
:language: java
:lines: 18-
+Persisted value comes with the right primitive methods to build powerful stateful applications.
+Calling ``PersistedValue#get`` will return the current value of an object stored in state, or ``null`` if nothing is set.
+Conversely, ``PersistedValue#set`` will update the value in state and ``PersistedValue#clear`` will delete the value from state.
+
+Collection Types
+================
+
+Along with ``PersistedValue``, the Java SDK supports two persisted collection types.
+``PersistedTable`` is a collection of keys and values, and ``PersistedAppendingBuffer`` is an append-only buffer.
+
+These types are functionally equivalent to ``PersistedValue<Map>`` and ``PersistedValue<Collection>`` respectively but may provide better performance in some situations.
+
+.. code-block:: java
+
+ @Persisted
+ PersistedTable<String, Integer> table =
+ PersistedTable.of("my-table", String.class, Integer.class);
+
+ @Persisted
+ PersistedAppendingBuffer<Integer> buffer =
+ PersistedAppendingBuffer.of("my-buffer", Integer.class);
+
+
Function Providers and Dependency Injection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
Stateful functions are created across a distributed cluster of nodes.
``StatefulFunctionProvider`` is a factory class for creating a new instance of a stateful function the first time it is activated.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/CustomProvider.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/CustomProvider.java
:language: java
:lines: 18-
@@ -132,11 +171,11 @@ If a stateful function requires custom configurations, they can be defined insid
This is also where shared physical resources, such as a database connection, can be created that are used by any number of virtual functions.
Now, tests can quickly provide mock, or test dependencies, without the need for complex dependency injection frameworks.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/FunctionTest.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/FunctionTest.java
:language: java
:lines: 18-
-.. _module:
+.. _java_module:
Stateful Function Modules
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -145,7 +184,7 @@ Modules define a Stateful Functions application's top-level entry point and are
They offer a single configuration method where stateful functions are bound to the system.
It also provides runtime configurations through the ``globalConfguration`` which is the union of all configurations in the applications ``flink-conf.yaml`` under the prefix ``statefun.module.global-config`` and any command line arguments passed in the form ``--key value``.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
:language: java
:lines: 18-
@@ -154,4 +193,4 @@ This means that every JAR should contain a file ``org.apache.flink.statefun.sdk.
.. code-block:: yaml
- BasicFunctionModule
+ org.apache.flink.statefun.docs.BasicFunctionModule
diff --git a/statefun-docs/docs/sdk/match_functions.rst b/statefun-docs/docs/sdk/java/match_functions.rst
similarity index 91%
rename from statefun-docs/docs/sdk/match_functions.rst
rename to statefun-docs/docs/sdk/java/match_functions.rst
index 46f3ae9..f30e372 100644
--- a/statefun-docs/docs/sdk/match_functions.rst
+++ b/statefun-docs/docs/sdk/java/match_functions.rst
@@ -27,7 +27,7 @@ Common Patterns
Imagine a greeter function that wants to print specialized greeters depending on the type of input.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/match/FnUserGreeter.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/match/FnUserGreeter.java
:language: java
:lines: 18-
@@ -42,7 +42,7 @@ Stateful match functions are an opinionated variant of stateful functions for pr
Developers outline expected types, optional predicates, and well-typed business logic and let the system dispatch each input to the correct action.
Variants are bound inside a ``configure`` method that is executed once the first time an instance is loaded.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/match/FnMatchGreeter.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/match/FnMatchGreeter.java
:language: java
:lines: 18-
@@ -53,7 +53,7 @@ Similar to the first example, match functions are partial by default and will th
They can be made complete by providing an ``otherwise`` clause that serves as a catch-all for unmatched input, think of it as a default clause in a Java switch statement.
The ``otherwise`` action takes its message as an untyped ``java.lang.Object``, allowing you to handle any unexpected messages.
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/match/FnMatchGreeterWithCatchAll.java
+.. literalinclude:: ../../../src/main/java/org/apache/flink/statefun/docs/match/FnMatchGreeterWithCatchAll.java
:language: java
:lines: 18-
:emphasize-lines: 15
diff --git a/statefun-docs/docs/sdk/persistence.rst b/statefun-docs/docs/sdk/persistence.rst
deleted file mode 100644
index bf86e4c..0000000
--- a/statefun-docs/docs/sdk/persistence.rst
+++ /dev/null
@@ -1,81 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-###########
-Persistence
-###########
-
-Stateful Functions treats state as a first class citizen and so all stateful functions can easily define state that is automatically made fault tolerant by the runtime.
-
-.. contents:: :local:
-
-.. _persisted-value:
-
-Defining a Persistent Values
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-All stateful functions may contain state by merely defining one or more ``PersistedValue`` fields.
-A ``PersistedValue`` is defined by its name and the class of the type that it stores.
-The data is always scoped to a specific function type and identifier.
-Below is a stateful function that greets users based on the number of times they have been seen.
-
-.. warning::
-
- All ``PersistedValue`` fields must be marked with an ``@Persisted`` annotation or they will not be made fault tolerant by the runtime.
-
-.. literalinclude:: ../../src/main/java/org/apache/flink/statefun/docs/FnUserGreeter.java
- :language: java
- :lines: 18-
-
-Persisted value comes with the right primitive methods to build powerful stateful applications.
-Calling ``PersistedValue#get`` will return the current value of an object stored in state, or ``null`` if nothing is set.
-Conversely, ``PersistedValue#set`` will update the value in state and ``PersistedValue#clear`` will delete the value from state.
-
-
-Supported Types
-^^^^^^^^^^^^^^^
-
-Stateful Functions applications are typically designed to run indefinitely or for long periods of time.
-As with all long-running services, the applications need to be updated to adapt to changing requirements.
-This goes the same for data schemas that the applications work against; they evolve along with the application.
-That is why the system limits the types that can be stored inside a ``PersistedValue`` to all Java primitives and complex types that support well defined schema migration semantics.
-
-.. note::
-
- Schema evolution is supported naturally with protobuf and json, and the project is working on connecting it to Flink’s schema evolution capabilities.
-
-POJO types
-""""""""""
-
-Stateful Functions recognizes data types as a POJO type if the following conditions are fulfilled:
-
-* The class is public and standalone (no non-static inner class)
-* The class has a public no-argument constructor
-* All non-static, non-transient fields in the class (and all superclasses) are either public (and non-final) or have a public getter- and a setter- method that follows the Java beans naming conventions for getters and setters.
-
-Apache Avro
-"""""""""""
-
-Stateful Functions can store any Apache Avro class and fully supports evolving schema of Avro type state, as long as the schema change is considered compatible by `Avro’s rules for schema resolution <https://avro.apache.org/docs/current/spec.html#Schema+Resolution>`_.
-
-Protocol Buffers
-""""""""""""""""
-
-Stateful Functions can store any `Protocol Buffer <https://developers.google.com/protocol-buffers/>`_ class and fully supports schema evolution as long as the schema change is considered compatible by ProtoBuff's rules for schema evolution.
-
-Json
-""""
-
-Stateful Functions can store any object that serializes as JSON.
diff --git a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
index ba71f72..b1eae5d 100644
--- a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
+++ b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/BasicFunctionModule.java
@@ -29,6 +29,6 @@ public class BasicFunctionModule implements StatefulFunctionModule {
// Stateful functions that do not require any configuration
// can declare their provider using java 8 lambda syntax
- binder.bindFunctionProvider(FnUser.TYPE, unused -> new FnUser());
+ binder.bindFunctionProvider(Identifiers.HELLO_TYPE, unused -> new FnHelloWorld());
}
}
diff --git a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java
index d5d513c..6326b66 100644
--- a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java
+++ b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java
@@ -24,10 +24,8 @@ import org.apache.flink.statefun.sdk.StatefulFunction;
/** A simple stateful function that sends a message to the user with id "user1" */
public class FnCaller implements StatefulFunction {
- public static final FunctionType TYPE = new FunctionType("example", "caller");
-
@Override
public void invoke(Context context, Object input) {
- context.send(FnUser.TYPE, "user1", new MyUserMessage());
+ context.send(Identifiers.HELLO_TYPE, "user1", new MyUserMessage());
}
}
diff --git a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/Identifiers.java
similarity index 66%
copy from statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java
copy to statefun-docs/src/main/java/org/apache/flink/statefun/docs/Identifiers.java
index d5d513c..b937150 100644
--- a/statefun-docs/src/main/java/org/apache/flink/statefun/docs/FnCaller.java
+++ b/statefun-docs/src/main/java/org/apache/flink/statefun/docs/Identifiers.java
@@ -17,17 +17,11 @@
*/
package org.apache.flink.statefun.docs;
-import org.apache.flink.statefun.sdk.Context;
import org.apache.flink.statefun.sdk.FunctionType;
-import org.apache.flink.statefun.sdk.StatefulFunction;
-/** A simple stateful function that sends a message to the user with id "user1" */
-public class FnCaller implements StatefulFunction {
+/** A function type that will be bound to {@link FnHelloWorld}.*/
+public class Identifiers {
- public static final FunctionType TYPE = new FunctionType("example", "caller");
-
- @Override
- public void invoke(Context context, Object input) {
- context.send(FnUser.TYPE, "user1", new MyUserMessage());
- }
+ public static final FunctionType HELLO_TYPE =
+ new FunctionType("apache/flink", "hello");
}