You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2018/04/30 17:17:29 UTC
[GitHub] lucperkins closed pull request #1601: Pulsar Functions state
storage documentation (WIP)
lucperkins closed pull request #1601: Pulsar Functions state storage documentation (WIP)
URL: https://github.com/apache/incubator-pulsar/pull/1601
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index 68fe4263c4..a28a83af65 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -44,6 +44,8 @@ groups:
endpoint: deployment
- title: Processing guarantees
endpoint: guarantees
+ - title: State storage
+ endpoint: state
- title: Metrics for Pulsar Functions
endpoint: metrics
diff --git a/site/docs/latest/functions/overview.md b/site/docs/latest/functions/overview.md
index c3cf23d65d..b795820b29 100644
--- a/site/docs/latest/functions/overview.md
+++ b/site/docs/latest/functions/overview.md
@@ -397,4 +397,9 @@ Pulsar Functions that use the [Pulsar Functions SDK](#sdk) can publish metrics t
## State storage
-Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. All Pulsar installations, including local {% popover standalone %} installations, include a deployment of BookKeeper {% popover bookies %}.
\ No newline at end of file
+Pulsar Functions are a great choice for performing stateless computations using Pulsar message data. But they can also be used for operations that do require state via the **state storage API**. For a more in-depth guide, see [State storage for Pulsar Functions](../state).
+
+{% include admonition.html type="info" content="Pulsar Function [counters](#counters) are simply a special case of Pulsar Function state storage." %}
+
+{% include admonition.html type="success" title="No need for an external database"
+content="Pulsar uses the [Apache BookKeeper](../../getting-started/ConceptsAndArchitecture#persistent-storage) log storage system for persistent storage of message data (and for other purposes). Pulsar Functions also use BookKeeper for state storage. This means that get state storage \"for free\" in Pulsar Functions, i.e. you get a fast, strongly consistent, highly available data storage system without needing to deploy your own." %}
\ No newline at end of file
diff --git a/site/docs/latest/functions/state.md b/site/docs/latest/functions/state.md
new file mode 100644
index 0000000000..1187929e51
--- /dev/null
+++ b/site/docs/latest/functions/state.md
@@ -0,0 +1,74 @@
+---
+title: State storage for Pulsar Functions
+preview: true
+---
+
+Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. All Pulsar installations, including local {% popover standalone %} installations, include a deployment of BookKeeper {% popover bookies %}---thus, state storage comes "out of the box" for Pulsar Functions.
+
+## API
+
+The state storage API for Pulsar Functions is very simple and consists of just a handful of operations:
+
+Operation | Description
+:---------|:------------
+Get value | Fetches the value associated with the specified key (if any)
+Put value | Updates the value associated with a given key
+Increment counter | Increments a specific counter specified by key (you can also decrement using negative increments)
+Get counter | Fetches the current value associated with a counter (or zero if the counter has never been used)
+
+## Deployment
+
+{% include admonition.html type="success" content="For most Pulsar installations, you won't need to specify an alternative state storage service URL. If you're just using Pulsar's built-in Apache BookKeeper storage system, Pulsar Functions running in both [local run](../deployment#local-run) and [cluster mode](../deployment#cluster-mode) can use state storage." %}
+
+## Java
+
+The state storage API for Java is provided by the {% javadoc Context client org.apache.pulsar.functions.api.Context %} class. Here's an example usage of the context object for state storage and retrieval:
+
+```java
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+public class StateFunction implements Function<String, String> {
+ private static final String VALUE = "some value";
+
+ public String process(String key, Context context) {
+ Logger LOG = context.getLogger();
+
+ // Fetch the value associated with the same key
+ CompletableFuture<Optional<ByteBuffer>> futureValue = context.getValue(key);
+
+ // Extract the fetched value and return it as the function's output
+ futureValue.thenAccept(value -> {
+ if (value.isPresent()) {
+ String fetchedvalue = new String(value.get().array());
+ return fetchedValue;
+ } else {
+ LOG.error("No value associated with the key {}", key);
+ return null;
+ }
+ });
+ }
+}
+```
+
+The function above listens for incoming strings, uses those strings as state keys, and then seeks to fetch the state value associated with that key. If a value is present, the function returns that value, otherwise the function returns nothing (`null` in this case).
+
+#### Java State storage API {#api}
+
+The state storage API for Pulsar Functions is a fairly simple get/put-style key/value interface with just a few methods:
+
+Operation | `Context` method
+:---------|:----------------
+Get value | `CompletableFuture<ByteBuffer> getValue(String key)`
+Put value | `CompletableFuture<Void> putValue(String key, ByteBuffer value)`
+Increment counter | `CompletableFuture<Void> incrCounter(String key, long increment)`
+Get counter value | `CompletableFuture<Long> getCounter(String key)`
+
+#### Data types
+
+In the Pulsar Functions state storage API, all keys are strings and all values are raw byte arrays. *All serialization to and deserialization from other data types must be handled by the function*.
\ No newline at end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services