[GitHub] lucperkins commented on a change in pull request #1481: Pulsar Functions overview

[GitBox <gi...@apache.org>]

lucperkins commented on a change in pull request #1481: Pulsar Functions overview
URL: https://github.com/apache/incubator-pulsar/pull/1481#discussion_r178604730
 
 

 ##########
 File path: site/docs/latest/functions/overview.md
 ##########
 @@ -52,51 +55,200 @@ $ bin/pulsar-functions localrun \
   --className org.apache.pulsar.functions.api.examples.ExclamationFunction
 ```
 
-### Supported languages
+## Configuration
 
-Pulsar Functions can currently be written in [Java](../../functions/api#java) and [Python](../../functions/api#python). Support for additional languages is coming soon.
+Pulsar Functions can be configured in two ways:
+
+* Via [command-line arguments](#cli) passed to the [`pulsar-admin functions`](../../reference/CliTools#pulsar-admin-functions) interface
+* Via [YAML](http://yaml.org/) configuration files
+
+If you're supplying a YAML configuration, you must specify a path to the file on the command line. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+  --functionConfigFile ./my-function.yaml
+```
 
-### Runtime
+And here's an example `my-function.yaml` file:
+
+```yaml
+name: my-function
+tenant: sample
+namespace: ns1
+jar: ./target/my-functions.jar
+className: org.example.pulsar.functions.MyFunction
+inputs:
+- persistent://sample/standalone/ns1/test_src
+output: persistent://sample/standalone/ns1/test_result
+```
 
-### Deployment modes
+You can also mix and match configuration methods by specifying some function attributes via the CLI and others via YAML configuration.
 
-* Local run
-* Cluster run
+## Supported languages
 
-### Delivery semantics
+Pulsar Functions can currently be written in [Java](../../functions/api#java) and [Python](../../functions/api#python). Support for additional languages is coming soon.
 
-* At most once
-* At least once
-* Effectively once
+## Function context {#context}
 
-### State storage
+Each Pulsar Function created using the [Pulsar Functions SDK](#sdk) has access to a context object that both provides:
 
-Although you can certainly use Pulsar Functions to perform stateless computations, many use cases demand robust state storage
+1. A wide variety of information about the function, including:
+  * The name of the function
+  * The {% popover tenant %} and {% popover namespace %} of the function
+  * [User-supplied configuration]() values
 
+### Language-native functions {#native}
 
-You can certainly use Pulsar Functions to perform stateless operations, 
+Both Java and Python support writing "native" functions, i.e. Pulsar Functions with no dependencies.
 
-By default, Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) for state storage.
+The benefit of native functions is that they don't have any dependencies beyond what's already available in Java/Python "out of the box." The downside is that they don't provide access to the function's [context](#context)
 
-### Metrics
+### The Pulsar Functions SDK {#sdk}
 
-Here's an example function that publishes a value of 1 to the `my-metric` metric.
+If you'd like a Pulsar Function to have access to a [context object](#context), you can use the Pulsar Functions SDK, available for both [Java](../api#java-sdk) and [Pythnon](../api#python-sdk).
+
+Here's an example Java function that uses information about its context:
 
 ```java
-public class MetricsFunction implements PulsarFunction<String, Void> {
+import org.apache.pulsar.functions.api.Context;
+import org.apache.pulsar.functions.api.Function;
+import org.slf4j.Logger;
+
+public class ContextAwareFunction implements Function<String, Void> {
     @Override
-    public Void process(String input, Context context) {
-        context.recordMetric("my-metric", 1);
+    public Void process(String input, Context, context) {
+        Logger LOG = context.getLogger();
+        String functionTenant = context.getTenant();
+        String functionNamespace = context.getNamespace();
+        String functionName = context.getName();
+        LOG.info("{}/{}/{}", functionTenant, functionNamespace, functionName);
         return null;
     }
 }
 ```
 
+## Deployment modes
+
+The Pulsar Functions feature was built to support a variety of deployment options. At the moment, there are two ways to run Pulsar Functions:
+
+Deployment mode | Description
+:---------------|:-----------
+Local run mode | The function runs in your local environment, for example on your laptop
+Cluster mode | The function runs *inside of* your Pulsar cluster, on the same machines as your Pulsar {% popover brokers %}
+
+### Local run mode {#local-run}
+
+If you run a Pulsar Function in **local run** mode, it will run on the machine from which the command is run (this could be your laptop, an [AWS EC2](https://aws.amazon.com/ec2/) instance, etc.). Here's an example [`localrun`](../../CliTools#pulsar-admin-functions-localrun) command:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+  --py myfunc.py \
+  --className myfunc.SomeFunction \
+  --inputs persistent://sample/standalone/ns1/input-1 \
+  --output persistent://sample/standalone/ns1/output-1
+```
+
+By default, the function will connect to a Pulsar cluster running on the same machine, via a local {% popover broker %} service URL of `pulsar://localhost:6650`. If you'd like to use local run mode to run a function but connect it to a non-local Pulsar cluster, you can specify a different broker URL using the `--brokerServiceUrl` flag. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions localrun \
+  --brokerServiceUrl pulsar://my-cluster-host:6650 \
+  # Other function parameters
+```
+
+### Cluster run mode {#cluster-run}
+
+When you run a Pulsar Function in **cluster mode**, the function code will be uploaded to a Pulsar {% popover broker %} and run *alongside the broker* rather than in your [local environment](#local-run). You can run a function in cluster mode using the [`create`](../../CliTools#pulsar-admin-functions-create) command. Here's an example:
+
+```bash
+$ bin/pulsar-admin functions create \
+  --py myfunc.py \
+  --className myfunc.SomeFunction \
+  --inputs persistent://sample/standalone/ns1/input-1 \
+  --output persistent://sample/standalone/ns1/output-1
+```
+
+This command will upload `myfunc.py` to Pulsar, which will use the code to start one [or more](#parallelism) instances of the function.
+
+### Parallelism
+
+By default, only one **instance** of a Pulsar Function runs when you create and run it in [cluster run mode](#cluster-run). You can also, however, run multiple instances in parallel. You can specify the number of instances when you create the function, or update an existing single-instance function with a new parallelism factor.
+
+This command, for example, would create and run a function with a parallelism of 5 (i.e. 5 instances):
+
+```bash
+$ bin/pulsar-admin functions create \
+  --name parallel-fun \
+  --tenant sample \
+  --namespace ns1 \
+  --py func.py \
+  --className func.ParallelFunction \
+  --parallelism 5
+```
+
 ### Logging
 
-### Data types
+Pulsar Functions created using the [Pulsar Functions SDK(#sdk) can produce logs. Those logs are then published on the Pulsar topic that you specify. The function created using this command, for example, would produce all logs on the `persistent://sample/standalone/ns1/my-func-1-log` topic:
 
 Review comment:
   Okay, that makes sense

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services