You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/14 21:10:16 UTC

[GitHub] [pulsar] addisonj opened a new pull request #7534: [docs] Improve configuration details of functions worker

addisonj opened a new pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534


   
   Fixes #6806
   
   ### Motivation
   
   The documentation for functions worker were lacking, as it didn't cover how to configure the functions worker in a variety of possible configurations of Pulsar, such as when the functions worker is run standalone or with TLS and/or authentication.
   
   
   ### Modifications
   This document expands the details for how to configure the functions
   worker based on a variety of scenarios.
   
   
   ### Verifying this change
   
   This is just docs, no test changes needed
   ### Does this pull request potentially affect one of the following parts:
   
   It doesn't effect anything other than docs :)
   
   ### Documentation
   
     - Does this pull request introduce a new feature? ho
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] addisonj commented on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
addisonj commented on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658892374


   @Anonymitaet Thanks for all the great feedback as well as links to style, that is really helpful :) 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on a change in pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#discussion_r454761678



##########
File path: site2/docs/functions-runtime.md
##########
@@ -138,20 +149,75 @@ roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: functions-worker
-subjects:
+subjectsKubernetesSec:
 - kind: ServiceAccount
   name: functions-worker
 ```
 
+If the service-account is not properly configured, you may see an error message similar to this:
+```bash
+22:04:27.696 [Timer-0] ERROR org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory - Error while trying to fetch configmap example-pulsar-4qvmb5gur3c6fc9dih0x1xn8b-function-worker-config at namespace pulsar
+io.kubernetes.client.ApiException: Forbidden
+	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:23673) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:23655) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:284) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:275) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_212]
+	at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_212]
+```
+
+### Kubernetes Secrets integration
+
+In order to safely distribute secrets, Pulasr Functions can reference Kubernetes secrets. To enable this, set the `secretsProviderConfiguratorClassName` to `org.apache.pulsar.functions.secretsproviderconfigurator.KubernetesSecretsProviderConfigurator`.
+
+Then, you can create a secret in the namespace where your functions will be deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.
+
+The following functions configuration would then allow us to reference that secret and have the value be mounted as an environment variable in the pod.
+
+```Yaml
+tenant: "mytenant"
+namespace: "mynamespace"
+name: "myfunction"
+topicName: "persistent://mytenant/mynamespace/myfuncinput"
+className: "com.company.pulsar.myfunction"
+
+secrets:
+  # the secret will be mounted from the `password` field in the `database-creds` secret as an env var called `DATABASE_PASSWORD`
+  DATABASE_PASSWORD:
+    path: "database-creds"
+    key: "password"
+
+```
+
+### Kubernetes Functions authentication
+
+
+When your Pulsar cluster uses authentication, the pod running your function needs a mechanism to authenticate with the Broker.

Review comment:
       ```suggestion
   When your Pulsar cluster uses authentication, the pod running your function needs a mechanism to authenticate with the broker.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -46,6 +46,17 @@ processContainerFactory:
 
 ## Configure Kubernetes runtime
 
+
+### How it works
+
+The Kubernetes runtime works by having the functions-worker generate and apply kubernetes manifests. In the event that the functions worker is running on Kubernetes already, it can use the `serviceAccount` that is associated with the pod the functions worker is running in. Otherwise, it can be configured to communicate with a Kubernetes cluster.

Review comment:
       ```suggestion
   The Kubernetes runtime works by having the functions worker generate and apply Kubernetes manifests. In the event that the functions worker is running on Kubernetes already, it can use the `serviceAccount` that is associated with the pod the functions worker is running in. Otherwise, it can be configured to communicate with a Kubernetes cluster.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -78,9 +89,23 @@ kubernetesContainerFactory:
   percentMemoryPadding: 10
 ```
 
-If you have already run a Pulsar cluster on Kubernetes, you can keep the settings unchanged at most of time.
+As stated earlier, If you already run your Functions Worker embedded in a Broker on Kubernetes, you can keep many of these settings as their default.

Review comment:
       ```suggestion
   As stated earlier, if you already run your functions worker embedded in a broker on Kubernetes, you can keep many of these settings as default.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -138,20 +149,75 @@ roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: functions-worker
-subjects:
+subjectsKubernetesSec:
 - kind: ServiceAccount
   name: functions-worker
 ```
 
+If the service-account is not properly configured, you may see an error message similar to this:
+```bash
+22:04:27.696 [Timer-0] ERROR org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory - Error while trying to fetch configmap example-pulsar-4qvmb5gur3c6fc9dih0x1xn8b-function-worker-config at namespace pulsar
+io.kubernetes.client.ApiException: Forbidden
+	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:23673) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:23655) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:284) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:275) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_212]
+	at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_212]
+```
+
+### Kubernetes Secrets integration
+
+In order to safely distribute secrets, Pulasr Functions can reference Kubernetes secrets. To enable this, set the `secretsProviderConfiguratorClassName` to `org.apache.pulsar.functions.secretsproviderconfigurator.KubernetesSecretsProviderConfigurator`.
+
+Then, you can create a secret in the namespace where your functions will be deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.
+
+The following functions configuration would then allow us to reference that secret and have the value be mounted as an environment variable in the pod.
+
+```Yaml
+tenant: "mytenant"
+namespace: "mynamespace"
+name: "myfunction"
+topicName: "persistent://mytenant/mynamespace/myfuncinput"
+className: "com.company.pulsar.myfunction"
+
+secrets:
+  # the secret will be mounted from the `password` field in the `database-creds` secret as an env var called `DATABASE_PASSWORD`
+  DATABASE_PASSWORD:
+    path: "database-creds"
+    key: "password"
+
+```
+
+### Kubernetes Functions authentication
+
+
+When your Pulsar cluster uses authentication, the pod running your function needs a mechanism to authenticate with the Broker.
+
+An interface, `org.apache.pulsar.functions.auth.KubernetesFunctionAuthProvider`, can be extended to provide support for any authentication mechanism. The `functionAuthProviderClassName` in `function-worker.yml` is used to specify your path to this implementation.
+
+Pulsar includes an implementation of this interface that is suitable for token auth, which should be configured like the following in the configuration:
+```Yaml
+functionAuthProviderClassName: org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider
+```
+
+For custom auth or TLS, the user will need to implement this interface (or use an alternative mechanism to provide auth)

Review comment:
       ```suggestion
   For custom authentication or TLS, you need to implement this interface (or use an alternative mechanism to provide authentication)
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -138,20 +149,75 @@ roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: functions-worker
-subjects:
+subjectsKubernetesSec:
 - kind: ServiceAccount
   name: functions-worker
 ```
 
+If the service-account is not properly configured, you may see an error message similar to this:
+```bash
+22:04:27.696 [Timer-0] ERROR org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory - Error while trying to fetch configmap example-pulsar-4qvmb5gur3c6fc9dih0x1xn8b-function-worker-config at namespace pulsar
+io.kubernetes.client.ApiException: Forbidden
+	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:23673) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:23655) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:284) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:275) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_212]
+	at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_212]
+```
+
+### Kubernetes Secrets integration
+
+In order to safely distribute secrets, Pulasr Functions can reference Kubernetes secrets. To enable this, set the `secretsProviderConfiguratorClassName` to `org.apache.pulsar.functions.secretsproviderconfigurator.KubernetesSecretsProviderConfigurator`.
+
+Then, you can create a secret in the namespace where your functions will be deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.
+
+The following functions configuration would then allow us to reference that secret and have the value be mounted as an environment variable in the pod.
+
+```Yaml
+tenant: "mytenant"
+namespace: "mynamespace"
+name: "myfunction"
+topicName: "persistent://mytenant/mynamespace/myfuncinput"
+className: "com.company.pulsar.myfunction"
+
+secrets:
+  # the secret will be mounted from the `password` field in the `database-creds` secret as an env var called `DATABASE_PASSWORD`
+  DATABASE_PASSWORD:
+    path: "database-creds"
+    key: "password"
+
+```
+
+### Kubernetes Functions authentication
+
+
+When your Pulsar cluster uses authentication, the pod running your function needs a mechanism to authenticate with the Broker.
+
+An interface, `org.apache.pulsar.functions.auth.KubernetesFunctionAuthProvider`, can be extended to provide support for any authentication mechanism. The `functionAuthProviderClassName` in `function-worker.yml` is used to specify your path to this implementation.
+
+Pulsar includes an implementation of this interface that is suitable for token auth, which should be configured like the following in the configuration:
+```Yaml
+functionAuthProviderClassName: org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider
+```
+
+For custom auth or TLS, the user will need to implement this interface (or use an alternative mechanism to provide auth)
+
+For token authentication, the way this works is that the functions worker captures the token that was used to deploy (or update) the function. This token is then saved as a secret and mounted into the pod.
+
+One thing to keep in mind is that if you use tokens that expire when deploying functions, these tokens will expire.
+
+
 ### Kubernetes CustomRuntimeOptions
 
-The functions (and sinks/sources) API provides a flag, `customRuntimeOptions` which can be used to pass options to the runtime to customize how the runtime operates.
+The Kubernetes integration also includes the ability for the user to implement a class which will customize how manifests will be generated. This is configured by setting the `runtimeCustomizerClassName` in the `functions-worker.yml` to the fully qualified class name. The interface the user must implement is the `org.apache.pulsar.functions.runtime.kubernetes.KubernetesManifestCustomizer` interface.
+
+The functions (and sinks/sources) API provides a flag, `customRuntimeOptions` which will be passed to this interface.

Review comment:
       ```suggestion
   The functions (and sinks/sources) API provides a flag, `customRuntimeOptions`, which is passed to this interface.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker
+
+When configuring a standalone Functions Worker, you need to specify a few properties in order for the Functions Worker to be able
+to communicate with the Broker. This requires many of the same properties to be set that the Broker requires, especially when using TLS.
+
+The following properties are the baseline of what is required:
+
+```Yaml
+workerPort: 8080
+workerPortTls: 8443 # when using TLS
+tlsCertificateFilePath: /etc/pulsar/tls/tls.crt # when using TLS
+tlsKeyFilePath: /etc/pulsar/tls/tls.key # when using TLS
+tlsTrustCertsFilePath: /etc/pulsar/tls/ca.crt # when using TLS
+pulsarServiceUrl: pulsar://broker.pulsar:6650/ # or pulsar+ssl://pulsar-prod-broker.pulsar:6651/ when using TLS
+pulsarWebServiceUrl: http://broker.pulsar:8080/ # or https://pulsar-prod-broker.pulsar:8443/ when using TLS
+useTls: true # when using TLS, critical!
+
+```
+
+#### With Authentication
+
+When running a Functions Worker in a standalone process (i.e. not embedded in the Broker) in a cluster with authentication, you must configure your Functions Worker to both be able to interact with the Broker *and* also to authenticate incoming requests as well. This requires many of the same properties to be set that the Broker requires for authentication/authorization.
+
+As an example, assuming we want to use token authentication, here is an example of properties we need to set in `function-worker.yml`
+
+```Yaml
+clientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
+clientAuthenticationParameters: file:///etc/pulsar/token/admin-token.txt
+configurationStoreServers: zookeeper-cluster:2181 # auth requires a connection to zookeeper
+authenticationProviders:
+ - "org.apache.pulsar.broker.authentication.AuthenticationProviderToken"
+authorizationEnabled: true
+authenticationEnabled: true
+superUserRoles:
+  - superuser
+  - proxy
+properties:
+  tokenSecretKey: file:///etc/pulsar/jwt/secret # if using a secret token
+  tokenPublicKey: file:///etc/pulsar/jwt/public.key # if using public/private key tokens
+```
+
+Notice that we must configure both the Function Worker authz/authn for the server to proper auth requests and configure the client to be authenticated to communicate with the broker.

Review comment:
       ```suggestion
   > #### Note 
   >
   > You must configure both the Function Worker authorization or authentication for the server to proper authentication requests and configure the client to be authenticated to communicate with the broker.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -46,6 +46,17 @@ processContainerFactory:
 
 ## Configure Kubernetes runtime
 
+
+### How it works
+
+The Kubernetes runtime works by having the functions-worker generate and apply kubernetes manifests. In the event that the functions worker is running on Kubernetes already, it can use the `serviceAccount` that is associated with the pod the functions worker is running in. Otherwise, it can be configured to communicate with a Kubernetes cluster.
+
+The manifests which the functions worker generates include a `StatefulSet`, a `Service (which is used to communicate with the pods), and a `Secret` for auth credentials (when applicable). The StatefulSet manifest (by default) has a single pod, with the number of replicas determined by the "parallelism" of the function. On pod boot, the pod will download the function payload (via the functions worker REST API). The pod's container image is configurable, but must have the functions runtime.
+
+The Kubernetes runtime also has supports for secrets, with the end user being able to create a kubernetes secret and have it be exposed as an environment variable in the pod (described below). Additionally, the Kubernetes runtime fairly extensible, with the user being able to implement classes that customize the way kubernetes manifests get generated, how auth data is passed to pods, and how secrets can be integrated.

Review comment:
       ```suggestion
   The Kubernetes runtime also supports secrets, with the end user being able to create a Kubernetes secret and have it be exposed as an environment variable in the pod (described below). Additionally, the Kubernetes runtime fairly extensible, with the user being able to implement classes that customize the way Kubernetes manifests get generated, how auth data is passed to pods, and how secrets can be integrated.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -78,9 +89,23 @@ kubernetesContainerFactory:
   percentMemoryPadding: 10
 ```
 
-If you have already run a Pulsar cluster on Kubernetes, you can keep the settings unchanged at most of time.
+As stated earlier, If you already run your Functions Worker embedded in a Broker on Kubernetes, you can keep many of these settings as their default.
+
+### Standalone Functions Worker on K8S
+
+If you run your Functions Worker standalone (i.e. not embedded) on Kubernetes, you will need to configure `pulsarSerivceUrl` to be the URL of the
+broker and `pulsarAdminUrl` as the URL to the Functions Worker.
+
+As an example, suppose both our Pulsar Brokers and Function Workers are run in the `pulsar` K8S namespace. Additionally, assuming the Brokers have a service called `brokers` and the Functions Worker has a service called `func-worker`, then the settings would be:

Review comment:
       ```suggestion
   As an example, suppose both our Pulsar brokers and Function Workers run in the `pulsar` K8S namespace. Additionally, assuming the brokers have a service called `brokers` and the Functions Worker has a service called `func-worker`, then the settings would be:
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -46,6 +46,17 @@ processContainerFactory:
 
 ## Configure Kubernetes runtime
 
+
+### How it works
+
+The Kubernetes runtime works by having the functions-worker generate and apply kubernetes manifests. In the event that the functions worker is running on Kubernetes already, it can use the `serviceAccount` that is associated with the pod the functions worker is running in. Otherwise, it can be configured to communicate with a Kubernetes cluster.
+
+The manifests which the functions worker generates include a `StatefulSet`, a `Service (which is used to communicate with the pods), and a `Secret` for auth credentials (when applicable). The StatefulSet manifest (by default) has a single pod, with the number of replicas determined by the "parallelism" of the function. On pod boot, the pod will download the function payload (via the functions worker REST API). The pod's container image is configurable, but must have the functions runtime.

Review comment:
       ```suggestion
   The manifests which the functions worker generates include a `StatefulSet`, a `Service` (which is used to communicate with the pods), and a `Secret` for auth credentials (when applicable). The `StatefulSet` manifest (by default) has a single pod, with the number of replicas determined by the "parallelism" of the function. On pod boot, the pod downloads the function payload (via the functions worker REST API). The pod's container image is configurable, but must have the functions runtime.
   ```
   
   Use present tense if you are covering facts that were, are, and forever shall be true.
   Always use the present tense in technical writing by default. Only use past or future tenses when you’re actually writing about something that happened in the past or will happen in the future.
   

##########
File path: site2/docs/functions-runtime.md
##########
@@ -78,9 +89,23 @@ kubernetesContainerFactory:
   percentMemoryPadding: 10
 ```
 
-If you have already run a Pulsar cluster on Kubernetes, you can keep the settings unchanged at most of time.
+As stated earlier, If you already run your Functions Worker embedded in a Broker on Kubernetes, you can keep many of these settings as their default.
+
+### Standalone Functions Worker on K8S
+
+If you run your Functions Worker standalone (i.e. not embedded) on Kubernetes, you will need to configure `pulsarSerivceUrl` to be the URL of the

Review comment:
       ```suggestion
   If you run your Functions Worker standalone (that is, not embedded) on Kubernetes, you need to configure `pulsarSerivceUrl` to be the URL of the
   ```
   
   IBM Writing Style Guide: Do not use Latin abbreviations, such as e.g., etc., and i.e.

##########
File path: site2/docs/functions-runtime.md
##########
@@ -138,20 +149,75 @@ roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: functions-worker
-subjects:
+subjectsKubernetesSec:
 - kind: ServiceAccount
   name: functions-worker
 ```
 
+If the service-account is not properly configured, you may see an error message similar to this:
+```bash
+22:04:27.696 [Timer-0] ERROR org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory - Error while trying to fetch configmap example-pulsar-4qvmb5gur3c6fc9dih0x1xn8b-function-worker-config at namespace pulsar
+io.kubernetes.client.ApiException: Forbidden
+	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:23673) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:23655) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:284) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:275) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_212]
+	at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_212]
+```
+
+### Kubernetes Secrets integration
+
+In order to safely distribute secrets, Pulasr Functions can reference Kubernetes secrets. To enable this, set the `secretsProviderConfiguratorClassName` to `org.apache.pulsar.functions.secretsproviderconfigurator.KubernetesSecretsProviderConfigurator`.
+
+Then, you can create a secret in the namespace where your functions will be deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.
+
+The following functions configuration would then allow us to reference that secret and have the value be mounted as an environment variable in the pod.
+
+```Yaml
+tenant: "mytenant"
+namespace: "mynamespace"
+name: "myfunction"
+topicName: "persistent://mytenant/mynamespace/myfuncinput"
+className: "com.company.pulsar.myfunction"
+
+secrets:
+  # the secret will be mounted from the `password` field in the `database-creds` secret as an env var called `DATABASE_PASSWORD`
+  DATABASE_PASSWORD:
+    path: "database-creds"
+    key: "password"
+
+```
+
+### Kubernetes Functions authentication
+
+
+When your Pulsar cluster uses authentication, the pod running your function needs a mechanism to authenticate with the Broker.
+
+An interface, `org.apache.pulsar.functions.auth.KubernetesFunctionAuthProvider`, can be extended to provide support for any authentication mechanism. The `functionAuthProviderClassName` in `function-worker.yml` is used to specify your path to this implementation.
+
+Pulsar includes an implementation of this interface that is suitable for token auth, which should be configured like the following in the configuration:
+```Yaml
+functionAuthProviderClassName: org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider
+```
+
+For custom auth or TLS, the user will need to implement this interface (or use an alternative mechanism to provide auth)
+
+For token authentication, the way this works is that the functions worker captures the token that was used to deploy (or update) the function. This token is then saved as a secret and mounted into the pod.
+
+One thing to keep in mind is that if you use tokens that expire when deploying functions, these tokens will expire.
+
+
 ### Kubernetes CustomRuntimeOptions
 
-The functions (and sinks/sources) API provides a flag, `customRuntimeOptions` which can be used to pass options to the runtime to customize how the runtime operates.
+The Kubernetes integration also includes the ability for the user to implement a class which will customize how manifests will be generated. This is configured by setting the `runtimeCustomizerClassName` in the `functions-worker.yml` to the fully qualified class name. The interface the user must implement is the `org.apache.pulsar.functions.runtime.kubernetes.KubernetesManifestCustomizer` interface.

Review comment:
       ```suggestion
   The Kubernetes integration also includes the ability for the user to implement a class which customizes how manifests is generated. This is configured by setting the `runtimeCustomizerClassName` in the `functions-worker.yml` to the fully qualified class name. The interface the you must implement is the `org.apache.pulsar.functions.runtime.kubernetes.KubernetesManifestCustomizer` interface.
   ```
   
   IBM Writing Style Guide: Use the second-person pronoun `you` as much as possible. The subject of an imperative sentence is understood to be you.

##########
File path: site2/docs/functions-runtime.md
##########
@@ -138,20 +149,75 @@ roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: functions-worker
-subjects:
+subjectsKubernetesSec:
 - kind: ServiceAccount
   name: functions-worker
 ```
 
+If the service-account is not properly configured, you may see an error message similar to this:
+```bash
+22:04:27.696 [Timer-0] ERROR org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory - Error while trying to fetch configmap example-pulsar-4qvmb5gur3c6fc9dih0x1xn8b-function-worker-config at namespace pulsar
+io.kubernetes.client.ApiException: Forbidden
+	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[io.kubernetes-client-java-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMapWithHttpInfo(CoreV1Api.java:23673) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at io.kubernetes.client.apis.CoreV1Api.readNamespacedConfigMap(CoreV1Api.java:23655) ~[io.kubernetes-client-java-api-2.0.0.jar:?]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory.fetchConfigMap(KubernetesRuntimeFactory.java:284) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at org.apache.pulsar.functions.runtime.KubernetesRuntimeFactory$1.run(KubernetesRuntimeFactory.java:275) [org.apache.pulsar-pulsar-functions-runtime-2.4.0-42c3bf949.jar:2.4.0-42c3bf949]
+	at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_212]
+	at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_212]
+```
+
+### Kubernetes Secrets integration
+
+In order to safely distribute secrets, Pulasr Functions can reference Kubernetes secrets. To enable this, set the `secretsProviderConfiguratorClassName` to `org.apache.pulsar.functions.secretsproviderconfigurator.KubernetesSecretsProviderConfigurator`.
+
+Then, you can create a secret in the namespace where your functions will be deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.

Review comment:
       ```suggestion
   Then, you can create a secret in the namespace where your functions are deployed. As an example, suppose we are deploying functions to the `pulsar-func` Kubernetes namespace, and we have a secret named `database-creds` with a field name `password`, which we want to mount in the pod as an environment variable called `DATABASE_PASSWORD`.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker
+
+When configuring a standalone Functions Worker, you need to specify a few properties in order for the Functions Worker to be able
+to communicate with the Broker. This requires many of the same properties to be set that the Broker requires, especially when using TLS.
+
+The following properties are the baseline of what is required:
+
+```Yaml
+workerPort: 8080
+workerPortTls: 8443 # when using TLS
+tlsCertificateFilePath: /etc/pulsar/tls/tls.crt # when using TLS
+tlsKeyFilePath: /etc/pulsar/tls/tls.key # when using TLS
+tlsTrustCertsFilePath: /etc/pulsar/tls/ca.crt # when using TLS
+pulsarServiceUrl: pulsar://broker.pulsar:6650/ # or pulsar+ssl://pulsar-prod-broker.pulsar:6651/ when using TLS
+pulsarWebServiceUrl: http://broker.pulsar:8080/ # or https://pulsar-prod-broker.pulsar:8443/ when using TLS
+useTls: true # when using TLS, critical!
+
+```
+
+#### With Authentication

Review comment:
       ```suggestion
   #### With authentication
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions

Review comment:
       ```suggestion
   If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster. Otherwise, the function
   ```
   
   IBM Writing Style Guide: 
   Do not punctuate sentences with exclamation points because their tone can be interpreted negatively,
   for example, as aggressive, condescending, or overly informal.
   Convey urgency or emphasis with the appropriate words, not with exclamation points. To call
   attention to important hints, tips, guidance, restrictions, or advice that might be overlooked, consider
   using a note that has a meaningful label.

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker

Review comment:
       ```suggestion
   ### Configure standalone Functions Worker
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.

Review comment:
       ```suggestion
   shares a namespace and potentially schedule across clusters.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker
+
+When configuring a standalone Functions Worker, you need to specify a few properties in order for the Functions Worker to be able
+to communicate with the Broker. This requires many of the same properties to be set that the Broker requires, especially when using TLS.
+
+The following properties are the baseline of what is required:
+
+```Yaml
+workerPort: 8080
+workerPortTls: 8443 # when using TLS
+tlsCertificateFilePath: /etc/pulsar/tls/tls.crt # when using TLS
+tlsKeyFilePath: /etc/pulsar/tls/tls.key # when using TLS
+tlsTrustCertsFilePath: /etc/pulsar/tls/ca.crt # when using TLS
+pulsarServiceUrl: pulsar://broker.pulsar:6650/ # or pulsar+ssl://pulsar-prod-broker.pulsar:6651/ when using TLS
+pulsarWebServiceUrl: http://broker.pulsar:8080/ # or https://pulsar-prod-broker.pulsar:8443/ when using TLS
+useTls: true # when using TLS, critical!
+
+```
+
+#### With Authentication
+
+When running a Functions Worker in a standalone process (i.e. not embedded in the Broker) in a cluster with authentication, you must configure your Functions Worker to both be able to interact with the Broker *and* also to authenticate incoming requests as well. This requires many of the same properties to be set that the Broker requires for authentication/authorization.
+
+As an example, assuming we want to use token authentication, here is an example of properties we need to set in `function-worker.yml`

Review comment:
       ```suggestion
   As an example, assuming you want to use token authentication. Here is an example of properties you need to set in `function-worker.yml`
   ```
   
   IBM Writing Style Guide:
   • Avoid the first-person pronouns `I` and `we`, except in these situations:
   ° In the question portion of frequently asked questions (FAQs)
   ° In articles, white papers, or documents that have listed authors and in which the authors describe their own actions or opinions

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.

Review comment:
       ```suggestion
   This ensures the two different Function Workers use distinct sets of topics for their internal coordination.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker
+
+When configuring a standalone Functions Worker, you need to specify a few properties in order for the Functions Worker to be able
+to communicate with the Broker. This requires many of the same properties to be set that the Broker requires, especially when using TLS.

Review comment:
       ```suggestion
   to communicate with the broker. This requires many of the same properties to be set that the broker requires, especially when using TLS.
   ```

##########
File path: site2/docs/functions-runtime.md
##########
@@ -183,3 +249,68 @@ for the `runtimeCustomerClassName` property. This implementation takes the follo
   }
 }
 ```
+
+## Other configuration considerations
+
+
+### In a cluster with geo-replication
+
+If you are running multiple clusters tied together with geo-replication, it is important to use a different function namespace for each cluster! Otherwise, the functions
+will share a namespace and potentially schedule across clusters.
+
+As an example, suppose we have clusters east-1 and west-1, we would configure the Functions Worker in east-1 like:
+```Yaml
+pulsarFunctionsCluster: east-1
+pulsarFunctionsNamespace: public/functions-east-1
+```
+
+and the cluster in west-1 like:
+```Yaml
+pulsarFunctionsCluster: west-1
+pulsarFunctionsNamespace: public/functions-west-1
+```
+
+This ensures the two different Function Workers will use distinct sets of topics for their internal co-ordination.
+
+### Configuring Standalone Functions Worker
+
+When configuring a standalone Functions Worker, you need to specify a few properties in order for the Functions Worker to be able
+to communicate with the Broker. This requires many of the same properties to be set that the Broker requires, especially when using TLS.
+
+The following properties are the baseline of what is required:
+
+```Yaml
+workerPort: 8080
+workerPortTls: 8443 # when using TLS
+tlsCertificateFilePath: /etc/pulsar/tls/tls.crt # when using TLS
+tlsKeyFilePath: /etc/pulsar/tls/tls.key # when using TLS
+tlsTrustCertsFilePath: /etc/pulsar/tls/ca.crt # when using TLS
+pulsarServiceUrl: pulsar://broker.pulsar:6650/ # or pulsar+ssl://pulsar-prod-broker.pulsar:6651/ when using TLS
+pulsarWebServiceUrl: http://broker.pulsar:8080/ # or https://pulsar-prod-broker.pulsar:8443/ when using TLS
+useTls: true # when using TLS, critical!
+
+```
+
+#### With Authentication
+
+When running a Functions Worker in a standalone process (i.e. not embedded in the Broker) in a cluster with authentication, you must configure your Functions Worker to both be able to interact with the Broker *and* also to authenticate incoming requests as well. This requires many of the same properties to be set that the Broker requires for authentication/authorization.

Review comment:
       ```suggestion
   When running a Functions Worker in a standalone process (that is, not embedded in the broker) in a cluster with authentication, you must configure your Functions Worker to both be able to interact with the broker *and* also to authenticate incoming requests as well. This requires many of the same properties to be set that the broker requires for authentication or authorization.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet edited a comment on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658506639


   @addisonj thank you so much for your tech inputs.
   
   @wolfstudy could u pls help review? Thanks. 
   FYI: @addisonj has added the following contents (in blue)
   ![image](https://user-images.githubusercontent.com/50226895/87498960-ab9a6e80-c68b-11ea-92e0-cadbbec37597.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet edited a comment on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658506639


   @wolfstudy could u pls help review? Thanks
   
   FYI @addisonj has added the following contents (in blue)
   ![image](https://user-images.githubusercontent.com/50226895/87497211-b6530480-c687-11ea-907d-e587581d4ca1.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] addisonj commented on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
addisonj commented on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658415855


   @Anonymitaet here are the docs for #6086, let me know who else should review with you being out :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet edited a comment on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658506639


   @wolfstudy could u pls help review? Thanks
   
   FYI @addisonj has added the following contents (in blue)
   ![image](https://user-images.githubusercontent.com/50226895/87498960-ab9a6e80-c68b-11ea-92e0-cadbbec37597.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-658506639


   @wolfstudy could u pls help review? Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] jiazhai merged pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
jiazhai merged pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7534: [docs] Improve configuration details of functions worker

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7534:
URL: https://github.com/apache/pulsar/pull/7534#issuecomment-659164323


   @addisonj thank you so much for providing this helpful information :-D


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org