You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/27 06:17:49 UTC

[GitHub] [flink-statefun] tzulitai opened a new pull request #263: [FLINK-23600] [docs] Restructure module configuration docs

tzulitai opened a new pull request #263:
URL: https://github.com/apache/flink-statefun/pull/263


   This PR adds new documentation for:
   - The new leaner style for `module.yaml` configurations.
   - The asynchronous transport that users can opt into.
   
   The PR also uses the opportunity to restructure pages related to composing the `module.yaml` configuration file, all under a new top-level category (same level as `SDKs`) called `Modules`. This subsumes the `IO Modules` section and puts the contents of those as a subsection under `Modules`.
   
   The easiest way to review this would be to build the docs locally using Hugo and going through the content under `Modules`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink-statefun] sjwiesman commented on pull request #263: [FLINK-23600] [docs] Restructure module configuration docs

Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #263:
URL: https://github.com/apache/flink-statefun/pull/263#issuecomment-907287311


   As all my comments are grammar nits, I'm going to go ahead and apply them while merging. Thanks for your work on this! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink-statefun] sjwiesman commented on a change in pull request #263: [FLINK-23600] [docs] Restructure module configuration docs

Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #263:
URL: https://github.com/apache/flink-statefun/pull/263#discussion_r697510518



##########
File path: docs/content/docs/modules/http-endpoint.md
##########
@@ -0,0 +1,206 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+A HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  transport:
+    timeouts:
+      call: 1 min
+      read: 10 sec
+      write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  timeouts:
+    call: 1 min
+    read: 10 sec
+    write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace `com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint URL template.
+
+### URL Template
+
+The URL template name may contain template parameters that are filled in based on the function's specific type.
+In the example below, a message sent to message type `com.example/greeter` will be sent to `http://bar.foo.com/greeter`.
+
+```yaml
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com/{function.name}
+```
+
+Templating parameterization works well with load balancers and service gateways.
+Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server, you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others still on physical servers.
+
+{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
+
+### Transport
+
+Switching between different transport clients is supported since `v2` of HTTP function endpoint definitions.
+
+The transport client to use is specified using `spec.transport.type`. If not specified, by default, Stateful Functions uses [OkHttp](https://square.github.io/okhttp/).
+
+All fields under `spec.transport` is used as the properties to configure the transport client. For example, the example below configures various timeout settings for the default `OkHttp` transport:
+
+```yaml
+spec:
+  transport:
+    timeouts:
+      call: 1 min
+      read: 30 sec
+      write: 20 sec
+```
+
+#### Asynchronous HTTP transport (Beta)
+
+Alternatively, Stateful Functions also ships a transport option based on asynchronous non-blocking IO, implemented with [Netty](https://netty.io/).

Review comment:
       ```suggestion
   Alternatively, Stateful Functions also ships a transport option based on asynchronous non-blocking IO, implemented with [Netty](https://netty.io/).
   This transport enables much higher resource utilization, higher throughput, and lower remote function invocation latency.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink-statefun] sjwiesman commented on a change in pull request #263: [FLINK-23600] [docs] Restructure module configuration docs

Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #263:
URL: https://github.com/apache/flink-statefun/pull/263#discussion_r697486471



##########
File path: docs/content/docs/modules/overview.md
##########
@@ -0,0 +1,66 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+  - /modules/
+permalink: /modules/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Application Modules
+
+An application module consists of multiple [components]({{< ref "docs/concepts/application-building-blocks" >}})
+that take part in a StateFun application. It includes the endpoints where functions can be reached along with ingress
+and egress definitions.
+
+Modules are defined using a YAML file. For example, below is a module that defines an HTTP function endpoint as well as
+a Kafka ingress and egress:
+
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com/{function.name}
+---
+kind: io.statefun.kafka.v1/ingress
+spec:
+  id: com.example/my-ingress
+  address: kafka-broker:9092
+  consumerGroupId: my-consumer-group
+  topics:
+    - topic: message-topic
+      valueType: io.statefun.types/string
+      targets:
+        - com.example/greeter
+---
+kind: io.statefun.kafka.v1/egress
+spec:
+  id: com.example/my-egress
+  address: kafka-broker:9092
+  deliverySemantic:
+    type: exactly-once
+    transactionTimeout: 15min
+---
+```
+
+A module YAML file can contain multiple YAML documents, separated by `---`, each representing a component to be included in the
+application. Each component is defined by a `kind` typename string, and a `spec` object containing the properties of the
+component.

Review comment:
       ```suggestion
   A module YAML file can contain multiple YAML documents, separated by `---`, each representing a component to be included in the application.
   Each component is defined by a kind typename string and a spec object containing the component's properties.
   ```

##########
File path: docs/content/docs/modules/io/overview.md
##########
@@ -0,0 +1,36 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+  - /modules/io/
+permalink: /modules/io/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# I/O Module
+
+Stateful Functions' I/O modules allow functions to receive and send messages to external systems.
+Based on the concept of Ingress (input) and Egress (output) points, and built on top of the Apache FlinkĀ® connector ecosystem, I/O modules enable functions to interact with the outside world through the style of message passing.
+
+Commonly used I/O modules are bundled into the runtime by default and can be configured direclty via the applications [module configuration]({{< ref "docs/modules/overview" >}}). 

Review comment:
       ```suggestion
   Commonly used I/O modules are bundled into the runtime by default and can be configured directly via the applications [module configuration]({{< ref "docs/modules/overview" >}}). 
   ```

##########
File path: docs/content/docs/modules/http-endpoint.md
##########
@@ -0,0 +1,206 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+A HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  transport:
+    timeouts:
+      call: 1 min
+      read: 10 sec
+      write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  timeouts:
+    call: 1 min
+    read: 10 sec
+    write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace `com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint URL template.
+
+### URL Template
+
+The URL template name may contain template parameters that are filled in based on the function's specific type.
+In the example below, a message sent to message type `com.example/greeter` will be sent to `http://bar.foo.com/greeter`.
+
+```yaml
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com/{function.name}
+```
+
+Templating parameterization works well with load balancers and service gateways.
+Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server, you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others still on physical servers.
+
+{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
+
+### Transport
+
+Switching between different transport clients is supported since `v2` of HTTP function endpoint definitions.
+
+The transport client to use is specified using `spec.transport.type`. If not specified, by default, Stateful Functions uses [OkHttp](https://square.github.io/okhttp/).
+
+All fields under `spec.transport` is used as the properties to configure the transport client. For example, the example below configures various timeout settings for the default `OkHttp` transport:
+
+```yaml
+spec:
+  transport:
+    timeouts:
+      call: 1 min
+      read: 30 sec
+      write: 20 sec
+```
+
+#### Asynchronous HTTP transport (Beta)
+
+Alternatively, Stateful Functions also ships a transport option based on asynchronous non-blocking IO, implemented with [Netty](https://netty.io/).

Review comment:
       We need one more sentence here motivating the feature. 

##########
File path: docs/content/docs/modules/http-endpoint.md
##########
@@ -0,0 +1,206 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+A HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  transport:
+    timeouts:
+      call: 1 min
+      read: 10 sec
+      write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  timeouts:
+    call: 1 min
+    read: 10 sec
+    write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace `com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint URL template.
+
+### URL Template
+
+The URL template name may contain template parameters that are filled in based on the function's specific type.
+In the example below, a message sent to message type `com.example/greeter` will be sent to `http://bar.foo.com/greeter`.
+
+```yaml
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com/{function.name}
+```
+
+Templating parameterization works well with load balancers and service gateways.
+Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server, you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others still on physical servers.

Review comment:
       ```suggestion
   Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server; you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others are still on physical servers.
   ```

##########
File path: docs/content/docs/modules/http-endpoint.md
##########
@@ -0,0 +1,206 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+A HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for

Review comment:
       English is a dumb language, even though HTTP doesn't start with a vowel it should still be "An HTTP". 
   
   ```suggestion
   An HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
   ```

##########
File path: docs/content/docs/modules/http-endpoint.md
##########
@@ -0,0 +1,206 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+A HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  transport:
+    timeouts:
+      call: 1 min
+      read: 10 sec
+      write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+  functions: com.example/*
+  urlPathTemplate: https://bar.foo.com:8080/{function.name}
+  timeouts:
+    call: 1 min
+    read: 10 sec
+    write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace `com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint URL template.
+
+### URL Template
+
+The URL template name may contain template parameters that are filled in based on the function's specific type.

Review comment:
       ```suggestion
   The URL template name may contain template parameters filled in dynamically based on the function's specific type.
   ```

##########
File path: docs/content/docs/modules/overview.md
##########
@@ -0,0 +1,66 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+  - /modules/
+permalink: /modules/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Application Modules
+
+An application module consists of multiple [components]({{< ref "docs/concepts/application-building-blocks" >}})
+that take part in a StateFun application. It includes the endpoints where functions can be reached along with ingress
+and egress definitions.

Review comment:
       ```suggestion
   An application module consists of multiple [components]({{< ref "docs/concepts/application-building-blocks" >}})
   that take part in a StateFun application. It includes the endpoints where the runtime can reach functions, along with ingress and egress definitions.
   ```

##########
File path: docs/content/docs/modules/io/overview.md
##########
@@ -0,0 +1,36 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+  - /modules/io/
+permalink: /modules/io/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# I/O Module
+
+Stateful Functions' I/O modules allow functions to receive and send messages to external systems.
+Based on the concept of Ingress (input) and Egress (output) points, and built on top of the Apache FlinkĀ® connector ecosystem, I/O modules enable functions to interact with the outside world through the style of message passing.
+
+Commonly used I/O modules are bundled into the runtime by default and can be configured direclty via the applications [module configuration]({{< ref "docs/modules/overview" >}}). 
+Additionally, custom connectors for other systems can be [plugged in]({{< ref "docs/modules/io/flink-connectors" >}}) to the runtime.
+
+Keep in mind that to use one of these connectors in an application, additional third party components are usually required, e.g. servers for the data stores or message queues.

Review comment:
       ```suggestion
   Remember, to use one of these connectors in an application, third-party components are usually required, e.g., servers for the data stores or message queues.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink-statefun] sjwiesman closed pull request #263: [FLINK-23600] [docs] Restructure module configuration docs

Posted by GitBox <gi...@apache.org>.
sjwiesman closed pull request #263:
URL: https://github.com/apache/flink-statefun/pull/263


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org