You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by sj...@apache.org on 2021/04/01 19:58:04 UTC

[flink-statefun-playground] branch dev updated: [FLINK-21828] Add a tutorial for AWS Lambda function deployments

This is an automated email from the ASF dual-hosted git repository.

sjwiesman pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/flink-statefun-playground.git


The following commit(s) were added to refs/heads/dev by this push:
     new e619c37  [FLINK-21828] Add a tutorial for AWS Lambda function deployments
e619c37 is described below

commit e619c371c7424606de436098fa27f10d219c82e3
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
AuthorDate: Fri Mar 26 13:44:33 2021 +0800

    [FLINK-21828] Add a tutorial for AWS Lambda function deployments
    
    This closes #10
---
 deployments/.gitignore                             |   3 +
 deployments/aws-lambda/README.md                   |  46 ++++++++++
 deployments/aws-lambda/pom.xml                     | 101 +++++++++++++++++++++
 .../aws/APIGatewayStateFunHandlerProxy.java        |  63 +++++++++++++
 .../playground/aws/GreeterAwsLambdaHandler.java    |  55 +++++++++++
 .../flink/statefun/playground/aws/GreeterFn.java   |  38 ++++++++
 6 files changed, 306 insertions(+)

diff --git a/deployments/.gitignore b/deployments/.gitignore
new file mode 100644
index 0000000..a65bf38
--- /dev/null
+++ b/deployments/.gitignore
@@ -0,0 +1,3 @@
+*.iml
+target/
+.idea/
diff --git a/deployments/aws-lambda/README.md b/deployments/aws-lambda/README.md
new file mode 100644
index 0000000..4c18461
--- /dev/null
+++ b/deployments/aws-lambda/README.md
@@ -0,0 +1,46 @@
+# Serving StateFun Functions with AWS Lambda
+
+This tutorial is intended to demonstrate how you can serve your functions with [AWS Lambda](https://aws.amazon.com/lambda/),
+a popular FaaS platform for running code with serverless deployments. With the rapid elasticity and simplicity of FaaS
+platforms, this makes it very easy to scale out your StateFun functions.
+
+If you are still new to StateFun and unfamiliar with the overall architecture of a StateFun application, we highly
+recommend to first take a look at some of the demo tutorials under the language-specific directories (such as the
+Greeter examples for [Java](../../java/greeter) or [Python](../../python/greeter)).
+
+## Deployment Steps
+
+### Build an AWS Lambda code package
+
+We're demonstrating Java here, so the code package you submit to AWS Lambda will be a JAR.
+
+```
+$ mvn clean install
+# or, if you don't have Maven installed and would like to use Docker:
+$ docker run -it --rm --name build -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.6.3-jdk-8 mvn clean install
+```
+
+This builds an uber JAR containing all required dependencies by the AWS Lambda runtime at `target/aws-lambda-example*.jar`.
+
+### Deploy to AWS Lambda
+
+- Upload this JAR to AWS Lambda. You can do this [manually via the web console](https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html),
+or using the [AWS CLI](https://aws.amazon.com/cli/).
+
+- The handler method should be set to `org.apache.flink.statefun.playground.aws.GreeterAwsLambdaHandler::handleRequest`.
+That refers to the entrypoint handle method you'll find in the [code](src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java).
+
+- After you upload and create the new AWS Lambda function, configure it to be
+[triggered by an API Gateway HTTP endpoint](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html).
+
+After completing this step, you should now have a HTTP endpoint URL (e.g. https://abcdxyz.execute-api.us-west-1.amazonaws.com/default/my-aws-lambda-fn)
+for invoking your StateFun functions.
+
+### Configure StateFun Module Specification
+
+The AWS Lambda endpoint URL should be configured as a function endpoint in your [Module Specification]().
+
+Note how you may use [templating]() in the endpoint URL for multiple functions under the same namespace.
+This allows you to add new AWS Lambda functions, potentially individually serving different StateFun functions, under
+separate paths of the same AWS API Gateway endpoint. This provides flexibility to upgrade your StateFun application
+by dynamically adding new functions without having to restart the StateFun runtime.
diff --git a/deployments/aws-lambda/pom.xml b/deployments/aws-lambda/pom.xml
new file mode 100644
index 0000000..82228b7
--- /dev/null
+++ b/deployments/aws-lambda/pom.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.flink</groupId>
+    <artifactId>aws-lambda-example</artifactId>
+    <version>3.0-SNAPSHOT</version>
+    <packaging>jar</packaging>
+
+    <properties>
+        <statefun.version>3.0-SNAPSHOT</statefun.version>
+        <maven.compiler.source>8</maven.compiler.source>
+        <maven.compiler.target>8</maven.compiler.target>
+    </properties>
+
+    <dependencies>
+        <!-- StateFun Java SDK -->
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>statefun-sdk-java</artifactId>
+            <version>${statefun.version}</version>
+        </dependency>
+
+        <!-- AWS Lambda -->
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-lambda-java-core</artifactId>
+            <version>1.2.1</version>
+        </dependency>
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-lambda-java-events</artifactId>
+            <version>2.2.9</version>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <!-- Build a fat jar -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <version>3.2.2</version>
+                <configuration>
+                    <createDependencyReducedPom>false</createDependencyReducedPom>
+                </configuration>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+
+            <!-- Java code style -->
+            <plugin>
+                <groupId>com.diffplug.spotless</groupId>
+                <artifactId>spotless-maven-plugin</artifactId>
+                <version>1.20.0</version>
+                <configuration>
+                    <java>
+                        <googleJavaFormat>
+                            <version>1.7</version>
+                            <style>GOOGLE</style>
+                        </googleJavaFormat>
+                        <removeUnusedImports/>
+                    </java>
+                </configuration>
+                <executions>
+                    <execution>
+                        <id>spotless-check</id>
+                        <phase>verify</phase>
+                        <goals>
+                            <goal>check</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java
new file mode 100644
index 0000000..7d4b7a5
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyRequestEvent;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyResponseEvent;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.flink.statefun.sdk.java.handler.RequestReplyHandler;
+import org.apache.flink.statefun.sdk.java.slice.Slice;
+import org.apache.flink.statefun.sdk.java.slice.Slices;
+
+/** Delegates AWS API Gateway requests to a StateFun {@link RequestReplyHandler}. */
+final class APIGatewayStateFunHandlerProxy {
+
+  /** Standard required response headers for StateFun */
+  private static final Map<String, String> RESPONSE_HEADER = new HashMap<>(1);
+
+  static {
+    RESPONSE_HEADER.put("Content-Type", "application/octet-stream");
+  }
+
+  private final RequestReplyHandler stateFunHandler;
+
+  APIGatewayStateFunHandlerProxy(RequestReplyHandler stateFunHandler) {
+    this.stateFunHandler = Objects.requireNonNull(stateFunHandler);
+  }
+
+  public APIGatewayV2ProxyResponseEvent handle(APIGatewayV2ProxyRequestEvent request) {
+    // Binary blobs (the invocation request) are always base64 encoded by the API Gateway
+    final byte[] decoded = Base64.getDecoder().decode(request.getBody());
+
+    final Slice responseSlice = stateFunHandler.handle(Slices.wrap(decoded)).join();
+    return createResponse(responseSlice);
+  }
+
+  private static APIGatewayV2ProxyResponseEvent createResponse(Slice responseSlice) {
+    final APIGatewayV2ProxyResponseEvent response = new APIGatewayV2ProxyResponseEvent();
+    response.setHeaders(RESPONSE_HEADER);
+    response.setIsBase64Encoded(true);
+    response.setStatusCode(200);
+    response.setBody(Base64.getEncoder().encodeToString(responseSlice.toByteArray()));
+
+    return response;
+  }
+}
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java
new file mode 100644
index 0000000..05366d9
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import com.amazonaws.services.lambda.runtime.Context;
+import com.amazonaws.services.lambda.runtime.RequestHandler;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyRequestEvent;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyResponseEvent;
+import org.apache.flink.statefun.sdk.java.StatefulFunctions;
+import org.apache.flink.statefun.sdk.java.handler.RequestReplyHandler;
+
+/**
+ * An <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> {@link RequestHandler} implementation
+ * for serving StateFun functions through <a href="https://aws.amazon.com/api-gateway/">AWS API
+ * Gateway</a>.
+ *
+ * <p>It is straightforward to integrate a StateFun {@link RequestReplyHandler} within an AWS
+ * Lambda's {@link RequestHandler}. As shown below, all AWS API Gateway HTTP requests received by
+ * the AWS Lambda function is simply delegated to a StateFun {@link RequestReplyHandler} via a
+ * {@link APIGatewayStateFunHandlerProxy}. Take a closer look at the {@link
+ * APIGatewayStateFunHandlerProxy} on details of how the request and response was handled.
+ */
+public final class GreeterAwsLambdaHandler
+    implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
+
+  private static final APIGatewayStateFunHandlerProxy HANDLER;
+
+  static {
+    final StatefulFunctions functions = new StatefulFunctions();
+    functions.withStatefulFunction(GreeterFn.SPEC);
+
+    HANDLER = new APIGatewayStateFunHandlerProxy(functions.requestReplyHandler());
+  }
+
+  @Override
+  public APIGatewayV2ProxyResponseEvent handleRequest(
+      APIGatewayV2ProxyRequestEvent event, Context context) {
+    return HANDLER.handle(event);
+  }
+}
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java
new file mode 100644
index 0000000..258beea
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.statefun.sdk.java.Context;
+import org.apache.flink.statefun.sdk.java.StatefulFunction;
+import org.apache.flink.statefun.sdk.java.StatefulFunctionSpec;
+import org.apache.flink.statefun.sdk.java.TypeName;
+import org.apache.flink.statefun.sdk.java.message.Message;
+
+public final class GreeterFn implements StatefulFunction {
+
+  static final TypeName TYPENAME = TypeName.typeNameOf("aws.lambda.example", "greeter");
+  static final StatefulFunctionSpec SPEC =
+      StatefulFunctionSpec.builder(TYPENAME).withSupplier(GreeterFn::new).build();
+
+  @Override
+  public CompletableFuture<Void> apply(Context context, Message message) {
+    System.out.println("Welcome, " + message.asUtf8String() + "!");
+    return context.done();
+  }
+}