You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by sj...@apache.org on 2021/04/01 19:58:04 UTC
[flink-statefun-playground] branch dev updated: [FLINK-21828] Add a
tutorial for AWS Lambda function deployments
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/flink-statefun-playground.git
The following commit(s) were added to refs/heads/dev by this push:
new e619c37 [FLINK-21828] Add a tutorial for AWS Lambda function deployments
e619c37 is described below
commit e619c371c7424606de436098fa27f10d219c82e3
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
AuthorDate: Fri Mar 26 13:44:33 2021 +0800
[FLINK-21828] Add a tutorial for AWS Lambda function deployments
This closes #10
---
deployments/.gitignore | 3 +
deployments/aws-lambda/README.md | 46 ++++++++++
deployments/aws-lambda/pom.xml | 101 +++++++++++++++++++++
.../aws/APIGatewayStateFunHandlerProxy.java | 63 +++++++++++++
.../playground/aws/GreeterAwsLambdaHandler.java | 55 +++++++++++
.../flink/statefun/playground/aws/GreeterFn.java | 38 ++++++++
6 files changed, 306 insertions(+)
diff --git a/deployments/.gitignore b/deployments/.gitignore
new file mode 100644
index 0000000..a65bf38
--- /dev/null
+++ b/deployments/.gitignore
@@ -0,0 +1,3 @@
+*.iml
+target/
+.idea/
diff --git a/deployments/aws-lambda/README.md b/deployments/aws-lambda/README.md
new file mode 100644
index 0000000..4c18461
--- /dev/null
+++ b/deployments/aws-lambda/README.md
@@ -0,0 +1,46 @@
+# Serving StateFun Functions with AWS Lambda
+
+This tutorial is intended to demonstrate how you can serve your functions with [AWS Lambda](https://aws.amazon.com/lambda/),
+a popular FaaS platform for running code with serverless deployments. With the rapid elasticity and simplicity of FaaS
+platforms, this makes it very easy to scale out your StateFun functions.
+
+If you are still new to StateFun and unfamiliar with the overall architecture of a StateFun application, we highly
+recommend to first take a look at some of the demo tutorials under the language-specific directories (such as the
+Greeter examples for [Java](../../java/greeter) or [Python](../../python/greeter)).
+
+## Deployment Steps
+
+### Build an AWS Lambda code package
+
+We're demonstrating Java here, so the code package you submit to AWS Lambda will be a JAR.
+
+```
+$ mvn clean install
+# or, if you don't have Maven installed and would like to use Docker:
+$ docker run -it --rm --name build -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.6.3-jdk-8 mvn clean install
+```
+
+This builds an uber JAR containing all required dependencies by the AWS Lambda runtime at `target/aws-lambda-example*.jar`.
+
+### Deploy to AWS Lambda
+
+- Upload this JAR to AWS Lambda. You can do this [manually via the web console](https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html),
+or using the [AWS CLI](https://aws.amazon.com/cli/).
+
+- The handler method should be set to `org.apache.flink.statefun.playground.aws.GreeterAwsLambdaHandler::handleRequest`.
+That refers to the entrypoint handle method you'll find in the [code](src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java).
+
+- After you upload and create the new AWS Lambda function, configure it to be
+[triggered by an API Gateway HTTP endpoint](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html).
+
+After completing this step, you should now have a HTTP endpoint URL (e.g. https://abcdxyz.execute-api.us-west-1.amazonaws.com/default/my-aws-lambda-fn)
+for invoking your StateFun functions.
+
+### Configure StateFun Module Specification
+
+The AWS Lambda endpoint URL should be configured as a function endpoint in your [Module Specification]().
+
+Note how you may use [templating]() in the endpoint URL for multiple functions under the same namespace.
+This allows you to add new AWS Lambda functions, potentially individually serving different StateFun functions, under
+separate paths of the same AWS API Gateway endpoint. This provides flexibility to upgrade your StateFun application
+by dynamically adding new functions without having to restart the StateFun runtime.
diff --git a/deployments/aws-lambda/pom.xml b/deployments/aws-lambda/pom.xml
new file mode 100644
index 0000000..82228b7
--- /dev/null
+++ b/deployments/aws-lambda/pom.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.apache.flink</groupId>
+ <artifactId>aws-lambda-example</artifactId>
+ <version>3.0-SNAPSHOT</version>
+ <packaging>jar</packaging>
+
+ <properties>
+ <statefun.version>3.0-SNAPSHOT</statefun.version>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <!-- StateFun Java SDK -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>statefun-sdk-java</artifactId>
+ <version>${statefun.version}</version>
+ </dependency>
+
+ <!-- AWS Lambda -->
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-lambda-java-core</artifactId>
+ <version>1.2.1</version>
+ </dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-lambda-java-events</artifactId>
+ <version>2.2.9</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <!-- Build a fat jar -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.2.2</version>
+ <configuration>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+ <!-- Java code style -->
+ <plugin>
+ <groupId>com.diffplug.spotless</groupId>
+ <artifactId>spotless-maven-plugin</artifactId>
+ <version>1.20.0</version>
+ <configuration>
+ <java>
+ <googleJavaFormat>
+ <version>1.7</version>
+ <style>GOOGLE</style>
+ </googleJavaFormat>
+ <removeUnusedImports/>
+ </java>
+ </configuration>
+ <executions>
+ <execution>
+ <id>spotless-check</id>
+ <phase>verify</phase>
+ <goals>
+ <goal>check</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java
new file mode 100644
index 0000000..7d4b7a5
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/APIGatewayStateFunHandlerProxy.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyRequestEvent;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyResponseEvent;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.flink.statefun.sdk.java.handler.RequestReplyHandler;
+import org.apache.flink.statefun.sdk.java.slice.Slice;
+import org.apache.flink.statefun.sdk.java.slice.Slices;
+
+/** Delegates AWS API Gateway requests to a StateFun {@link RequestReplyHandler}. */
+final class APIGatewayStateFunHandlerProxy {
+
+ /** Standard required response headers for StateFun */
+ private static final Map<String, String> RESPONSE_HEADER = new HashMap<>(1);
+
+ static {
+ RESPONSE_HEADER.put("Content-Type", "application/octet-stream");
+ }
+
+ private final RequestReplyHandler stateFunHandler;
+
+ APIGatewayStateFunHandlerProxy(RequestReplyHandler stateFunHandler) {
+ this.stateFunHandler = Objects.requireNonNull(stateFunHandler);
+ }
+
+ public APIGatewayV2ProxyResponseEvent handle(APIGatewayV2ProxyRequestEvent request) {
+ // Binary blobs (the invocation request) are always base64 encoded by the API Gateway
+ final byte[] decoded = Base64.getDecoder().decode(request.getBody());
+
+ final Slice responseSlice = stateFunHandler.handle(Slices.wrap(decoded)).join();
+ return createResponse(responseSlice);
+ }
+
+ private static APIGatewayV2ProxyResponseEvent createResponse(Slice responseSlice) {
+ final APIGatewayV2ProxyResponseEvent response = new APIGatewayV2ProxyResponseEvent();
+ response.setHeaders(RESPONSE_HEADER);
+ response.setIsBase64Encoded(true);
+ response.setStatusCode(200);
+ response.setBody(Base64.getEncoder().encodeToString(responseSlice.toByteArray()));
+
+ return response;
+ }
+}
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java
new file mode 100644
index 0000000..05366d9
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterAwsLambdaHandler.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import com.amazonaws.services.lambda.runtime.Context;
+import com.amazonaws.services.lambda.runtime.RequestHandler;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyRequestEvent;
+import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyResponseEvent;
+import org.apache.flink.statefun.sdk.java.StatefulFunctions;
+import org.apache.flink.statefun.sdk.java.handler.RequestReplyHandler;
+
+/**
+ * An <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> {@link RequestHandler} implementation
+ * for serving StateFun functions through <a href="https://aws.amazon.com/api-gateway/">AWS API
+ * Gateway</a>.
+ *
+ * <p>It is straightforward to integrate a StateFun {@link RequestReplyHandler} within an AWS
+ * Lambda's {@link RequestHandler}. As shown below, all AWS API Gateway HTTP requests received by
+ * the AWS Lambda function is simply delegated to a StateFun {@link RequestReplyHandler} via a
+ * {@link APIGatewayStateFunHandlerProxy}. Take a closer look at the {@link
+ * APIGatewayStateFunHandlerProxy} on details of how the request and response was handled.
+ */
+public final class GreeterAwsLambdaHandler
+ implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
+
+ private static final APIGatewayStateFunHandlerProxy HANDLER;
+
+ static {
+ final StatefulFunctions functions = new StatefulFunctions();
+ functions.withStatefulFunction(GreeterFn.SPEC);
+
+ HANDLER = new APIGatewayStateFunHandlerProxy(functions.requestReplyHandler());
+ }
+
+ @Override
+ public APIGatewayV2ProxyResponseEvent handleRequest(
+ APIGatewayV2ProxyRequestEvent event, Context context) {
+ return HANDLER.handle(event);
+ }
+}
diff --git a/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java
new file mode 100644
index 0000000..258beea
--- /dev/null
+++ b/deployments/aws-lambda/src/main/java/org/apache/flink/statefun/playground/aws/GreeterFn.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.statefun.playground.aws;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.statefun.sdk.java.Context;
+import org.apache.flink.statefun.sdk.java.StatefulFunction;
+import org.apache.flink.statefun.sdk.java.StatefulFunctionSpec;
+import org.apache.flink.statefun.sdk.java.TypeName;
+import org.apache.flink.statefun.sdk.java.message.Message;
+
+public final class GreeterFn implements StatefulFunction {
+
+ static final TypeName TYPENAME = TypeName.typeNameOf("aws.lambda.example", "greeter");
+ static final StatefulFunctionSpec SPEC =
+ StatefulFunctionSpec.builder(TYPENAME).withSupplier(GreeterFn::new).build();
+
+ @Override
+ public CompletableFuture<Void> apply(Context context, Message message) {
+ System.out.println("Welcome, " + message.asUtf8String() + "!");
+ return context.done();
+ }
+}