You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2020/07/16 06:47:55 UTC
[camel-k-examples] 01/01: Added an example of Kinesis Firehose with
S3 bucket as destination
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch ENTESB-14043
in repository https://gitbox.apache.org/repos/asf/camel-k-examples.git
commit e5004abf0e872c76472933e5620a02c3f2628c4f
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Jul 16 08:46:27 2020 +0200
Added an example of Kinesis Firehose with S3 bucket as destination
---
.../aws-kinesis-firehose-producer.groovy | 9 +
.../aws-s3-consumer.groovy | 8 +
.../aws-s3-kinesis-firehose-channel.yaml | 7 +
.../aws-s3-kinesis-firehose-source.yaml | 35 ++++
.../aws-services.properties | 6 +
.../readme.didact.md | 184 +++++++++++++++++++++
05-aws-kinesis-firehose-source-basic/readme.md | 1 +
7 files changed, 250 insertions(+)
diff --git a/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy b/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy
new file mode 100644
index 0000000..34939ca
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy
@@ -0,0 +1,9 @@
+// camel-k: dependency=camel-aws2-kinesis
+//
+// Apache Camel Kinesis Consumer
+//
+// This is just a sample producer for AWS that creates 100 messages every 3 seconds
+from('timer:java?period=3000&repeatCount=100')
+ .setBody()
+ .simple('Hello Camel K')
+ .to('aws2-kinesis-firehose:stream?accessKey={{aws.kinesis.accessKey}}&secretKey={{aws.kinesis.secretKey}}®ion={{aws.kinesis.region}}')
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy b/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy
new file mode 100644
index 0000000..f58d8e3
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy
@@ -0,0 +1,8 @@
+// camel-k: dependency=camel-aws2-s3 dependency=camel-base64
+//
+// Apache Camel S3 Consumer
+//
+
+// So we unmarshal it, extract the data element which is in base64 format and decode it
+from('knative:channel/aws-s3-kinesis-firehose')
+ .log('Received: ${body}')
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml
new file mode 100644
index 0000000..f9f2fa7
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml
@@ -0,0 +1,7 @@
+#
+# Knative Channel
+#
+apiVersion: messaging.knative.dev/v1alpha1
+kind: InMemoryChannel
+metadata:
+ name: aws-s3-kinesis-firehose
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml
new file mode 100644
index 0000000..d0ff2ae
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml
@@ -0,0 +1,35 @@
+#
+# Apache Camel AWS S3 Source
+#
+apiVersion: sources.knative.dev/v1alpha1
+kind: CamelSource
+metadata:
+ name: camel-aws-s3-kinesis-firehose-source
+spec:
+ source:
+ integration:
+ configuration:
+ - type: secret
+ value: aws-services
+ dependencies:
+ # Needed for the json part on the flow/steps below
+ - camel:jackson
+ # Needed for the AWS S3 component component
+ - camel:camel-aws2-s3
+ flow:
+ from:
+ uri: aws2-s3:datalake25
+ parameters:
+ secretKey: "{{aws.s3.secretKey}}"
+ accessKey: "{{aws.s3.accessKey}}"
+ region: "{{aws.s3.region}}"
+ moveAfterRead: "false"
+ deleteAfterRead: "false"
+ autocloseBody: "false"
+ steps:
+ - to: "log:received?showAll=true&multiline=true"
+ sink:
+ ref:
+ apiVersion: messaging.knative.dev/v1beta1
+ kind: InMemoryChannel
+ name: aws-s3-kinesis-firehose
diff --git a/05-aws-kinesis-firehose-source-basic/aws-services.properties b/05-aws-kinesis-firehose-source-basic/aws-services.properties
new file mode 100644
index 0000000..20b72aa
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-services.properties
@@ -0,0 +1,6 @@
+aws.kinesis.accessKey=accessKey
+aws.kinesis.secretKey=secretKey
+aws.kinesis.region=region
+aws.s3.accessKey=accessKey
+aws.s3.secretKey=secretKey
+aws.s3.region=region
diff --git a/05-aws-kinesis-firehose-source-basic/readme.didact.md b/05-aws-kinesis-firehose-source-basic/readme.didact.md
new file mode 100644
index 0000000..cd6600c
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/readme.didact.md
@@ -0,0 +1,184 @@
+# Camel AWS Kinesis Firehose Source Basic Example
+
+This example demonstrates how to get started with Camel based Knative sources by showing you some of the most important
+features that you should know before trying to develop more complex examples.
+
+You can find more information about Apache Camel and Apache Camel K on the [official Camel website](https://camel.apache.org).
+
+## Before you begin
+
+Read the general instructions in the [root README.md file](../README.md) for setting up your environment and the Kubernetes cluster before looking at this example.
+
+Make sure you've read the [installation instructions](https://camel.apache.org/camel-k/latest/installation/installation.html) for your specific
+cluster before starting the example.
+
+You should open this file with [Didact](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-didact) if available on your IDE.
+
+You must create an AWS Kinesis Firehose delivery stream named `stream` and point it to an S3 bucket called `datalake` followed by a random number on your AWS account.
+
+## Requirements
+
+<a href='didact://?commandId=vscode.didact.validateAllRequirements' title='Validate all requirements!'><button>Validate all Requirements at Once!</button></a>
+
+**Kubectl CLI**
+
+The Kubernetes `kubectl` CLI tool will be used to interact with the Kubernetes cluster.
+
+[Check if the Kubectl CLI is installed](didact://?commandId=vscode.didact.cliCommandSuccessful&text=kubectl-requirements-status$$kubectl%20help&completion=Checked%20kubectl%20tool%20availability "Tests to see if `kubectl help` returns a 0 return code"){.didact}
+
+*Status: unknown*{#kubectl-requirements-status}
+
+**Connection to a Kubernetes cluster**
+
+You need to connect to a Kubernetes cluster in order to run the example.
+
+[Check if you're connected to a Kubernetes cluster](didact://?commandId=vscode.didact.cliCommandSuccessful&text=cluster-requirements-status$$kubectl%20get%20pod&completion=Checked%20Kubernetes%20connection "Tests to see if `kubectl get pod` returns a 0 return code"){.didact}
+
+*Status: unknown*{#cluster-requirements-status}
+
+**Apache Camel K CLI ("kamel")**
+
+You need the Apache Camel K CLI ("kamel") in order to access all Camel K features.
+
+[Check if the Apache Camel K CLI ("kamel") is installed](didact://?commandId=vscode.didact.requirementCheck&text=kamel-requirements-status$$kamel%20version$$Camel%20K%20Client&completion=Checked%20if%20Camel%20K%20CLI%20is%20available%20on%20this%20system. "Tests to see if `kamel version` returns a result"){.didact}
+
+*Status: unknown*{#kamel-requirements-status}
+
+**Knative installed on the cluster**
+
+The cluster also needs to have Knative installed and working. Refer to the [official Knative documentation](https://knative.dev/v0.15-docs/install/) for information on how to install it in your cluster.
+
+[Check if the Knative Serving is installed](didact://?commandId=vscode.didact.requirementCheck&text=kserving-project-check$$kubectl%20api-resources%20--api-group=serving.knative.dev$$kservice%2Cksvc&completion=Verified%20Knative%20services%20installation. "Verifies if Knative Serving is installed"){.didact}
+
+*Status: unknown*{#kserving-project-check}
+
+[Check if the Knative Eventing is installed](didact://?commandId=vscode.didact.requirementCheck&text=keventing-project-check$$kubectl%20api-resources%20--api-group=messaging.knative.dev$$inmemorychannels&completion=Verified%20Knative%20eventing%20services%20installation. "Verifies if Knative Eventing is installed"){.didact}
+
+*Status: unknown*{#keventing-project-check}
+
+**Knative Camel Source installed on the cluster**
+
+The cluster also needs to have installed the Knative Camel Source from the camel.yaml in the [Eventing Sources release page](https://github.com/knative/eventing-contrib/releases/tag/v0.15.0)
+
+[Check if the Knative Camel Source is installed](didact://?commandId=vscode.didact.requirementCheck&text=kservice-project-check$$kubectl%20api-resources%20--api-group=sources.knative.dev$$camelsources&completion=Verified%20Knative%20Camel%20Source%20installation. "Verifies if Knative Camel Source is installed"){.didact}
+
+*Status: unknown*{#kservice-project-check}
+
+### Optional Requirements
+
+The following requirements are optional. They don't prevent the execution of the demo, but may make it easier to follow.
+
+**VS Code Extension Pack for Apache Camel**
+
+The VS Code Extension Pack for Apache Camel provides a collection of useful tools for Apache Camel K developers,
+such as code completion and integrated lifecycle management. They are **recommended** for the tutorial, but they are **not**
+required.
+
+You can install it from the VS Code Extensions marketplace.
+
+[Check if the VS Code Extension Pack for Apache Camel by Red Hat is installed](didact://?commandId=vscode.didact.extensionRequirementCheck&text=extension-requirement-status$$redhat.apache-camel-extension-pack&completion=Camel%20extension%20pack%20is%20available%20on%20this%20system. "Checks the VS Code workspace to make sure the extension pack is installed"){.didact}
+
+*Status: unknown*{#extension-requirement-status}
+
+## 1. Preparing the namespace
+
+Let's open a terminal and go to the example directory:
+
+```
+cd 05-aws-kinesis-firehose-source-basic
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$cd%2005-aws-kinesis-firehose-source-basic&completion=Executed%20command. "Opens a new terminal and sends the command above"){.didact})
+
+
+We're going to create a namespace named `aws-kinesis-firehose-event-source` for running the example. To create it, execute the following command:
+
+```
+kubectl create namespace aws-kinesis-firehose-event-source
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20create%20namespace%20aws-kinesis-firehose-event-source&completion=New%20project%20creation. "Opens a new terminal and sends the command above"){.didact})
+
+Now we can set the `aws-kinesis-firehose-event-source` namespace as default namespace for the following commands:
+
+```
+kubectl config set-context --current --namespace=aws-kinesis-firehose-event-source
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20config%20set-context%20--current%20--namespace%3Daws-kinesis-firehose-event-source&completion=New%20project%20creation. "Opens a new terminal and sends the command above"){.didact})
+
+You need to install Camel K in the `aws-kinesis-firehose-event-source` namespace (or globally in the whole cluster).
+In many settings (e.g. OpenShift, CRC), it's sufficient to execute the following command to install Camel K:
+
+```
+kamel install
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20install&completion=Installing%20Camel%20K. "Opens a new terminal and sends the command above"){.didact})
+
+NOTE: The `kamel install` command requires some prerequisites to be successful in some situations, e.g. you need to enable the registry addon on Minikube. Refer to the [Camel K install guide](https://camel.apache.org/camel-k/latest/installation/installation.html) for cluster-specific instructions.
+
+To check that Camel K is installed we'll retrieve the IntegrationPlatform object from the namespace:
+
+```
+kubectl get integrationplatform
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20get%20integrationplatform&completion=Executed%20Command. "Opens a new terminal and sends the command above"){.didact})
+
+You should find an IntegrationPlatform in status `Ready`.
+
+You can now proceed to the next section.
+
+## 2. Preparing the environment
+
+This repository contains a simple [aws-services.properties](didact://?commandId=vscode.openFolder&projectFilePath=04-aws-kinesis-source-basic/aws-services.properties&completion=Opened%20the%aws-services.properties%20file "Opens the aws-kinesis.properties file"){.didact} that contains the access key and secret key for accessing the AWS Kinesis stream.
+
+```
+kubectl create secret generic aws-services --from-file=aws-services.properties
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20create%20secret%20generic%20aws-services%20--from-file%3Daws-services.properties&completion=secret%20%22aws-services%22%20created. "Create a secret with AWS Kinesis credentials"){.didact})
+
+As the example levareges [Knative Eventing channels](https://knative.dev/v0.15-docs/eventing/channels/), we need to create the one that the example will use:
+
+```
+kubectl apply -f aws-s3-kinesis-firehose-channel.yaml
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20apply%20-f%20aws-s3-kinesis-firehose-channel.yaml&completion=inmemorychannel.messaging.knative.dev/aws-s3-kinesis-firehose$20created. "Create a Knative InMemoryChannel named aws-kinesis"){.didact})
+
+
+## 2. Running a Camel Source
+
+This repository contains a simple Camel Source based on the [AWS Kinesis component](https://camel.apache.org/components/latest/aws-kinesis-component.html) that forward streaming events received on the AWS Kinesis delivery stream to a Knative channel named `aws-s3-kinesis-firehose`.
+
+Use the following command to deploy the Camel Source:
+
+```
+kubectl apply -f aws-s3-kinesis-firehose-source.yaml
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20apply%20-f%20aws-s3-kinesis-firehose-source.yaml&completion=camelsource.sources.knative.dev/camel-aws-s3-kinesis-firehose-source%20created. "Opens a new terminal and sends the command above"){.didact})
+
+## 2. Running a basic integration to create Kinesis events for consumption by the Camel Source
+
+You need a producer adding data to a Kinesis stream to try this example. This integration
+comes with a sample producer that will send 100 messages with the text `Hello Camel K`
+every 3 seconds.
+
+```
+kamel run --secret aws-services aws-kinesis-firehose-producer.groovy
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20run%20--secret%20aws-kinesis%20aws-kinesis-firehose-producer.groovy&completion=Camel%20K%20aws-kinesis-firehose-producer%20integration%20run%20in%20dev%20mode. "Opens a new terminal and sends the command above"){.didact})
+
+If everything is ok, after the build phase finishes, you should see the Camel integration running.
+
+## 3. Running a basic integration to forward Kinesis events to the console
+
+```
+kamel run aws-s3-consumer.groovy --dev
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20run%20aws-s3-consumer.groovy%20--dev&completion=Camel%20K%20aws-s3-consumer%20integration%20run%20in%20dev%20mode. "Opens a new terminal and sends the command above"){.didact})
+
+If everything is ok, after the build phase finishes, you should see the Camel integration running.
+
+## 4. Uninstall
+
+To cleanup everything, execute the following command:
+
+```kubectl delete namespace aws-kinesis-firehose-event-source```
+
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20delete%20namespace%20aws-kinesis-source-basic&completion=Removed%20the%20namespace%20from%20the%20cluster. "Cleans up the cluster after running the example"){.didact})
diff --git a/05-aws-kinesis-firehose-source-basic/readme.md b/05-aws-kinesis-firehose-source-basic/readme.md
new file mode 120000
index 0000000..c0bf82f
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/readme.md
@@ -0,0 +1 @@
+readme.didact.md
\ No newline at end of file