You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2020/07/16 06:47:54 UTC
[camel-k-examples] branch ENTESB-14043 created (now e5004ab)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch ENTESB-14043
in repository https://gitbox.apache.org/repos/asf/camel-k-examples.git.
at e5004ab Added an example of Kinesis Firehose with S3 bucket as destination
This branch includes the following new commits:
new e5004ab Added an example of Kinesis Firehose with S3 bucket as destination
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-k-examples] 01/01: Added an example of Kinesis Firehose with
S3 bucket as destination
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch ENTESB-14043
in repository https://gitbox.apache.org/repos/asf/camel-k-examples.git
commit e5004abf0e872c76472933e5620a02c3f2628c4f
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Jul 16 08:46:27 2020 +0200
Added an example of Kinesis Firehose with S3 bucket as destination
---
.../aws-kinesis-firehose-producer.groovy | 9 +
.../aws-s3-consumer.groovy | 8 +
.../aws-s3-kinesis-firehose-channel.yaml | 7 +
.../aws-s3-kinesis-firehose-source.yaml | 35 ++++
.../aws-services.properties | 6 +
.../readme.didact.md | 184 +++++++++++++++++++++
05-aws-kinesis-firehose-source-basic/readme.md | 1 +
7 files changed, 250 insertions(+)
diff --git a/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy b/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy
new file mode 100644
index 0000000..34939ca
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-kinesis-firehose-producer.groovy
@@ -0,0 +1,9 @@
+// camel-k: dependency=camel-aws2-kinesis
+//
+// Apache Camel Kinesis Consumer
+//
+// This is just a sample producer for AWS that creates 100 messages every 3 seconds
+from('timer:java?period=3000&repeatCount=100')
+ .setBody()
+ .simple('Hello Camel K')
+ .to('aws2-kinesis-firehose:stream?accessKey={{aws.kinesis.accessKey}}&secretKey={{aws.kinesis.secretKey}}®ion={{aws.kinesis.region}}')
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy b/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy
new file mode 100644
index 0000000..f58d8e3
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-consumer.groovy
@@ -0,0 +1,8 @@
+// camel-k: dependency=camel-aws2-s3 dependency=camel-base64
+//
+// Apache Camel S3 Consumer
+//
+
+// So we unmarshal it, extract the data element which is in base64 format and decode it
+from('knative:channel/aws-s3-kinesis-firehose')
+ .log('Received: ${body}')
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml
new file mode 100644
index 0000000..f9f2fa7
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-channel.yaml
@@ -0,0 +1,7 @@
+#
+# Knative Channel
+#
+apiVersion: messaging.knative.dev/v1alpha1
+kind: InMemoryChannel
+metadata:
+ name: aws-s3-kinesis-firehose
diff --git a/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml
new file mode 100644
index 0000000..d0ff2ae
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-s3-kinesis-firehose-source.yaml
@@ -0,0 +1,35 @@
+#
+# Apache Camel AWS S3 Source
+#
+apiVersion: sources.knative.dev/v1alpha1
+kind: CamelSource
+metadata:
+ name: camel-aws-s3-kinesis-firehose-source
+spec:
+ source:
+ integration:
+ configuration:
+ - type: secret
+ value: aws-services
+ dependencies:
+ # Needed for the json part on the flow/steps below
+ - camel:jackson
+ # Needed for the AWS S3 component component
+ - camel:camel-aws2-s3
+ flow:
+ from:
+ uri: aws2-s3:datalake25
+ parameters:
+ secretKey: "{{aws.s3.secretKey}}"
+ accessKey: "{{aws.s3.accessKey}}"
+ region: "{{aws.s3.region}}"
+ moveAfterRead: "false"
+ deleteAfterRead: "false"
+ autocloseBody: "false"
+ steps:
+ - to: "log:received?showAll=true&multiline=true"
+ sink:
+ ref:
+ apiVersion: messaging.knative.dev/v1beta1
+ kind: InMemoryChannel
+ name: aws-s3-kinesis-firehose
diff --git a/05-aws-kinesis-firehose-source-basic/aws-services.properties b/05-aws-kinesis-firehose-source-basic/aws-services.properties
new file mode 100644
index 0000000..20b72aa
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/aws-services.properties
@@ -0,0 +1,6 @@
+aws.kinesis.accessKey=accessKey
+aws.kinesis.secretKey=secretKey
+aws.kinesis.region=region
+aws.s3.accessKey=accessKey
+aws.s3.secretKey=secretKey
+aws.s3.region=region
diff --git a/05-aws-kinesis-firehose-source-basic/readme.didact.md b/05-aws-kinesis-firehose-source-basic/readme.didact.md
new file mode 100644
index 0000000..cd6600c
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/readme.didact.md
@@ -0,0 +1,184 @@
+# Camel AWS Kinesis Firehose Source Basic Example
+
+This example demonstrates how to get started with Camel based Knative sources by showing you some of the most important
+features that you should know before trying to develop more complex examples.
+
+You can find more information about Apache Camel and Apache Camel K on the [official Camel website](https://camel.apache.org).
+
+## Before you begin
+
+Read the general instructions in the [root README.md file](../README.md) for setting up your environment and the Kubernetes cluster before looking at this example.
+
+Make sure you've read the [installation instructions](https://camel.apache.org/camel-k/latest/installation/installation.html) for your specific
+cluster before starting the example.
+
+You should open this file with [Didact](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-didact) if available on your IDE.
+
+You must create an AWS Kinesis Firehose delivery stream named `stream` and point it to an S3 bucket called `datalake` followed by a random number on your AWS account.
+
+## Requirements
+
+<a href='didact://?commandId=vscode.didact.validateAllRequirements' title='Validate all requirements!'><button>Validate all Requirements at Once!</button></a>
+
+**Kubectl CLI**
+
+The Kubernetes `kubectl` CLI tool will be used to interact with the Kubernetes cluster.
+
+[Check if the Kubectl CLI is installed](didact://?commandId=vscode.didact.cliCommandSuccessful&text=kubectl-requirements-status$$kubectl%20help&completion=Checked%20kubectl%20tool%20availability "Tests to see if `kubectl help` returns a 0 return code"){.didact}
+
+*Status: unknown*{#kubectl-requirements-status}
+
+**Connection to a Kubernetes cluster**
+
+You need to connect to a Kubernetes cluster in order to run the example.
+
+[Check if you're connected to a Kubernetes cluster](didact://?commandId=vscode.didact.cliCommandSuccessful&text=cluster-requirements-status$$kubectl%20get%20pod&completion=Checked%20Kubernetes%20connection "Tests to see if `kubectl get pod` returns a 0 return code"){.didact}
+
+*Status: unknown*{#cluster-requirements-status}
+
+**Apache Camel K CLI ("kamel")**
+
+You need the Apache Camel K CLI ("kamel") in order to access all Camel K features.
+
+[Check if the Apache Camel K CLI ("kamel") is installed](didact://?commandId=vscode.didact.requirementCheck&text=kamel-requirements-status$$kamel%20version$$Camel%20K%20Client&completion=Checked%20if%20Camel%20K%20CLI%20is%20available%20on%20this%20system. "Tests to see if `kamel version` returns a result"){.didact}
+
+*Status: unknown*{#kamel-requirements-status}
+
+**Knative installed on the cluster**
+
+The cluster also needs to have Knative installed and working. Refer to the [official Knative documentation](https://knative.dev/v0.15-docs/install/) for information on how to install it in your cluster.
+
+[Check if the Knative Serving is installed](didact://?commandId=vscode.didact.requirementCheck&text=kserving-project-check$$kubectl%20api-resources%20--api-group=serving.knative.dev$$kservice%2Cksvc&completion=Verified%20Knative%20services%20installation. "Verifies if Knative Serving is installed"){.didact}
+
+*Status: unknown*{#kserving-project-check}
+
+[Check if the Knative Eventing is installed](didact://?commandId=vscode.didact.requirementCheck&text=keventing-project-check$$kubectl%20api-resources%20--api-group=messaging.knative.dev$$inmemorychannels&completion=Verified%20Knative%20eventing%20services%20installation. "Verifies if Knative Eventing is installed"){.didact}
+
+*Status: unknown*{#keventing-project-check}
+
+**Knative Camel Source installed on the cluster**
+
+The cluster also needs to have installed the Knative Camel Source from the camel.yaml in the [Eventing Sources release page](https://github.com/knative/eventing-contrib/releases/tag/v0.15.0)
+
+[Check if the Knative Camel Source is installed](didact://?commandId=vscode.didact.requirementCheck&text=kservice-project-check$$kubectl%20api-resources%20--api-group=sources.knative.dev$$camelsources&completion=Verified%20Knative%20Camel%20Source%20installation. "Verifies if Knative Camel Source is installed"){.didact}
+
+*Status: unknown*{#kservice-project-check}
+
+### Optional Requirements
+
+The following requirements are optional. They don't prevent the execution of the demo, but may make it easier to follow.
+
+**VS Code Extension Pack for Apache Camel**
+
+The VS Code Extension Pack for Apache Camel provides a collection of useful tools for Apache Camel K developers,
+such as code completion and integrated lifecycle management. They are **recommended** for the tutorial, but they are **not**
+required.
+
+You can install it from the VS Code Extensions marketplace.
+
+[Check if the VS Code Extension Pack for Apache Camel by Red Hat is installed](didact://?commandId=vscode.didact.extensionRequirementCheck&text=extension-requirement-status$$redhat.apache-camel-extension-pack&completion=Camel%20extension%20pack%20is%20available%20on%20this%20system. "Checks the VS Code workspace to make sure the extension pack is installed"){.didact}
+
+*Status: unknown*{#extension-requirement-status}
+
+## 1. Preparing the namespace
+
+Let's open a terminal and go to the example directory:
+
+```
+cd 05-aws-kinesis-firehose-source-basic
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$cd%2005-aws-kinesis-firehose-source-basic&completion=Executed%20command. "Opens a new terminal and sends the command above"){.didact})
+
+
+We're going to create a namespace named `aws-kinesis-firehose-event-source` for running the example. To create it, execute the following command:
+
+```
+kubectl create namespace aws-kinesis-firehose-event-source
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20create%20namespace%20aws-kinesis-firehose-event-source&completion=New%20project%20creation. "Opens a new terminal and sends the command above"){.didact})
+
+Now we can set the `aws-kinesis-firehose-event-source` namespace as default namespace for the following commands:
+
+```
+kubectl config set-context --current --namespace=aws-kinesis-firehose-event-source
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20config%20set-context%20--current%20--namespace%3Daws-kinesis-firehose-event-source&completion=New%20project%20creation. "Opens a new terminal and sends the command above"){.didact})
+
+You need to install Camel K in the `aws-kinesis-firehose-event-source` namespace (or globally in the whole cluster).
+In many settings (e.g. OpenShift, CRC), it's sufficient to execute the following command to install Camel K:
+
+```
+kamel install
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20install&completion=Installing%20Camel%20K. "Opens a new terminal and sends the command above"){.didact})
+
+NOTE: The `kamel install` command requires some prerequisites to be successful in some situations, e.g. you need to enable the registry addon on Minikube. Refer to the [Camel K install guide](https://camel.apache.org/camel-k/latest/installation/installation.html) for cluster-specific instructions.
+
+To check that Camel K is installed we'll retrieve the IntegrationPlatform object from the namespace:
+
+```
+kubectl get integrationplatform
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20get%20integrationplatform&completion=Executed%20Command. "Opens a new terminal and sends the command above"){.didact})
+
+You should find an IntegrationPlatform in status `Ready`.
+
+You can now proceed to the next section.
+
+## 2. Preparing the environment
+
+This repository contains a simple [aws-services.properties](didact://?commandId=vscode.openFolder&projectFilePath=04-aws-kinesis-source-basic/aws-services.properties&completion=Opened%20the%aws-services.properties%20file "Opens the aws-kinesis.properties file"){.didact} that contains the access key and secret key for accessing the AWS Kinesis stream.
+
+```
+kubectl create secret generic aws-services --from-file=aws-services.properties
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20create%20secret%20generic%20aws-services%20--from-file%3Daws-services.properties&completion=secret%20%22aws-services%22%20created. "Create a secret with AWS Kinesis credentials"){.didact})
+
+As the example levareges [Knative Eventing channels](https://knative.dev/v0.15-docs/eventing/channels/), we need to create the one that the example will use:
+
+```
+kubectl apply -f aws-s3-kinesis-firehose-channel.yaml
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20apply%20-f%20aws-s3-kinesis-firehose-channel.yaml&completion=inmemorychannel.messaging.knative.dev/aws-s3-kinesis-firehose$20created. "Create a Knative InMemoryChannel named aws-kinesis"){.didact})
+
+
+## 2. Running a Camel Source
+
+This repository contains a simple Camel Source based on the [AWS Kinesis component](https://camel.apache.org/components/latest/aws-kinesis-component.html) that forward streaming events received on the AWS Kinesis delivery stream to a Knative channel named `aws-s3-kinesis-firehose`.
+
+Use the following command to deploy the Camel Source:
+
+```
+kubectl apply -f aws-s3-kinesis-firehose-source.yaml
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20apply%20-f%20aws-s3-kinesis-firehose-source.yaml&completion=camelsource.sources.knative.dev/camel-aws-s3-kinesis-firehose-source%20created. "Opens a new terminal and sends the command above"){.didact})
+
+## 2. Running a basic integration to create Kinesis events for consumption by the Camel Source
+
+You need a producer adding data to a Kinesis stream to try this example. This integration
+comes with a sample producer that will send 100 messages with the text `Hello Camel K`
+every 3 seconds.
+
+```
+kamel run --secret aws-services aws-kinesis-firehose-producer.groovy
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20run%20--secret%20aws-kinesis%20aws-kinesis-firehose-producer.groovy&completion=Camel%20K%20aws-kinesis-firehose-producer%20integration%20run%20in%20dev%20mode. "Opens a new terminal and sends the command above"){.didact})
+
+If everything is ok, after the build phase finishes, you should see the Camel integration running.
+
+## 3. Running a basic integration to forward Kinesis events to the console
+
+```
+kamel run aws-s3-consumer.groovy --dev
+```
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kamel%20run%20aws-s3-consumer.groovy%20--dev&completion=Camel%20K%20aws-s3-consumer%20integration%20run%20in%20dev%20mode. "Opens a new terminal and sends the command above"){.didact})
+
+If everything is ok, after the build phase finishes, you should see the Camel integration running.
+
+## 4. Uninstall
+
+To cleanup everything, execute the following command:
+
+```kubectl delete namespace aws-kinesis-firehose-event-source```
+
+([^ execute](didact://?commandId=vscode.didact.sendNamedTerminalAString&text=camelTerm$$kubectl%20delete%20namespace%20aws-kinesis-source-basic&completion=Removed%20the%20namespace%20from%20the%20cluster. "Cleans up the cluster after running the example"){.didact})
diff --git a/05-aws-kinesis-firehose-source-basic/readme.md b/05-aws-kinesis-firehose-source-basic/readme.md
new file mode 120000
index 0000000..c0bf82f
--- /dev/null
+++ b/05-aws-kinesis-firehose-source-basic/readme.md
@@ -0,0 +1 @@
+readme.didact.md
\ No newline at end of file