You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2022/11/02 14:20:43 UTC
[camel-kamelets-examples] 01/01: Added an example of AWS S3 CDC Source Kamelet usage
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch aws-s3-cdc-example
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
commit 70ae9bcaed880fa93575f575070593b14efd2fb1
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Wed Nov 2 15:20:14 2022 +0100
Added an example of AWS S3 CDC Source Kamelet usage
Signed-off-by: Andrea Cosentino <an...@gmail.com>
---
jbang/aws-s3-cdc/README.adoc | 153 +++++++++++++++++++++++++++++++++++
jbang/aws-s3-cdc/aws-s3-cdc-log.yaml | 31 +++++++
jbang/aws-s3-cdc/example-file.txt | 1 +
jbang/aws-s3-cdc/policy-queue.json | 3 +
4 files changed, 188 insertions(+)
diff --git a/jbang/aws-s3-cdc/README.adoc b/jbang/aws-s3-cdc/README.adoc
new file mode 100644
index 0000000..301e3e8
--- /dev/null
+++ b/jbang/aws-s3-cdc/README.adoc
@@ -0,0 +1,153 @@
+== AWS S3 CDC Example
+
+In this sample you'll use the AWS S3 CDC Source Kamelet.
+
+Through the usage of Eventbridge and SQS Services you'll be able to consume events from specific bucket.
+
+=== Install JBang
+
+First install JBang according to https://www.jbang.dev
+
+When JBang is installed then you should be able to run from a shell:
+
+[source,sh]
+----
+$ jbang --version
+----
+
+This will output the version of JBang.
+
+To run this example you can either install Camel on JBang via:
+
+[source,sh]
+----
+$ jbang app install camel@apache/camel
+----
+
+Which allows to run CamelJBang with `camel` as shown below.
+
+=== Setup the AWS S3 bucket, SQS Queue and Eventbrige Rule
+
+You'll need a fully working AWS CLI locally.
+
+Create a bucket on AWS on a particular region
+
+[source,sh]
+----
+aws s3api create-bucket --bucket cdc-s3-bucket --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1
+----
+
+Enable Eventbridge notification on the bucket
+
+[source,sh]
+----
+aws s3api put-bucket-notification-configuration --bucket cdc-s3-bucket --region eu-west-1 --notification-configuration '{ "EventBridgeConfiguration": {} }'
+----
+
+Add an Eventbridge rule on the bucket
+
+[source,sh]
+----
+aws events put-rule --name "s3-events-cdc" --event-pattern "{\"source\":[\"aws.s3\"], \"detail\": {\"bucket\": { \"name\": [\"cdc-s3-bucket\"]}}}" --region=eu-west-1
+----
+
+Create an SQS Queue where we're going to receive the notification
+
+[source,sh]
+----
+aws sqs create-queue --queue-name test-queue-3 --region eu-west-1
+----
+
+Modify Access Policy for the queue just created. Don't forget to edit the policy-queue.json file (by adding the correct rule_arn and queue_arn), before submitting the command.
+
+[source,sh]
+----
+aws sqs set-queue-attributes --queue-url <just_created_queue_arn> --attributes file://policy-queue.json
+----
+
+Add a target for Eventbridge Rule which will be the SQS Queue just created
+
+[source,sh]
+----
+aws events put-targets --rule s3-events-cdc --targets "Id"="sqs-sub","Arn"="<just_created_queue_arn>" --region eu-west-1
+----
+
+=== How to run
+
+Then you can run this example using:
+
+[source,sh]
+----
+$ camel run aws-s3-cdc-log.yaml
+----
+
+Or run it even shorter:
+
+[source,sh]
+----
+$ camel run *
+----
+
+Or run with JBang using the longer command line (without installing camel as app in JBang):
+
+[source,sh]
+----
+$ jbang camel@apache/camel run aws-s3-cdc-log.yaml
+----
+
+=== Developer Web Console
+
+You can enable the developer console via `--console` flag as show:
+
+[source,sh]
+----
+$ camel run aws-s3-cdc-log.yaml --console
+----
+
+Then you can browse: http://localhost:8080/q/dev to introspect the running Camel applicaton.
+
+=== Create and delete an object
+
+While the integration is running you can run the following commands:
+
+[source,sh]
+----
+aws s3api put-object --bucket cdc-s3-bucket --key example-file-uploaded.txt --body example-file.txt --region eu-west-1
+{
+ "ETag": "\"2b6153cbc11ea06a6d0b502a64b77e3c\""
+}
+aws s3api delete-object --bucket cdc-s3-bucket --key example-file.txt
+----
+
+[source,sh]
+----
+022-11-02 15:13:03.747 INFO 120300 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local)
+2022-11-02 15:13:08.570 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (aws-s3-cdc-log) is starting
+2022-11-02 15:13:08.726 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] queueNameOrArn=test-queue-3
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] accessKey=xxxxxx
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] region=eu-west-1
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] secretKey=xxxxxx
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] getObject=true
+2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true
+2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://aws-s3-cdc-source)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started aws-s3-cdc-source-1 (aws2-sqs://test-queue-3)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source)
+2022-11-02 15:13:08.742 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (aws-s3-cdc-log) started in 4s272ms (build:106ms init:3s995ms start:171ms JVM-uptime:6s)
+
+
+2022-11-02 15:13:30.977 INFO 120300 --- [://test-queue-3] info : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: Text example]
+2022-11-02 15:13:41.250 INFO 120300 --- [://test-queue-3] info : Exchange[ExchangePattern: InOnly, BodyType: com.fasterxml.jackson.databind.node.ObjectNode, Body: { "version" : "0", "id" : "d54290df-2ad9-31ff-308b-8331fee7344a", "detail-type" : "Object Deleted", "source" : "aws.s3", "account" : "xxxx", "time" : "2022-11-02T14:13:37Z", "region" : "eu-west-1", "resources" : [ "arn:aws:s3:::cdc-s3-bucket" ], "detail" : { "version" : "0", " [...]
+----
+
+
+=== Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+https://camel.apache.org/community/support/[let us know].
+
+We also love contributors, so
+https://camel.apache.org/community/contributing/[get involved] :-)
+
+The Camel riders!
diff --git a/jbang/aws-s3-cdc/aws-s3-cdc-log.yaml b/jbang/aws-s3-cdc/aws-s3-cdc-log.yaml
new file mode 100644
index 0000000..103e3df
--- /dev/null
+++ b/jbang/aws-s3-cdc/aws-s3-cdc-log.yaml
@@ -0,0 +1,31 @@
+## ---------------------------------------------------------------------------
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements. See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License. You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+## ---------------------------------------------------------------------------
+
+- route:
+ from:
+ uri: "kamelet:aws-s3-cdc-source"
+ parameters:
+ queueNameOrArn: "test-queue-3"
+ accessKey: <access_key>
+ secretKey: <secret_key>
+ region: "eu-west-1"
+ getObject: "true"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showStreams: true
diff --git a/jbang/aws-s3-cdc/example-file.txt b/jbang/aws-s3-cdc/example-file.txt
new file mode 100644
index 0000000..105c45e
--- /dev/null
+++ b/jbang/aws-s3-cdc/example-file.txt
@@ -0,0 +1 @@
+Text example
diff --git a/jbang/aws-s3-cdc/policy-queue.json b/jbang/aws-s3-cdc/policy-queue.json
new file mode 100644
index 0000000..b6d0964
--- /dev/null
+++ b/jbang/aws-s3-cdc/policy-queue.json
@@ -0,0 +1,3 @@
+{
+ "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"<queue_arn>/SQSDefaultPolicy\",\"Statement\":[{\"Sid\": \"EventsToMyQueue\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"events.amazonaws.com\"}, \"Action\": \"sqs:SendMessage\", \"Resource\": \"<queue_arn>\", \"Condition\": {\"ArnEquals\": {\"aws:SourceArn\": \"<rule_arn>\"}}}]}"
+}