You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2022/12/05 10:32:37 UTC
[camel-kamelets-examples] branch main updated: Added an Azure Storage Blob CDC example
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
The following commit(s) were added to refs/heads/main by this push:
new 7d5486b Added an Azure Storage Blob CDC example
7d5486b is described below
commit 7d5486bb3e34c1c362ea82e00b8f65e5e0cb73b8
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Mon Dec 5 11:30:14 2022 +0100
Added an Azure Storage Blob CDC example
Signed-off-by: Andrea Cosentino <an...@gmail.com>
---
jbang/azure-storage-blob-cdc/README.adoc | 186 +++++++++++++++++++++
.../azure-storage-blob-cdc.yaml | 33 ++++
jbang/azure-storage-blob-cdc/azure.prop | 1 +
jbang/azure-storage-blob-cdc/terraform/main.tf | 76 +++++++++
4 files changed, 296 insertions(+)
diff --git a/jbang/azure-storage-blob-cdc/README.adoc b/jbang/azure-storage-blob-cdc/README.adoc
new file mode 100644
index 0000000..537fc7c
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/README.adoc
@@ -0,0 +1,186 @@
+== Azure Storage Blob CDC Example
+
+In this sample you'll use the Azure Storage Blob CDC Source Kamelet.
+
+Through the usage of Event Grid and Servicebus Services you'll be able to consume events from specific containers.
+
+=== Install JBang
+
+First install JBang according to https://www.jbang.dev
+
+When JBang is installed then you should be able to run from a shell:
+
+[source,sh]
+----
+$ jbang --version
+----
+
+This will output the version of JBang.
+
+To run this example you can either install Camel on JBang via:
+
+[source,sh]
+----
+$ jbang app install camel@apache/camel
+----
+
+Which allows to run CamelJBang with `camel` as shown below.
+
+=== Setup the Servicebus, Azure Storage Blob and Eventgrid
+
+Running the terraform configuration provided in terraform folder.
+
+[source,sh]
+----
+cd terraform/
+----
+
+and then run
+
+[source,sh]
+----
+terraform init
+----
+
+At this point you should be to run the configuration
+
+[source,sh]
+----
+terraform apply
+----
+
+Enter yes and wait for the terraform configuration to end.
+
+=== Setup credentials
+
+In the `azure.prop` add the correct accessKey for the Azure Storage Blob account.
+
+In the `azure-storage-blob-cdc.yaml` add the correct connection String for the Servicebus Queue.
+
+=== How to run
+
+Then you can run this example using:
+
+[source,sh]
+----
+$ camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml
+----
+
+Replace the local kamelet dir with your local directory.
+
+Or run with JBang using the longer command line (without installing camel as app in JBang):
+
+[source,sh]
+----
+$ jbang run camel@apache/camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml
+----
+
+Replace the local kamelet dir with your local directory.
+
+=== Developer Web Console
+
+You can enable the developer console via `--console` flag as show:
+
+[source,sh]
+----
+$ camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml --console
+----
+
+Then you can browse: http://localhost:8080/q/dev to introspect the running Camel applicaton.
+
+=== Create and delete an object
+
+While the integration is running you can run the following commands:
+
+[source,sh]
+----
+az storage blob upload --account-name kameletsaccount --container-name kamelets --file nominatim-geocode-action.kamelet.yaml --account-key <accountKey>
+az storage blob delete --account-name kameletsaccount --container-name kamelets --name nominatim-geocode-action.kamelet.yaml --account-key <accountKey>
+----
+
+You should see at first the content of the uploaded file and in the second message the deletion of the blob from the container.
+
+[source,sh]
+----
+2022-12-05 10:08:37.134 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.20.0-SNAPSHOT is starting
+2022-12-05 10:08:37.340 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Using Java 11.0.16.1 with PID 54702. Started by oscerd in /home/oscerd/workspace/apache-camel/camel-kamelets-examples/jbang/azure-storage-blob-cdc
+2022-12-05 10:08:37.350 INFO 54702 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local)
+2022-12-05 10:08:39.949 INFO 54702 --- [ main] .main.download.MavenDependencyDownloader : Downloaded: org.apache.camel:camel-azure-servicebus:3.20.0-SNAPSHOT (took: 1s87ms)
+2022-12-05 10:08:41.307 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) is starting
+2022-12-05 10:08:41.512 INFO 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : {"az.sdk.message":"Setting entity from connection string.","entityPath":"kamelets_servicebus_queue"}
+2022-12-05 10:08:41.514 WARN 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : 'enableAutoComplete' is not needed in for RECEIVE_AND_DELETE mode.
+2022-12-05 10:08:41.709 INFO 54702 --- [ main] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Setting next AMQP channel.","entityPath":"N/A"}
+2022-12-05 10:08:41.711 INFO 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : # of open clients with shared connection: 1
+2022-12-05 10:08:41.727 INFO 54702 --- [ main] servicebus.ServiceBusReceiverAsyncClient : {"az.sdk.message":"Creating consumer.","linkName":"kamelets_servicebus_queue_18caee_1670231321727","entityPath":"kamelets_servicebus_queue"}
+2022-12-05 10:08:41.735 INFO 54702 --- [ main] mentation.ServiceBusReceiveLinkProcessor : Requesting a new AmqpReceiveLink from upstream.
+2022-12-05 10:08:41.759 INFO 54702 --- [ main] re.amqp.implementation.ReactorConnection : {"az.sdk.message":"Creating and starting connection.","connectionId":"MF_a947d3_1670231321658","hostName":"kamelets-servicebus-namespace.servicebus.windows.net","port":5671}
+2022-12-05 10:08:41.782 INFO 54702 --- [ main] core.amqp.implementation.ReactorExecutor : {"az.sdk.message":"Starting reactor.","connectionId":"MF_a947d3_1670231321658"}
+2022-12-05 10:08:41.792 INFO 54702 --- [ctor-executor-1] implementation.handler.ConnectionHandler : {"az.sdk.message":"onConnectionInit","connectionId":"MF_a947d3_1670231321658","hostName":"kamelets-servicebus-namespace.servicebus.windows.net","namespace":"kamelets-servicebus-namespace.servicebus.windows.net"}
+2022-12-05 10:08:41.793 INFO 54702 --- [ctor-executor-1] qp.implementation.handler.ReactorHandler : {"az.sdk.message":"reactor.onReactorInit","connectionId":"MF_a947d3_1670231321658"}
+2022-12-05 10:08:41.794 INFO 54702 --- [ctor-executor-1] implementation.handler.ConnectionHandler : {"az.sdk.message":"onConnectionLocalOpen","connectionId":"MF_a947d3_1670231321658","errorCondition":null,"errorDescription":null,"hostName":"kamelets-servicebus-namespace.servicebus.windows.net"}
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] topicOrQueueName=kamelets_servicebus_queue
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] connectionString=xxxxxx
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] getBlob=true
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] accountName=kameletsaccount
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] containerName=kamelets
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] accessKey=xxxxxx
+2022-12-05 10:08:41.814 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://azure-storage-blob-cdc-source)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started azure-storage-blob-cdc-source-1 (azure-servicebus://kamelets_servicebus_queue)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) started in 3s404ms (build:100ms init:2s780ms start:524ms JVM-uptime:5s)
+2022-12-05 10:08:54.590 INFO 54702 --- [oundedElastic-1] storage-blob-cdc-source.kamelet.yaml:138 : Microsoft.Storage.BlobCreated - /blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml - nominatim-geocode-action.kamelet.yaml
+2022-12-05 10:08:55.370 INFO 54702 --- [oundedElastic-1] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: # ---------------------------------------------------------------------------# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# Th [...]
+2022-12-05 10:08:55.372 INFO 54702 --- [oundedElastic-1] mentation.ServiceBusReceiveLinkProcessor : {"az.sdk.message":"Adding credits.","prefetch":0,"requested":2,"linkCredits":0,"expectedTotalCredit":2,"queuedMessages":1,"creditsToAdd":1,"messageQueueSize":0}
+2022-12-05 10:09:05.021 INFO 54702 --- [oundedElastic-1] storage-blob-cdc-source.kamelet.yaml:138 : Microsoft.Storage.BlobDeleted - /blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml - nominatim-geocode-action.kamelet.yaml
+2022-12-05 10:09:05.026 INFO 54702 --- [oundedElastic-1] log-sink : Exchange[ExchangePattern: InOnly, BodyType: com.fasterxml.jackson.databind.node.ObjectNode, Body: { "topic" : "/subscriptions/13d32827-569a-4093-bb0a-6d4bcebc757b/resourceGroups/kamelets-resources/providers/Microsoft.Storage/storageAccounts/kameletsaccount", "subject" : "/blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml", "eventType" : "Microsoft.Sto [...]
+2022-12-05 10:09:05.027 INFO 54702 --- [oundedElastic-1] mentation.ServiceBusReceiveLinkProcessor : {"az.sdk.message":"Adding credits.","prefetch":0,"requested":2,"linkCredits":0,"expectedTotalCredit":2,"queuedMessages":1,"creditsToAdd":1,"messageQueueSize":0}
+^C2022-12-05 10:09:08.094 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) is shutting down (timeout:10s)
+2022-12-05 10:09:08.099 INFO 54702 --- [ - ShutdownTask] servicebus.ServiceBusReceiverAsyncClient : Removing receiver links.
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] aging.servicebus.ServiceBusClientBuilder : {"az.sdk.message":"Closing a dependent client.","numberOfOpenClients":0}
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] aging.servicebus.ServiceBusClientBuilder : No more open clients, closing shared connection.
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Upstream connection publisher was completed. Terminating processor.","entityPath":"N/A"}
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] re.amqp.implementation.ReactorConnection : {"az.sdk.message":"Disposing of ReactorConnection.","connectionId":"MF_a947d3_1670231321658","isTransient":false,"isInitiatedByClient":true,"shutdownMessage":"Disposed by client."}
+2022-12-05 10:09:08.104 INFO 54702 --- [ - ShutdownTask] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Channel is disposed.","entityPath":"N/A"}
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes stopped (stopped:3)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped log-sink-2 (kamelet://source)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped azure-storage-blob-cdc-source-1 (azure-servicebus://kamelets_servicebus_queue)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped route1 (kamelet://azure-storage-blob-cdc-source)
+2022-12-05 10:09:08.126 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) shutdown in 32ms (uptime:26s JVM-uptime:31s)
+2022-12-05 10:09:08.126 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.20.0-SNAPSHOT shutdown
+
+----
+
+=== Cleanup Servicebus, Azure Storage Blob and Eventgrid through Terraform
+
+You'll need to cleanup everything from AWS console or CLI.
+
+If you used terraform it will be enough to run terraform destroy
+
+[source,sh]
+----
+cd terraform/
+----
+
+At this point you should be to run the destroy
+
+[source,sh]
+----
+terraform destroy
+----
+
+You'll need to specify the same var used for terraform apply.
+
+At the end the Azure enviroment on your account will be clean.
+
+=== Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+https://camel.apache.org/community/support/[let us know].
+
+We also love contributors, so
+https://camel.apache.org/community/contributing/[get involved] :-)
+
+The Camel riders!
diff --git a/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml b/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml
new file mode 100644
index 0000000..a5dbef0
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml
@@ -0,0 +1,33 @@
+## ---------------------------------------------------------------------------
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements. See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License. You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+## ---------------------------------------------------------------------------
+
+# camel-k: dependency=camel:aws-secrets-manager
+
+- route:
+ from:
+ uri: "kamelet:azure-storage-blob-cdc-source"
+ parameters:
+ accountName: "kameletsaccount"
+ containerName: "kamelets"
+ getBlob: true
+ topicOrQueueName: "kamelets_servicebus_queue"
+ connectionString: "RAW(<connectionString>)"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showStreams: "true"
diff --git a/jbang/azure-storage-blob-cdc/azure.prop b/jbang/azure-storage-blob-cdc/azure.prop
new file mode 100644
index 0000000..038c729
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/azure.prop
@@ -0,0 +1 @@
+camel.kamelet.azure-storage-blob-cdc-source.accessKey=
diff --git a/jbang/azure-storage-blob-cdc/terraform/main.tf b/jbang/azure-storage-blob-cdc/terraform/main.tf
new file mode 100644
index 0000000..cacd853
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/terraform/main.tf
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# We strongly recommend using the required_providers block to set the
+# Azure Provider source and version being used
+terraform {
+ required_providers {
+ azurerm = {
+ source = "hashicorp/azurerm"
+ version = "=3.0.0"
+ }
+ }
+}
+
+# Configure the Microsoft Azure Provider
+provider "azurerm" {
+ features {}
+}
+
+resource "azurerm_resource_group" "kamelets" {
+ name = "kamelets-resources"
+ location = "West Europe"
+}
+
+resource "azurerm_storage_account" "kamelets" {
+ name = "kameletsaccount"
+ resource_group_name = azurerm_resource_group.kamelets.name
+ location = azurerm_resource_group.kamelets.location
+ account_tier = "Standard"
+ account_replication_type = "LRS"
+}
+
+resource "azurerm_storage_container" "camelkamelets" {
+ name = "kamelets"
+ storage_account_name = azurerm_storage_account.kamelets.name
+ container_access_type = "private"
+}
+
+resource "azurerm_servicebus_namespace" "example" {
+ name = "kamelets-servicebus-namespace"
+ location = azurerm_resource_group.kamelets.location
+ resource_group_name = azurerm_resource_group.kamelets.name
+ sku = "Standard"
+}
+
+resource "azurerm_servicebus_queue" "example" {
+ name = "kamelets_servicebus_queue"
+ namespace_id = azurerm_servicebus_namespace.example.id
+
+ enable_partitioning = true
+}
+
+resource "azurerm_eventgrid_event_subscription" "example" {
+ name = "example-aees"
+ scope = azurerm_storage_account.kamelets.id
+
+ service_bus_queue_endpoint_id = azurerm_servicebus_queue.example.id
+
+ included_event_types = [
+ "Microsoft.Storage.BlobCreated", "Microsoft.Storage.BlobDeleted"
+ ]
+}