You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2022/12/05 10:31:03 UTC
[camel-kamelets-examples] branch azure-storage-blob-cdc-ex created (now b9b61cb)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch azure-storage-blob-cdc-ex
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
at b9b61cb Added an Azure Storage Blob CDC example
This branch includes the following new commits:
new b9b61cb Added an Azure Storage Blob CDC example
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-kamelets-examples] 01/01: Added an Azure Storage Blob CDC example
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch azure-storage-blob-cdc-ex
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
commit b9b61cbd30dbb17f6cdb23619f766608f2e48bd4
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Mon Dec 5 11:30:14 2022 +0100
Added an Azure Storage Blob CDC example
Signed-off-by: Andrea Cosentino <an...@gmail.com>
---
jbang/azure-storage-blob-cdc/README.adoc | 186 +++++++++++++++++++++
.../azure-storage-blob-cdc.yaml | 33 ++++
jbang/azure-storage-blob-cdc/azure.prop | 1 +
jbang/azure-storage-blob-cdc/terraform/main.tf | 76 +++++++++
4 files changed, 296 insertions(+)
diff --git a/jbang/azure-storage-blob-cdc/README.adoc b/jbang/azure-storage-blob-cdc/README.adoc
new file mode 100644
index 0000000..537fc7c
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/README.adoc
@@ -0,0 +1,186 @@
+== Azure Storage Blob CDC Example
+
+In this sample you'll use the Azure Storage Blob CDC Source Kamelet.
+
+Through the usage of Event Grid and Servicebus Services you'll be able to consume events from specific containers.
+
+=== Install JBang
+
+First install JBang according to https://www.jbang.dev
+
+When JBang is installed then you should be able to run from a shell:
+
+[source,sh]
+----
+$ jbang --version
+----
+
+This will output the version of JBang.
+
+To run this example you can either install Camel on JBang via:
+
+[source,sh]
+----
+$ jbang app install camel@apache/camel
+----
+
+Which allows to run CamelJBang with `camel` as shown below.
+
+=== Setup the Servicebus, Azure Storage Blob and Eventgrid
+
+Running the terraform configuration provided in terraform folder.
+
+[source,sh]
+----
+cd terraform/
+----
+
+and then run
+
+[source,sh]
+----
+terraform init
+----
+
+At this point you should be to run the configuration
+
+[source,sh]
+----
+terraform apply
+----
+
+Enter yes and wait for the terraform configuration to end.
+
+=== Setup credentials
+
+In the `azure.prop` add the correct accessKey for the Azure Storage Blob account.
+
+In the `azure-storage-blob-cdc.yaml` add the correct connection String for the Servicebus Queue.
+
+=== How to run
+
+Then you can run this example using:
+
+[source,sh]
+----
+$ camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml
+----
+
+Replace the local kamelet dir with your local directory.
+
+Or run with JBang using the longer command line (without installing camel as app in JBang):
+
+[source,sh]
+----
+$ jbang run camel@apache/camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml
+----
+
+Replace the local kamelet dir with your local directory.
+
+=== Developer Web Console
+
+You can enable the developer console via `--console` flag as show:
+
+[source,sh]
+----
+$ camel run --local-kamelet-dir=<local-kamelets-dir> --properties=azure.prop azure-storage-blob-cdc.yaml --console
+----
+
+Then you can browse: http://localhost:8080/q/dev to introspect the running Camel applicaton.
+
+=== Create and delete an object
+
+While the integration is running you can run the following commands:
+
+[source,sh]
+----
+az storage blob upload --account-name kameletsaccount --container-name kamelets --file nominatim-geocode-action.kamelet.yaml --account-key <accountKey>
+az storage blob delete --account-name kameletsaccount --container-name kamelets --name nominatim-geocode-action.kamelet.yaml --account-key <accountKey>
+----
+
+You should see at first the content of the uploaded file and in the second message the deletion of the blob from the container.
+
+[source,sh]
+----
+2022-12-05 10:08:37.134 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.20.0-SNAPSHOT is starting
+2022-12-05 10:08:37.340 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Using Java 11.0.16.1 with PID 54702. Started by oscerd in /home/oscerd/workspace/apache-camel/camel-kamelets-examples/jbang/azure-storage-blob-cdc
+2022-12-05 10:08:37.350 INFO 54702 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local)
+2022-12-05 10:08:39.949 INFO 54702 --- [ main] .main.download.MavenDependencyDownloader : Downloaded: org.apache.camel:camel-azure-servicebus:3.20.0-SNAPSHOT (took: 1s87ms)
+2022-12-05 10:08:41.307 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) is starting
+2022-12-05 10:08:41.512 INFO 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : {"az.sdk.message":"Setting entity from connection string.","entityPath":"kamelets_servicebus_queue"}
+2022-12-05 10:08:41.514 WARN 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : 'enableAutoComplete' is not needed in for RECEIVE_AND_DELETE mode.
+2022-12-05 10:08:41.709 INFO 54702 --- [ main] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Setting next AMQP channel.","entityPath":"N/A"}
+2022-12-05 10:08:41.711 INFO 54702 --- [ main] aging.servicebus.ServiceBusClientBuilder : # of open clients with shared connection: 1
+2022-12-05 10:08:41.727 INFO 54702 --- [ main] servicebus.ServiceBusReceiverAsyncClient : {"az.sdk.message":"Creating consumer.","linkName":"kamelets_servicebus_queue_18caee_1670231321727","entityPath":"kamelets_servicebus_queue"}
+2022-12-05 10:08:41.735 INFO 54702 --- [ main] mentation.ServiceBusReceiveLinkProcessor : Requesting a new AmqpReceiveLink from upstream.
+2022-12-05 10:08:41.759 INFO 54702 --- [ main] re.amqp.implementation.ReactorConnection : {"az.sdk.message":"Creating and starting connection.","connectionId":"MF_a947d3_1670231321658","hostName":"kamelets-servicebus-namespace.servicebus.windows.net","port":5671}
+2022-12-05 10:08:41.782 INFO 54702 --- [ main] core.amqp.implementation.ReactorExecutor : {"az.sdk.message":"Starting reactor.","connectionId":"MF_a947d3_1670231321658"}
+2022-12-05 10:08:41.792 INFO 54702 --- [ctor-executor-1] implementation.handler.ConnectionHandler : {"az.sdk.message":"onConnectionInit","connectionId":"MF_a947d3_1670231321658","hostName":"kamelets-servicebus-namespace.servicebus.windows.net","namespace":"kamelets-servicebus-namespace.servicebus.windows.net"}
+2022-12-05 10:08:41.793 INFO 54702 --- [ctor-executor-1] qp.implementation.handler.ReactorHandler : {"az.sdk.message":"reactor.onReactorInit","connectionId":"MF_a947d3_1670231321658"}
+2022-12-05 10:08:41.794 INFO 54702 --- [ctor-executor-1] implementation.handler.ConnectionHandler : {"az.sdk.message":"onConnectionLocalOpen","connectionId":"MF_a947d3_1670231321658","errorCondition":null,"errorDescription":null,"hostName":"kamelets-servicebus-namespace.servicebus.windows.net"}
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] topicOrQueueName=kamelets_servicebus_queue
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] connectionString=xxxxxx
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] getBlob=true
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] accountName=kameletsaccount
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] containerName=kamelets
+2022-12-05 10:08:41.813 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [blob-cdc-source.kamelet.yaml] accessKey=xxxxxx
+2022-12-05 10:08:41.814 INFO 54702 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://azure-storage-blob-cdc-source)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started azure-storage-blob-cdc-source-1 (azure-servicebus://kamelets_servicebus_queue)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source)
+2022-12-05 10:08:41.832 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) started in 3s404ms (build:100ms init:2s780ms start:524ms JVM-uptime:5s)
+2022-12-05 10:08:54.590 INFO 54702 --- [oundedElastic-1] storage-blob-cdc-source.kamelet.yaml:138 : Microsoft.Storage.BlobCreated - /blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml - nominatim-geocode-action.kamelet.yaml
+2022-12-05 10:08:55.370 INFO 54702 --- [oundedElastic-1] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: # ---------------------------------------------------------------------------# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# Th [...]
+2022-12-05 10:08:55.372 INFO 54702 --- [oundedElastic-1] mentation.ServiceBusReceiveLinkProcessor : {"az.sdk.message":"Adding credits.","prefetch":0,"requested":2,"linkCredits":0,"expectedTotalCredit":2,"queuedMessages":1,"creditsToAdd":1,"messageQueueSize":0}
+2022-12-05 10:09:05.021 INFO 54702 --- [oundedElastic-1] storage-blob-cdc-source.kamelet.yaml:138 : Microsoft.Storage.BlobDeleted - /blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml - nominatim-geocode-action.kamelet.yaml
+2022-12-05 10:09:05.026 INFO 54702 --- [oundedElastic-1] log-sink : Exchange[ExchangePattern: InOnly, BodyType: com.fasterxml.jackson.databind.node.ObjectNode, Body: { "topic" : "/subscriptions/13d32827-569a-4093-bb0a-6d4bcebc757b/resourceGroups/kamelets-resources/providers/Microsoft.Storage/storageAccounts/kameletsaccount", "subject" : "/blobServices/default/containers/kamelets/blobs/nominatim-geocode-action.kamelet.yaml", "eventType" : "Microsoft.Sto [...]
+2022-12-05 10:09:05.027 INFO 54702 --- [oundedElastic-1] mentation.ServiceBusReceiveLinkProcessor : {"az.sdk.message":"Adding credits.","prefetch":0,"requested":2,"linkCredits":0,"expectedTotalCredit":2,"queuedMessages":1,"creditsToAdd":1,"messageQueueSize":0}
+^C2022-12-05 10:09:08.094 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) is shutting down (timeout:10s)
+2022-12-05 10:09:08.099 INFO 54702 --- [ - ShutdownTask] servicebus.ServiceBusReceiverAsyncClient : Removing receiver links.
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] aging.servicebus.ServiceBusClientBuilder : {"az.sdk.message":"Closing a dependent client.","numberOfOpenClients":0}
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] aging.servicebus.ServiceBusClientBuilder : No more open clients, closing shared connection.
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Upstream connection publisher was completed. Terminating processor.","entityPath":"N/A"}
+2022-12-05 10:09:08.103 INFO 54702 --- [ - ShutdownTask] re.amqp.implementation.ReactorConnection : {"az.sdk.message":"Disposing of ReactorConnection.","connectionId":"MF_a947d3_1670231321658","isTransient":false,"isInitiatedByClient":true,"shutdownMessage":"Disposed by client."}
+2022-12-05 10:09:08.104 INFO 54702 --- [ - ShutdownTask] ementation.ServiceBusConnectionProcessor : {"az.sdk.message":"Channel is disposed.","entityPath":"N/A"}
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes stopped (stopped:3)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped log-sink-2 (kamelet://source)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped azure-storage-blob-cdc-source-1 (azure-servicebus://kamelets_servicebus_queue)
+2022-12-05 10:09:08.120 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Stopped route1 (kamelet://azure-storage-blob-cdc-source)
+2022-12-05 10:09:08.126 INFO 54702 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (azure-storage-blob-cdc) shutdown in 32ms (uptime:26s JVM-uptime:31s)
+2022-12-05 10:09:08.126 INFO 54702 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.20.0-SNAPSHOT shutdown
+
+----
+
+=== Cleanup Servicebus, Azure Storage Blob and Eventgrid through Terraform
+
+You'll need to cleanup everything from AWS console or CLI.
+
+If you used terraform it will be enough to run terraform destroy
+
+[source,sh]
+----
+cd terraform/
+----
+
+At this point you should be to run the destroy
+
+[source,sh]
+----
+terraform destroy
+----
+
+You'll need to specify the same var used for terraform apply.
+
+At the end the Azure enviroment on your account will be clean.
+
+=== Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+https://camel.apache.org/community/support/[let us know].
+
+We also love contributors, so
+https://camel.apache.org/community/contributing/[get involved] :-)
+
+The Camel riders!
diff --git a/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml b/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml
new file mode 100644
index 0000000..a5dbef0
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/azure-storage-blob-cdc.yaml
@@ -0,0 +1,33 @@
+## ---------------------------------------------------------------------------
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements. See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License. You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+## ---------------------------------------------------------------------------
+
+# camel-k: dependency=camel:aws-secrets-manager
+
+- route:
+ from:
+ uri: "kamelet:azure-storage-blob-cdc-source"
+ parameters:
+ accountName: "kameletsaccount"
+ containerName: "kamelets"
+ getBlob: true
+ topicOrQueueName: "kamelets_servicebus_queue"
+ connectionString: "RAW(<connectionString>)"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showStreams: "true"
diff --git a/jbang/azure-storage-blob-cdc/azure.prop b/jbang/azure-storage-blob-cdc/azure.prop
new file mode 100644
index 0000000..038c729
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/azure.prop
@@ -0,0 +1 @@
+camel.kamelet.azure-storage-blob-cdc-source.accessKey=
diff --git a/jbang/azure-storage-blob-cdc/terraform/main.tf b/jbang/azure-storage-blob-cdc/terraform/main.tf
new file mode 100644
index 0000000..cacd853
--- /dev/null
+++ b/jbang/azure-storage-blob-cdc/terraform/main.tf
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# We strongly recommend using the required_providers block to set the
+# Azure Provider source and version being used
+terraform {
+ required_providers {
+ azurerm = {
+ source = "hashicorp/azurerm"
+ version = "=3.0.0"
+ }
+ }
+}
+
+# Configure the Microsoft Azure Provider
+provider "azurerm" {
+ features {}
+}
+
+resource "azurerm_resource_group" "kamelets" {
+ name = "kamelets-resources"
+ location = "West Europe"
+}
+
+resource "azurerm_storage_account" "kamelets" {
+ name = "kameletsaccount"
+ resource_group_name = azurerm_resource_group.kamelets.name
+ location = azurerm_resource_group.kamelets.location
+ account_tier = "Standard"
+ account_replication_type = "LRS"
+}
+
+resource "azurerm_storage_container" "camelkamelets" {
+ name = "kamelets"
+ storage_account_name = azurerm_storage_account.kamelets.name
+ container_access_type = "private"
+}
+
+resource "azurerm_servicebus_namespace" "example" {
+ name = "kamelets-servicebus-namespace"
+ location = azurerm_resource_group.kamelets.location
+ resource_group_name = azurerm_resource_group.kamelets.name
+ sku = "Standard"
+}
+
+resource "azurerm_servicebus_queue" "example" {
+ name = "kamelets_servicebus_queue"
+ namespace_id = azurerm_servicebus_namespace.example.id
+
+ enable_partitioning = true
+}
+
+resource "azurerm_eventgrid_event_subscription" "example" {
+ name = "example-aees"
+ scope = azurerm_storage_account.kamelets.id
+
+ service_bus_queue_endpoint_id = azurerm_servicebus_queue.example.id
+
+ included_event_types = [
+ "Microsoft.Storage.BlobCreated", "Microsoft.Storage.BlobDeleted"
+ ]
+}