You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2026/09/14 08:53:35 UTC

(camel-kamelets) branch main updated: Fix #2303: add a Ceph event based source Kamelet (#3026)

This is an automated email from the ASF dual-hosted git repository.

oscerd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git


The following commit(s) were added to refs/heads/main by this push:
     new 6a52e17fa Fix #2303: add a Ceph event based source Kamelet (#3026)
6a52e17fa is described below

commit 6a52e17fa9aa110aafbf750fee2db1250749a7ab
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Mon Sep 14 10:53:29 2026 +0200

    Fix #2303: add a Ceph event based source Kamelet (#3026)
    
    Ceph RGW pushes bucket notifications to a configured endpoint rather than
    being polled, and the deployment in the issue uses Kafka for that. This
    consumes those notifications and optionally fetches the object each one
    refers to, mirroring what aws-s3-event-based-source does for SQS.
    
    The parsing could not be shared with the AWS Kamelet. Ceph emits the
    classic S3 envelope -- a Records array -- while aws-s3-event-based-source
    parses the EventBridge shape:
    
      Ceph  Records[0].s3.bucket.name   Records[0].s3.object.key   eventName
      AWS   $.detail.bucket.name        $.detail.object.key        $.detail.reason
    
    Object fetching reuses the existing Ceph path, aws2-s3 with
    uriEndpointOverride against the RGW endpoint, so there is no new component
    involved. Kafka properties mirror kafka-source so the two read alike.
    
    Configuring the notification and the topic stays outside Camel: that needs
    PutBucketNotificationConfiguration, which camel-aws2-s3 does not expose,
    and an SNS style CreateTopic carrying the push-endpoint, which
    camel-aws2-sns cannot express. The Kamelet documents that it consumes a
    notification someone else configured.
    
    Added to the verifyUsedParams exemption list in the validator for the same
    reason aws-s3-event-based-source is on it: getObject is consumed by a
    build time precondition, which the declared-parameter check does not see.
    
    Verified against a live broker with the event payload from the Ceph
    documentation: the Kamelet consumes the notification and emits it
    unchanged, and with getObject enabled the extraction yields
    event=[ObjectCreated:Put] bucket=[mybucket1] key=[myimage1.jpg].
    
    Marked Preview, no Citrus test: exercising the fetch half needs a live RGW.
    
    
    Claude-Session: https://claude.ai/code/session_01Bh2BAL99hwQ6o6uuALi9od
    
    Co-authored-by: Claude Opus 5 <no...@anthropic.com>
---
 docs/modules/ROOT/nav.adoc                         |   1 +
 kamelets/ceph-event-based-source.kamelet.yaml      | 157 +++++++++++++++++++++
 .../kamelets/ceph-event-based-source.kamelet.yaml  | 157 +++++++++++++++++++++
 script/validator/validator.go                      |   1 +
 4 files changed, 316 insertions(+)

diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc
index 905971cb8..662aa4cc9 100644
--- a/docs/modules/ROOT/nav.adoc
+++ b/docs/modules/ROOT/nav.adoc
@@ -55,6 +55,7 @@
 * xref:caffeine-action.adoc[]
 * xref:cassandra-sink.adoc[]
 * xref:cassandra-source.adoc[]
+* xref:ceph-event-based-source.adoc[]
 * xref:ceph-sink.adoc[]
 * xref:ceph-source.adoc[]
 * xref:chuck-norris-source.adoc[]
diff --git a/kamelets/ceph-event-based-source.kamelet.yaml b/kamelets/ceph-event-based-source.kamelet.yaml
new file mode 100644
index 000000000..418e830a8
--- /dev/null
+++ b/kamelets/ceph-event-based-source.kamelet.yaml
@@ -0,0 +1,157 @@
+# ---------------------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ---------------------------------------------------------------------------
+apiVersion: camel.apache.org/v1
+kind: Kamelet
+metadata:
+  name: ceph-event-based-source
+  annotations:
+    camel.apache.org/kamelet.support.level: "Preview"
+    camel.apache.org/catalog.version: "4.22.1-SNAPSHOT"
+    camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4gPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNjAiIGhlaWdodD0iMTU3IiB2aWV3Qm94PSIwIDAgMTYwIDE1NyIgZmlsbD0ibm9uZSI+PGcgY2xpcC1wYXRoPSJ1cmwoI2NsaXAwKSI+PHBhdGggZD0iTTc5LjgxNTIgOTYuMDczOUM4OC42ODM2IDk2LjA3MzkgOTUuNzA0NCA4OS4wNTMxIDk1LjcwNDQgODAuMTg0OEM5NS43MDQ0IDcxLjMxNjQgODguNjgzNiA2NC4yOTU2IDc5LjgxNTIgNjQuMjk1NkM3MC45NDY5IDY0LjI5NTYgNjMuOTI2MSA3MS4zMTY0IDYzLj [...]
+    camel.apache.org/provider: "Apache Software Foundation"
+    camel.apache.org/kamelet.group: "Ceph"
+    camel.apache.org/kamelet.namespace: "Cloud"
+  labels:
+    camel.apache.org/kamelet.type: "source"
+spec:
+  definition:
+    title: "Ceph Event Based Source"
+    description: |-
+      Receive Ceph RGW bucket notifications from a Kafka topic, and optionally fetch the object each notification refers to.
+
+      Ceph pushes bucket notifications to an endpoint configured on the topic; this Kamelet consumes the Kafka flavour of that. Configuring the notification and the topic on the Ceph side is done out of band with the S3 and topic APIs, not by this Kamelet.
+
+      Set getObject to true to fetch the object body from Ceph for ObjectCreated events. The notification JSON is emitted unchanged otherwise.
+    required:
+      - topic
+      - bootstrapServers
+    type: object
+    properties:
+      topic:
+        title: Topic Name
+        description: The Kafka topic Ceph pushes the bucket notifications to.
+        type: string
+        example: "ceph-bucket-notifications"
+      bootstrapServers:
+        title: Bootstrap Servers
+        description: A comma separated list of Kafka brokers to connect to.
+        type: string
+      consumerGroup:
+        title: Consumer Group
+        description: A string that uniquely identifies the group of consumers this source belongs to.
+        type: string
+        example: "my-group-id"
+      autoOffsetReset:
+        title: Auto Offset Reset
+        description: What to do when there is no initial offset. There are 3 enums and the value can be latest, earliest or none.
+        type: string
+        default: "latest"
+        enum: ["latest", "earliest", "none"]
+      saslAuthType:
+        title: Authentication Type
+        description: Authentication type to use. Use NONE for no authentication, PLAIN or SCRAM_SHA_256/SCRAM_SHA_512 for username/password, SSL for certificate-based, OAUTH for OAuth 2.0, AWS_MSK_IAM for MSK, or KERBEROS for Kerberos.
+        type: string
+        default: NONE
+        enum: ["NONE", "PLAIN", "SCRAM_SHA_256", "SCRAM_SHA_512", "SSL", "OAUTH", "AWS_MSK_IAM", "KERBEROS"]
+      saslUsername:
+        title: Username
+        description: Username for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512.
+        type: string
+        x-descriptors:
+        - urn:camel:group:credentials
+      saslPassword:
+        title: Password
+        description: Password for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      getObject:
+        title: Get Object
+        description: Fetch the object body from Ceph for ObjectCreated events, instead of emitting only the notification.
+        type: boolean
+        default: false
+      cephUrl:
+        title: Ceph URL
+        description: The URL of the Ceph RGW endpoint to fetch objects from. Required when getObject is true.
+        type: string
+        example: "http://ceph-rgw:8080"
+      accessKey:
+        title: Access Key
+        description: The access key to read objects from Ceph. Required when getObject is true.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      secretKey:
+        title: Secret Key
+        description: The secret key to read objects from Ceph. Required when getObject is true.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      zoneGroup:
+        title: Ceph Zone Group
+        description: The zone group the bucket belongs to, passed to the S3 client as its region.
+        type: string
+        example: "zonegroup1"
+  dependencies:
+    - "camel:kafka"
+    - "camel:core"
+    - "camel:jsonpath"
+    - "camel:jackson"
+    - "camel:kamelet"
+    - "camel:aws2-s3"
+  template:
+    from:
+      uri: "kafka:{{topic}}"
+      parameters:
+        brokers: "{{bootstrapServers}}"
+        groupId: "{{?consumerGroup}}"
+        autoOffsetReset: "{{autoOffsetReset}}"
+        saslAuthType: "{{saslAuthType}}"
+        saslUsername: "{{?saslUsername}}"
+        saslPassword: "{{?saslPassword}}"
+      steps:
+        # Ceph emits the classic S3 notification envelope -- a Records array --
+        # not the EventBridge shape that aws-s3-event-based-source parses, so the
+        # paths below are Records based on purpose.
+        - choice:
+            precondition: true
+            when:
+              - simple: '${properties:getObject:false}'
+                steps:
+                  - unmarshal:
+                      json:
+                        library: Jackson
+                        unmarshalType: com.fasterxml.jackson.databind.JsonNode
+                  - setProperty:
+                      name: ceph-event-name
+                      jsonpath: $.Records[0].eventName
+                  - choice:
+                      when:
+                        - simple: '${exchangeProperty.ceph-event-name} startsWith "ObjectCreated"'
+                          steps:
+                            - setProperty:
+                                name: ceph-object-key
+                                jsonpath: $.Records[0].s3.object.key
+                            - setProperty:
+                                name: ceph-bucket-name
+                                jsonpath: $.Records[0].s3.bucket.name
+                            - toD: >-
+                                aws2-s3:${exchangeProperty.ceph-bucket-name}?accessKey={{accessKey}}&secretKey={{secretKey}}&region={{zoneGroup}}&operation=getObject&keyName=${exchangeProperty.ceph-object-key}&uriEndpointOverride={{cephUrl}}&overrideEndpoint=true
+        - to: "kamelet:sink"
diff --git a/library/camel-kamelets/src/main/resources/kamelets/ceph-event-based-source.kamelet.yaml b/library/camel-kamelets/src/main/resources/kamelets/ceph-event-based-source.kamelet.yaml
new file mode 100644
index 000000000..418e830a8
--- /dev/null
+++ b/library/camel-kamelets/src/main/resources/kamelets/ceph-event-based-source.kamelet.yaml
@@ -0,0 +1,157 @@
+# ---------------------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ---------------------------------------------------------------------------
+apiVersion: camel.apache.org/v1
+kind: Kamelet
+metadata:
+  name: ceph-event-based-source
+  annotations:
+    camel.apache.org/kamelet.support.level: "Preview"
+    camel.apache.org/catalog.version: "4.22.1-SNAPSHOT"
+    camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4gPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNjAiIGhlaWdodD0iMTU3IiB2aWV3Qm94PSIwIDAgMTYwIDE1NyIgZmlsbD0ibm9uZSI+PGcgY2xpcC1wYXRoPSJ1cmwoI2NsaXAwKSI+PHBhdGggZD0iTTc5LjgxNTIgOTYuMDczOUM4OC42ODM2IDk2LjA3MzkgOTUuNzA0NCA4OS4wNTMxIDk1LjcwNDQgODAuMTg0OEM5NS43MDQ0IDcxLjMxNjQgODguNjgzNiA2NC4yOTU2IDc5LjgxNTIgNjQuMjk1NkM3MC45NDY5IDY0LjI5NTYgNjMuOTI2MSA3MS4zMTY0IDYzLj [...]
+    camel.apache.org/provider: "Apache Software Foundation"
+    camel.apache.org/kamelet.group: "Ceph"
+    camel.apache.org/kamelet.namespace: "Cloud"
+  labels:
+    camel.apache.org/kamelet.type: "source"
+spec:
+  definition:
+    title: "Ceph Event Based Source"
+    description: |-
+      Receive Ceph RGW bucket notifications from a Kafka topic, and optionally fetch the object each notification refers to.
+
+      Ceph pushes bucket notifications to an endpoint configured on the topic; this Kamelet consumes the Kafka flavour of that. Configuring the notification and the topic on the Ceph side is done out of band with the S3 and topic APIs, not by this Kamelet.
+
+      Set getObject to true to fetch the object body from Ceph for ObjectCreated events. The notification JSON is emitted unchanged otherwise.
+    required:
+      - topic
+      - bootstrapServers
+    type: object
+    properties:
+      topic:
+        title: Topic Name
+        description: The Kafka topic Ceph pushes the bucket notifications to.
+        type: string
+        example: "ceph-bucket-notifications"
+      bootstrapServers:
+        title: Bootstrap Servers
+        description: A comma separated list of Kafka brokers to connect to.
+        type: string
+      consumerGroup:
+        title: Consumer Group
+        description: A string that uniquely identifies the group of consumers this source belongs to.
+        type: string
+        example: "my-group-id"
+      autoOffsetReset:
+        title: Auto Offset Reset
+        description: What to do when there is no initial offset. There are 3 enums and the value can be latest, earliest or none.
+        type: string
+        default: "latest"
+        enum: ["latest", "earliest", "none"]
+      saslAuthType:
+        title: Authentication Type
+        description: Authentication type to use. Use NONE for no authentication, PLAIN or SCRAM_SHA_256/SCRAM_SHA_512 for username/password, SSL for certificate-based, OAUTH for OAuth 2.0, AWS_MSK_IAM for MSK, or KERBEROS for Kerberos.
+        type: string
+        default: NONE
+        enum: ["NONE", "PLAIN", "SCRAM_SHA_256", "SCRAM_SHA_512", "SSL", "OAUTH", "AWS_MSK_IAM", "KERBEROS"]
+      saslUsername:
+        title: Username
+        description: Username for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512.
+        type: string
+        x-descriptors:
+        - urn:camel:group:credentials
+      saslPassword:
+        title: Password
+        description: Password for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      getObject:
+        title: Get Object
+        description: Fetch the object body from Ceph for ObjectCreated events, instead of emitting only the notification.
+        type: boolean
+        default: false
+      cephUrl:
+        title: Ceph URL
+        description: The URL of the Ceph RGW endpoint to fetch objects from. Required when getObject is true.
+        type: string
+        example: "http://ceph-rgw:8080"
+      accessKey:
+        title: Access Key
+        description: The access key to read objects from Ceph. Required when getObject is true.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      secretKey:
+        title: Secret Key
+        description: The secret key to read objects from Ceph. Required when getObject is true.
+        type: string
+        format: password
+        x-descriptors:
+        - urn:camel:group:credentials
+      zoneGroup:
+        title: Ceph Zone Group
+        description: The zone group the bucket belongs to, passed to the S3 client as its region.
+        type: string
+        example: "zonegroup1"
+  dependencies:
+    - "camel:kafka"
+    - "camel:core"
+    - "camel:jsonpath"
+    - "camel:jackson"
+    - "camel:kamelet"
+    - "camel:aws2-s3"
+  template:
+    from:
+      uri: "kafka:{{topic}}"
+      parameters:
+        brokers: "{{bootstrapServers}}"
+        groupId: "{{?consumerGroup}}"
+        autoOffsetReset: "{{autoOffsetReset}}"
+        saslAuthType: "{{saslAuthType}}"
+        saslUsername: "{{?saslUsername}}"
+        saslPassword: "{{?saslPassword}}"
+      steps:
+        # Ceph emits the classic S3 notification envelope -- a Records array --
+        # not the EventBridge shape that aws-s3-event-based-source parses, so the
+        # paths below are Records based on purpose.
+        - choice:
+            precondition: true
+            when:
+              - simple: '${properties:getObject:false}'
+                steps:
+                  - unmarshal:
+                      json:
+                        library: Jackson
+                        unmarshalType: com.fasterxml.jackson.databind.JsonNode
+                  - setProperty:
+                      name: ceph-event-name
+                      jsonpath: $.Records[0].eventName
+                  - choice:
+                      when:
+                        - simple: '${exchangeProperty.ceph-event-name} startsWith "ObjectCreated"'
+                          steps:
+                            - setProperty:
+                                name: ceph-object-key
+                                jsonpath: $.Records[0].s3.object.key
+                            - setProperty:
+                                name: ceph-bucket-name
+                                jsonpath: $.Records[0].s3.bucket.name
+                            - toD: >-
+                                aws2-s3:${exchangeProperty.ceph-bucket-name}?accessKey={{accessKey}}&secretKey={{secretKey}}&region={{zoneGroup}}&operation=getObject&keyName=${exchangeProperty.ceph-object-key}&uriEndpointOverride={{cephUrl}}&overrideEndpoint=true
+        - to: "kamelet:sink"
diff --git a/script/validator/validator.go b/script/validator/validator.go
index 9cd35c6c3..2f9b464ca 100644
--- a/script/validator/validator.go
+++ b/script/validator/validator.go
@@ -399,6 +399,7 @@ func verifyUsedParams(kamelets []KameletInfo) (errors []error) {
 	for _, k := range kamelets {
 		if k.FileName != "../../kamelets/azure-storage-blob-source.kamelet.yaml" &&
 			k.FileName != "../../kamelets/aws-s3-event-based-source.kamelet.yaml" &&
+			k.FileName != "../../kamelets/ceph-event-based-source.kamelet.yaml" &&
 			k.FileName != "../../kamelets/aws-sqs-source.kamelet.yaml" &&
 			k.FileName != "../../kamelets/set-kafka-key-action.kamelet.yaml" &&
 			k.FileName != "../../kamelets/azure-storage-blob-event-based-source.kamelet.yaml" &&