You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "matzew (via GitHub)" <gi...@apache.org> on 2024/12/03 08:50:16 UTC
[I] Ceph: support bucket notifications? [camel-kamelets]
matzew opened a new issue, #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303
The ceph source and sink from kamelet seems to be dealing with objects and not notifications.
We have something like this currently in Knative: https://github.com/knative-extensions/eventing-ceph/blob/main/pkg/adapter/adapter.go#L91
However, with the general move towards `eventing-integrations` and using Kamelets inside, I was wondering if it is an option to add bucket notifications to the kamelet catalog.
this can do much more than what the current eventing-ceph code does. e.g. configuring the notifications, and maybe integrate with the ceph-source so that objects will be consumed based on the notifications.
creating a notification based mechanism instead of polling (as the current ceph-source does) is much more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "yuvalif (via GitHub)" <gi...@apache.org>.
yuvalif commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-2517113293
yes, it is similar concept, but in case of ceph we ae using kafka not sqs.
also in cepoh we have some extensions to the bucket notification configuration that you might want to expose to the conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "yuvalif (via GitHub)" <gi...@apache.org>.
yuvalif commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-5630058767
> I would like to build the `ceph-event-based-source` — Kafka in, notification parsed, optional object fetch through the existing Ceph S3 path — unless you would rather it wait. [@yuvalif](https://github.com/yuvalif), a sample of the notification JSON your deployment produces would make the parsing exact rather than assumed.
this would be great. event structure from our docs: https://docs.ceph.com/en/latest/radosgw/notifications/#events
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-2514503719
We could do something like the event-based s3: https://github.com/apache/camel-kamelets/blob/main/kamelets/aws-s3-event-based-source.kamelet.yaml
There, we are consuming events from bucket and react to an upload. It should really be the same for Ceph. Does this answer your question?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-2517325745
The underline component is using AWS SDK S3 Java client, so unless there is a good reason or something useful on S3 too, we won't add more configuration in the component.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "yuvalif (via GitHub)" <gi...@apache.org>.
yuvalif commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-2518438935
> The underline component is using AWS SDK S3 Java client, so unless there is a good reason or something useful on S3 too, we won't add more configuration in the component.
to setup bucket notifications and the queue you would need S3 and SQS APIs. but this limits you to AWS, since there is no open source implementation of SQS.
in the case of ceph, the queue is provided by kafka, which is available to anyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-5581974861
Correcting my previous comment — I wrote it without reading the 2024 thread above, which was careless, and it shows in two places.
**What I got wrong.** I said "for Ceph the transport is whatever the operator configured, so there is no single wiring to capture". @yuvalif had already answered that: *"in case of ceph we are using kafka not sqs"*. Ceph RGW does support HTTP and AMQP, but the deployment being asked about uses Kafka, and that is a specific wiring, not an open-ended one.
I also presented the `aws-s3-event-based-source` analogy as a finding when @oscerd had made exactly that point in the first reply, two years ago.
**What that changes.** My "option 2" — a Ceph event-based source — was the weaker of the two options in my comment, on the grounds that it would be "kafka-source with a different name". That reasoning does not survive the correction. `aws-s3-event-based-source` is not "sqs-source with a different name" either: it consumes the queue, parses the S3 event, and optionally fetches the object that the event points at. A Ceph equivalent would do the same three things with Kafka in place of SQS.
So by the catalog's own precedent the Kamelet is justified, and @yuvalif's point sharpens it — the AWS variant is locked to SQS, whereas a Kafka-backed one works against any Ceph deployment.
**What stands.** The two things I checked still hold, and neither is affected by the thread:
- there is no `camel-ceph` component; `ceph-source` / `ceph-sink` are `aws2-s3` with `uriEndpointOverride`, so the object-fetch half is already available
- `camel-aws2-s3` exposes no notification-configuration operation, so *configuring* the notification stays outside Camel — which is consistent with @oscerd's position in this thread that the component will not grow Ceph-specific configuration
That leaves @yuvalif's remark about *"extensions to the bucket notification configuration that you might want to expose"* as the one open question I cannot resolve from here: those extensions are set when creating the notification, which is the part that is not a Camel route. If they also appear **in the delivered event payload**, a Kamelet could surface them and that is worth knowing; if they are only configuration, the earlier answer in this thread applies.
I would like to build the `ceph-event-based-source` — Kafka in, notification parsed, optional object fetch through the existing Ceph S3 path — unless you would rather it wait. @yuvalif, a sample of the notification JSON your deployment produces would make the parsing exact rather than assumed.
Apologies for the noise of the previous comment.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "yuvalif (via GitHub)" <gi...@apache.org>.
yuvalif commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-5581936527
@oscerd why is `PutBucketNotificationConfiguration` different than the long list bucket level admin calls already supported?
```
deleteBucket
createBucket
putBucketTagging,
deleteBucketTagging
putBucketVersioning
putBucketPolicy
deleteBucketPolicy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2303:
URL: https://github.com/apache/camel-kamelets/issues/2303#issuecomment-5581113902
Looked into what this would actually take. The short version: the consuming half already works today by composition, and the part that does not exist is not a Camel route at all.
## There is no Ceph component to extend
`ceph-source` and `ceph-sink` are not backed by a `camel-ceph` component — there is none in the catalog. They are `aws2-s3` pointed at the RGW endpoint through Ceph's S3-compatible API:
```yaml
dependencies:
- "camel:aws2-s3"
template:
from:
uri: "aws2-s3:{{bucketName}}"
parameters:
uriEndpointOverride: "{{cephUrl}}"
overrideEndpoint: "true"
```
So "add bucket notifications to the Ceph Kamelets" has no component-level hook to hang off. It splits into two quite different problems.
## Receiving notifications: already possible
Ceph RGW **pushes** bucket notifications to a configured endpoint — HTTP, AMQP 0.9.1/1.0, or Kafka. Camel does not poll for them, it receives them, and the catalog already has a source for each of those transports:
```
http-source webhook-source kafka-source jms-amqp-10-source
```
So the event-driven pipeline the issue asks for — "a notification based mechanism instead of polling" — is available now by pointing Ceph at, say, a Kafka topic and binding `kafka-source`. The payload is S3-event shaped JSON, so it can be filtered and routed with the existing action Kamelets.
That is worth stating plainly, because it means nobody is blocked.
## Configuring notifications: not a route
The remaining piece — "configuring the notifications" — is an S3 `PutBucketNotificationConfiguration` call against the bucket. That is administration, done once, not per-message, and `camel-aws2-s3` does not expose it. Its full operation list is:
```
copyObject, listObjects, deleteObject, deleteObjects, deleteBucket, listBuckets,
getObject, getObjectRange, createDownloadLink, createUploadLink, headBucket,
headObject, restoreObject, getObjectTagging, putObjectTagging, deleteObjectTagging,
getObjectAcl, putObjectAcl, createBucket, getBucketTagging, putBucketTagging,
deleteBucketTagging, getBucketVersioning, putBucketVersioning, getBucketPolicy,
putBucketPolicy, deleteBucketPolicy
```
No notification operation among them. Adding one would be a change to `camel-aws2-s3` in `apache/camel`, and even then, wrapping a one-off admin call in a Kamelet is a poor fit — Kamelets are route templates, and this is closer to `aws cli` or Terraform territory.
## What could reasonably be added here
If the goal is to make the composition obvious rather than to invent a component:
1. **A documented Pipe example** showing Ceph notifications landing on Kafka or an HTTP endpoint and being consumed — nothing new in the catalog, but it removes the discovery problem, which reads like the real complaint.
2. **A `ceph-notification-source`** that is a thin wrapper over one transport with Ceph-shaped defaults. Honest question whether that earns its place: it would be `kafka-source` with a different name, and the catalog already resists that kind of aliasing.
The comparison with `aws-s3-event-based-source` is instructive — that one exists because AWS routes events through SQS and there is a specific, non-obvious wiring worth encapsulating. For Ceph the transport is whatever the operator configured, so there is no single wiring to capture.
Leaving this open for a maintainer decision rather than closing it, since option 1 is cheap and might be exactly what was wanted. Happy to write that example up if you think it is worth having.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ceph: support bucket notifications? [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd closed issue #2303: Ceph: support bucket notifications?
URL: https://github.com/apache/camel-kamelets/issues/2303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org