You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "shubhamcoc (via GitHub)" <gi...@apache.org> on 2023/08/29 12:27:33 UTC
[GitHub] [camel-kafka-connector] shubhamcoc opened a new issue, #1560: How to create an object in s3 bucket using camle-minio-sink plugin
shubhamcoc opened a new issue, #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560
Hi,
I am trying to take backup of multiple Kafka topics data in s3 bucket, current observation is, minio-sink plugin is creating random file in same bucket without storing the metadata, as a result, during the restore all the data saved into bucket will be restore in all the topics.
I can see in the [kamalet yaml](https://github.com/apache/camel-kafka-connector/blob/main/connectors/camel-minio-sink-kafka-connector/src/main/resources/kamelets/minio-sink.kamelet.yaml), it is mention, we have to set header properties called file to upload the data into the filename. Do anyone know how to configure it?
I find one property to set the keyname in endpoint, but it is not working as intended. There is a similar question in stack-overflow https://stackoverflow.com/questions/74662272/kafka-connect-camel-s3-sink-keyname-attribute-behavior.
Any help is appreciated. Thanks in advance. Kindly let me know if it is possible to store data into a particular object in a bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698608686
Not, ootb, if you want to do something like that you should use plain camel and aggregate the messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698639344
There is also the minio component there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698596711
How are you sending the data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698627286
> Or you can try the aggregation in CKC: https://camel.apache.org/camel-kafka-connector/3.18.x/user-guide/aggregation.html
ok I will check the aggregation first, if it helps then fine or else will check camel-core component. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc closed issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc closed issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
URL: https://github.com/apache/camel-kafka-connector/issues/1560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698612898
Or you can try the aggregation in CKC: https://camel.apache.org/camel-kafka-connector/3.18.x/user-guide/aggregation.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698600447
> using the command
>
> ```
> kcat -P -b localhost:9092 -X security.protocol=SSL -X ssl.key.location=/ssl/certsClient/tls.key -X ssl.certificate.location=/ssl/certsClient/tls.crt -X ssl.ca.location=/ssl/certsCl
> ient/ca.crt -t kafkatopic-sample -H "CamelHeader.CamelMinioObjectName=kafkatopic-sample-test"
> ```
My bad the correct header name is file
So it should be
```
kcat -P -b localhost:9092 -X security.protocol=SSL -X ssl.key.location=/ssl/certsClient/tls.key -X ssl.certificate.location=/ssl/certsClient/tls.crt -X ssl.ca.location=/ssl/certsClient/ca.crt -t kafkatopic-sample -H "CamelHeader.file=kafkatopic-sample-test"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698628354
> > What do you mean by plain camel? can we connect to s3 using plain camel without plugins?
>
> For sure you can. https://camel.apache.org/components/4.0.x/aws2-s3-component.html
This is for aws S3, unfortunately ours is not hosted on AWS, and I am not sure if it is generic for all S3 storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698593922
Hi @oscerd, I tried to send the data using kafkacat with header, as suggested in above comment, but it is still uploading data using exchange ID. Do we need to configure anything related to header configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1697913400
Thanks @oscerd for the response, I think it will be helpful if you add it in the documentation as well. I will try it out and will ask again if we face some issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1699300814
Thanks, closing the ticket as the solution is working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1697706287
You need to use an header with the minio sink Kamelet.
Supposing you're using kafkacat while moving the file, you should set a kafka header in your record:
./kafkacat -P -b localhost:9092 -t mytopic -H "CamelHeader.CamelMinioObjectName=fileName"
And the record value will be stored.
I'll add the ability to set the objectName statically too in the Kamelet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698611667
I mean using Apache Camel without kafka connect. https://camel.apache.org/camel-core/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698604876
Thanks, this time, it worked, another question is, whenever we send a msg, the file gets overwrite, is there any way to append the messages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698610865
What do you mean by plain camel? can we connect to s3 using plain camel without plugins?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698615258
> What do you mean by plain camel? can we connect to s3 using plain camel without plugins?
For sure you can. https://camel.apache.org/components/4.0.x/aws2-s3-component.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] shubhamcoc commented on issue #1560: How to create an object in s3 bucket using camle-minio-sink plugin
Posted by "shubhamcoc (via GitHub)" <gi...@apache.org>.
shubhamcoc commented on issue #1560:
URL: https://github.com/apache/camel-kafka-connector/issues/1560#issuecomment-1698599233
using the command
```
kcat -P -b localhost:9092 -X security.protocol=SSL -X ssl.key.location=/ssl/certsClient/tls.key -X ssl.certificate.location=/ssl/certsClient/tls.crt -X ssl.ca.location=/ssl/certsCl
ient/ca.crt -t kafkatopic-sample -H "CamelHeader.CamelMinioObjectName=kafkatopic-sample-test"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org