You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2022/03/10 07:51:39 UTC
[GitHub] [camel-kafka-connector] akdevops opened a new issue #1346: camel kafka connector - s3 source
akdevops opened a new issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346
We are planning to use camel-aws2-s3-kafka-connector-0.11.5-package.tar.gz with strimzi/kafka-3.0.0 version.
With this version we want separate (new) line while consuming the data in kafka topic.
How can we fulfill this requirement? Please help with existing (if any) solutions.
eg.
**file added to s3 bucket - test.txt**
```
{
"Product": "XXX",
"ProductVertical": "XXX",
"Payload": {
"TrackingID": "XXX",
"Name": "XXX",
"UpdatedAt": "2016-04-26T19:48:40Z"
}
}
{
"Product": "XXX",
"ProductVertical": "XXX",
"Payload": {
"TrackingID": "XXX",
"Name": "XXX",
"UpdatedAt": "2016-04-26T19:48:40Z"
}
}
```
**Expecting output after consuming the data in kafkatopic -** (each record should be in new separate line)
```
{"Product":"XXX","ProductVertical":"XXX","Timestamp":"2022-03-02","Payload":{"TrackingID":XXX"....}}
{"Product":"XXX","ProductVertical":"XXX","Timestamp":"2022-03-02","Payload":{"TrackingID":XXX"....}}
```
**Connector configuration -**
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: s3-source-connector
namespace: test
labels:
strimzi.io/cluster: test-connect
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
tasksMax: 1
config:
topics: s3-test-topic
value.converter: org.apache.kafka.connect.converters.ByteArrayConverter
camel.source.path.bucketNameOrArn: quickstart-kafka-s3-connector
camel.source.endpoint.region: us-west-2
camel.source.endpoint.useDefaultCredentialsProvider: true
camel.component.aws2-s3.amazonS3Client: true
camel.component.aws2-s3.includeFolders: false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] akdevops edited a comment on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
akdevops edited a comment on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063965760
Okay. Can you please share any example for the transformer (mentioned above) or any ref. of how we can implement? Also just wanted to know if there are any limitations like max file size etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] akdevops edited a comment on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
akdevops edited a comment on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063965760
Okay. Can you please share any example for the transformer (mentioned above) or any ref. of how we can implement? Also just wanted to know if there are any limitations like max file size, file type (zip, csv) etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063959135
No, the only way is writing an SMT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] akdevops commented on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
akdevops commented on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063965760
Okay. Can you please share any example for the transformer (mentioned above)? Also just wanted to know if there are any limitations like max file size etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063923399
I think you can use a transformer for that. Here's one example (it's for the File connector, but the principle is the same):
https://github.com/apache/camel-kafka-connector/blob/main/connectors/camel-file-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/file/transformers/FileTransforms.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] akdevops commented on issue #1346: camel kafka connector - s3 source
Posted by GitBox <gi...@apache.org>.
akdevops commented on issue #1346:
URL: https://github.com/apache/camel-kafka-connector/issues/1346#issuecomment-1063952832
Thank you for your quick response. I will try this for sure, also is it not possible with the existing s3 source connector itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org