You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/06/26 11:20:42 UTC

[GitHub] [camel-kafka-connector] saachinsiva opened a new issue #1224: Access headers on S3 Source connector

saachinsiva opened a new issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224


   Hello
   
   I have a requirement where json files will be published by our client into a bucket which should then be published to Kafka topic and then we have lot of consumers who consumes this message and one of the consumers is Elastic Search. Currently, I use ByteArrayConverter but provided that this will always be a json , is there anyway to use actual Json converter as it allows me to use SMT to manipulate the json fields. Also the name of the file is the unique id in the S3 bucket and I am wonder if there is a way to use the file name as Kafka message key. I can see that S3 file name is coming as one of the headers but I am not sure how can I use that header as key. Please advise.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] saachinsiva edited a comment on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
saachinsiva edited a comment on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869820997


   @oscerd 
   
   I tried using marshal but I still set the value as bytes in the topic. Also Can you please let me know which SMT to use to get the header value.
   
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.json.JsonConverter",
       "value.converter.schemas.enable":"false",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true,
       "camel.source.marshal":"json-jackson"
   }
   ```
   
   Ouptut (using AKHQ):
   
   ![image](https://user-images.githubusercontent.com/7923274/123667853-8bb27b00-d808-11eb-883a-73ebf4ab67f1.png)
   
   
   However if I use byte array converter without marshal then I see as below but ofcourse I cannot use SMT to parse field
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true
   }
   ```
   
   ![image](https://user-images.githubusercontent.com/7923274/123669807-7c343180-d80a-11eb-86a7-d25398de0bda.png)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869349990


   You can add
   
   camel.source.marshal=json-jackson
   
   To your configuration to get a json. In your SMT, you can get the header from the record, and set it as key.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] saachinsiva edited a comment on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
saachinsiva edited a comment on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869820997


   @oscerd 
   
   I tried using marshal but I still set the value as bytes in the topic. Also Can you please let me know which SMT to use to get the header value.
   
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.json.JsonConverter",
       "value.converter.schemas.enable":"false",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true,
       "camel.source.marshal":"json-jackson"
   }
   ```
   
   Ouptut (using AKHQ):
   
   ![image](https://user-images.githubusercontent.com/7923274/123667853-8bb27b00-d808-11eb-883a-73ebf4ab67f1.png)
   
   
   However if I use byte array converter without marshal then I see as below but ofcourse I cannot use SMT to parse field
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true
   }
   ```
   
   ![image](https://user-images.githubusercontent.com/7923274/123669807-7c343180-d80a-11eb-86a7-d25398de0bda.png)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] saachinsiva commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
saachinsiva commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869820997


   @oscerd 
   
   I tried using marshal but I still set the value as bytes in the topic. Also please me know which SMT to use to get the header value.
   
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.json.JsonConverter",
       "value.converter.schemas.enable":"false",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true,
       "camel.source.marshal":"json-jackson"
   }
   ```
   
   Ouptut (using AKHQ):
   
   ![image](https://user-images.githubusercontent.com/7923274/123667853-8bb27b00-d808-11eb-883a-73ebf4ab67f1.png)
   
   
   However if I use byte array converter without marshal then I see as below but ofcourse I cannot use SMT to parse field
   ```
   {
       "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector",
       "tasks.max": "1",
       "topics": "local.ssl.test",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
       "camel.source.path.bucketNameOrArn": "test-bucket",
       "camel.component.aws2-s3.includeBody": true,
       "camel.component.aws2-s3.delimiter": "/",
       "camel.component.aws2-s3.prefix": "local/",
       "camel.source.endpoint.useDefaultCredentialsProvider": true,
       "camel.source.endpoint.autocloseBody": true
   }
   ```
   
   ![image](https://user-images.githubusercontent.com/7923274/123669807-7c343180-d80a-11eb-86a7-d25398de0bda.png)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869834035


   For the smt you have to write it. We cannot put the whole file content as string, what happen if you have 10 gb files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] saachinsiva commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
saachinsiva commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869820997






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd closed issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
oscerd closed issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] saachinsiva commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
saachinsiva commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869929172


   @oscerd 
   
   Understood.  At least for my case I can live with it as our json will always be small in our use case and we can manage this with our own SMT.
   
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd commented on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869834035


   For the smt you have to write it. We cannot put the whole file content as string, what happen if you have 10 gb files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel-kafka-connector] oscerd edited a comment on issue #1224: Access headers on S3 Source connector

Posted by GitBox <gi...@apache.org>.
oscerd edited a comment on issue #1224:
URL: https://github.com/apache/camel-kafka-connector/issues/1224#issuecomment-869349990


   You can add
   
   camel.source.marshal=json-jackson
   
   To your configuration to get a json. In your SMT, you can get the file name header from the record, and set it as key.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org