You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/11/24 10:02:06 UTC

[GitHub] [camel-kafka-connector] jaapterwoerds opened a new issue #1292: Ensure Kafka metadata is avaiable in Sink task as in a standalone Kafka Camel source

jaapterwoerds opened a new issue #1292:
URL: https://github.com/apache/camel-kafka-connector/issues/1292


   I was surprised I don't seem to be able to use the key, offset, partition and topic like I could when using Camel standalone.
   
   I've got some code laying around so I can use this in a dynamic file name. I couldn't get the escaping of the Kafka header names to work(these constant values contain a dot, so that's why I l replace it by underscores here. I read somewhere this is a know limitation in camel, by I couldn't find the reference to that doc back.
   ```
   @Override
   public void put(Collection<SinkRecord> sinkRecords) {
   ...
       for (SinkRecord record : sinkRecords) {
   ...
   
           exchange.getMessage().setHeader(KafkaConstants.PARTITION.replace('.','_'), record.kafkaPartition());
           exchange.getMessage().setHeader(KafkaConstants.TOPIC.replace('.','_'), record.topic());
           exchange.getMessage().setHeader(KafkaConstants.OFFSET.replace('.','_'), record.kafkaOffset());
           exchange.getMessage().setHeader(KafkaConstants.HEADERS.replace('.','_'), record.headers());
           exchange.getMessage().setHeader(KafkaConstants.TIMESTAMP.replace('.','_'), record.timestamp());
   ```
   And in the connector configuration:
   ```
   "camel.sink.path.fileName":"${header[kafka_TOPIC]}-${header[kafka_PARTITION]}-${header[kafka_OFFSET]}"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #1292: Ensure Kafka metadata is avaiable in Sink task as in a standalone Kafka Camel source

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1292:
URL: https://github.com/apache/camel-kafka-connector/issues/1292#issuecomment-977806347


   @jaapterwoerds thanks for reporting this, I guess we should do something like this in the general Sink connector. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] valdar commented on issue #1292: Ensure Kafka metadata is avaiable in Sink task as in a standalone Kafka Camel source

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1292:
URL: https://github.com/apache/camel-kafka-connector/issues/1292#issuecomment-977921100


   @jaapterwoerds would you be interested in contributing this feature (with all guidance and support I gang give) ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] valdar commented on issue #1292: Ensure Kafka metadata is avaiable in Sink task as in a standalone Kafka Camel source

Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1292:
URL: https://github.com/apache/camel-kafka-connector/issues/1292#issuecomment-977920146


   AH nice! +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #1292: Ensure Kafka metadata is avaiable in Sink task as in a standalone Kafka Camel source

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1292:
URL: https://github.com/apache/camel-kafka-connector/issues/1292#issuecomment-977805615


   @valdar I think this would make sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org