You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/02/09 09:34:24 UTC

[GitHub] [camel-kafka-connector] swierszczu4j opened a new issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

swierszczu4j opened a new issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988


   As title says - I have a problem in which I would like to rewrite kafka record (key + payload) from one topic to another, but no matter if I'm using Sink or Source option of this connector, the outbound topic receives message without a key (key ends up being null), which also means partition ordering guarantee is lost.
   
   My setup:
   `
   curl --location --request PUT 'http://localhost:8083/connectors/camel-sink-connector/config' \
   --header 'Content-Type: application/json' \
   --data-raw '{
       "connector.class": "org.apache.camel.kafkaconnector.kafka.CamelKafkaSinkConnector",
       "tasks.max": "1",
       "camel.sink.contentLogLevel": "DEBUG",
       "camel.sink.path.topic": "example.topic.2",
       "camel.sink.endpoint.brokers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
       "topics": "example.topic",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.storage.StringConverter",
       "consumer.override.isolation.level": "read_committed"
   }'
   `
   
   Using: https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-kafka-kafka-sink-connector.html in version 0.7.1
   
   And while sending message (key, payload) onto inbound topic - example.topic - on outbound - example.topic.2 - I receive (null, payload). Is there a way to adjust this configuration to have desire behavior - key is also being past to an endpoint?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-781332678


   For the source side you have to specify this configuration option
   https://github.com/apache/camel-kafka-connector/blob/master/core/src/main/java/org/apache/camel/kafkaconnector/CamelSourceConnectorConfig.java#L70
   
   which is null by default. On the sink side, the key will be in this header https://github.com/apache/camel-kafka-connector/blob/master/core/src/main/java/org/apache/camel/kafkaconnector/CamelSinkTask.java#L45
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j closed issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j closed issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-814100539


   @swierszczu4j I opened this PR on your reproducer
   https://github.com/swierszczu4j/camel-kafka-kafka-connector-problem/pull/1/files
   
   Basically you need to add a "CamelHeader.kafka.KEY" header with the key value while sending message to the inbound topic.
   
   I know it wasn't clear completely, but it's not easy to generalize the Camel Headers usage in the ckc context.
   
   Please let me know if it works for you and sorry for the long delay.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-814876735


   Yes, it should work in that way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-814684878


   @oscerd Thank U very much - indeed it works now! Just out of curiosity, for Source Connector solution would be similar, so to add this config options:
   https://github.com/apache/camel-kafka-connector/blob/master/core/src/main/java/org/apache/camel/kafkaconnector/CamelSourceConnectorConfig.java#L74
   With value "CamelHeader.kafka.KEY", correct?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-809983478


   I'll try to have a look this week. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-809979317


   Hey @oscerd  - have U had time to look into the showcase repo? Do U need additional info?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-788892856


   Hey @swierszczu4j absolutely. It would be a good starting point. Thanks for your time.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-799306463


   Hello - sorry for a delay. Here's a repo with showcase:
   [https://github.com/swierszczu4j/camel-kafka-kafka-connector-problem](https://github.com/swierszczu4j/camel-kafka-kafka-connector-problem)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-816716747


   Hey @oscerd - last thing, as I have tried to make it work with Source connector, yet could not have key on the outbound. The test to reproduce is in the repo: https://github.com/swierszczu4j/camel-kafka-kafka-connector-problem/blob/main/src/test/kotlin/com/example/demo/ShowcaseSourceConnectorTest.kt
   I followed the last comment, but it's not working. Can U help also with this one, please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-788890894


   Hey - thanks for the reply! Much appreciated! As I'm new to the Camel itself I must admit that I do not know how to use properties mentioned by U or what to place as a value for this keys in my connector config so to have rewrite of the kafka message key from one topic to another. All I was able to achieve is to have this key as camel specific header, but message key still is null. Do U think preparing a simple repo usecasing what I want to achieve could be of a help to U in sorting out my issue?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] swierszczu4j commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
swierszczu4j commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-817175446


   @oscerd yes - you are right. I've just simply changed the value in the connector's config from "CamelHeader.kafka.KEY" to "kafka.KEY" and it works & test passes. Thank U very much for all the help! Closing the issue :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-816718879


   I'll have a look. Can you try just with kafka.KEY without CamelHeader? Just a thought.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-kafka-connector] oscerd commented on issue #988: Missing message key on outbound using Camel Kafka KafkaConnector

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #988:
URL: https://github.com/apache/camel-kafka-connector/issues/988#issuecomment-809981008


   Hello @swierszczu4j still didn't find the time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org