You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "jakubmalek (via GitHub)" <gi...@apache.org> on 2023/12/18 15:58:31 UTC
[I] Implicit unmarshalling in Cassandra Sink Connector Kamelet descriptor [camel-kafka-connector]
jakubmalek opened a new issue, #1579:
URL: https://github.com/apache/camel-kafka-connector/issues/1579
The Cassandra Sink Connector [Kamelet](https://github.com/apache/camel-kamelets/blob/main/kamelets/cassandra-sink.kamelet.yaml) descriptor is configured by default with the Jackson unmarshalling step:
```yaml
spec:
...
template:
from:
uri: "kamelet:source"
steps:
- unmarshal:
json:
library: Jackson
useList: true
```
The implicit unmarshalling is problematic as it doesn't really fit to the Kafka Connect (converter/transforms) data flow in my opinion, as it requires additional serialization/deserialization steps for the Apache Camel pipeline.
In the case of the Cassandra connector specifically, the problem is more apparent as the JSON array format is not usable for the CQL statement builder.
The problem with the JSON format is the data-type ambiguity, for example in the Cassandra you might have int64 column but in the JSON it's just a number that can deserialized either to int32 or int64, depending on the value. There are more problems for example how to pass the timestamp value in the JSON in a reliable way.
Normally, in the Kafka Connector, you would use the transformation plugins to cast the types into a specific format, or use the Struct data type, instead of schema-less JSON.
But because of the unmarshalling to Jackson step, the connect record value, even with the right types must stil be marshalled into JSON again, which leads to the type ambiguity described above.
IMHO the default Kamelet definitions for all connectors should be stripped out with any implicit marshalling or unmarshalling steps.
Alternatively there should be a way to remove this step with the means of connector configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Implicit unmarshalling in Cassandra Sink Connector Kamelet descriptor [camel-kafka-connector]
Posted by "jakubmalek (via GitHub)" <gi...@apache.org>.
jakubmalek commented on issue #1579:
URL: https://github.com/apache/camel-kafka-connector/issues/1579#issuecomment-1860928502
It would be a breaking change indeed, so I understand the hesitation.
For more intermediate solution, I would at least appreciate the option to remove the default configuration for marshalling with configuration, for example something like: `camel.sink.unmarshal: none`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Implicit unmarshalling in Cassandra Sink Connector Kamelet descriptor [camel-kafka-connector]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1579:
URL: https://github.com/apache/camel-kafka-connector/issues/1579#issuecomment-1860899965
The main problem is just that Kamelets are not exclusively for Kafka Connect Runtime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Implicit unmarshalling in Cassandra Sink Connector Kamelet descriptor [camel-kafka-connector]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1579:
URL: https://github.com/apache/camel-kafka-connector/issues/1579#issuecomment-1860902410
I can think about change it, but I don't think it's so easy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org