You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2022/12/09 12:07:51 UTC
[GitHub] [camel-kafka-connector] w4rdy opened a new issue, #1487: Consume JSON string for Cassandra sink connector
w4rdy opened a new issue, #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487
Hello,
I'm trying to get the Cassandra sink connector to work with a JSON string produced to a Kafka topic. I've managed to get the connector to add rows to my table when producing the Kafka message:
```
["1","John",1670428382089]
```
But would instead like to produce messages on to the topic in the following format:
```
{
"id" : "1",
"name" : "John",
"created_at" : 1670428382089
}
```
I've attempted to use `value.converter=org.apache.kafka.connect.json.JsonConverter` in my sink properties, but receive the exception: `NoTypeConversionAvailableException: No type converter available to convert from type: java.util.HashMap to the required type: java.io.InputStream`.
Is it possible to achieve this with the Camel Cassandra sink connector?
___
Cassandra table:
```cql
USE connect;
CREATE TABLE person (
id TEXT PRIMARY KEY,
name TEXT,
created_at TIMESTAMP
);
```
Sink properties:
```properties
name=person-cassandra-sink-connector
topics=person
tasks.max=1
connector.class=org.apache.camel.kafkaconnector.cassandrasink.CamelCassandrasinkSinkConnector
value.converter=org.apache.kafka.connect.storage.StringConverter
camel.kamelet.cassandra-sink.connectionHost=cassandra
camel.kamelet.cassandra-sink.connectionPort=9042
camel.kamelet.cassandra-sink.keyspace=connect
camel.kamelet.cassandra-sink.query=insert into person(id, name, created_at) values (?, ?, ?)
camel.kamelet.cassandra-sink.prepareStatements=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] w4rdy commented on issue #1487: Consume JSON string for Cassandra sink connector
Posted by GitBox <gi...@apache.org>.
w4rdy commented on issue #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487#issuecomment-1344377068
Thanks for the quick reply whilst on PTO. Using a custom SMT was going to be my last resort, I was just wondering if this connector implemented something similar to the Datastax connector, which could map JSON fields to Cassandra columns (https://docs.datastax.com/en/kafka/doc/kafka/kafkaStringJson.html) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] w4rdy commented on issue #1487: Consume JSON string for Cassandra sink connector
Posted by "w4rdy (via GitHub)" <gi...@apache.org>.
w4rdy commented on issue #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487#issuecomment-1410051332
We reverted to implementing our own SMT that did the trick in the end, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] w4rdy closed issue #1487: Consume JSON string for Cassandra sink connector
Posted by "w4rdy (via GitHub)" <gi...@apache.org>.
w4rdy closed issue #1487: Consume JSON string for Cassandra sink connector
URL: https://github.com/apache/camel-kafka-connector/issues/1487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1487: Consume JSON string for Cassandra sink connector
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487#issuecomment-1344310282
More specifically: maybe you want to implement a *custom* SMT to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] aozmen121 commented on issue #1487: Consume JSON string for Cassandra sink connector
Posted by GitBox <gi...@apache.org>.
aozmen121 commented on issue #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487#issuecomment-1344233677
We're also getting a similar issue as above. @oscerd & @orpiske Could you please see if we're doing something wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1487: Consume JSON string for Cassandra sink connector
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1487:
URL: https://github.com/apache/camel-kafka-connector/issues/1487#issuecomment-1344307834
> We're also getting a similar issue as above. @oscerd & @orpiske Could you please see if we're doing something wrong?
Folks, I won't be able to look too much into this today as I am in PTO, but ... by the way it sounds, it seems like something you would need to use a [Single Message Transformation (SMT)](https://camel.apache.org/camel-kafka-connector/3.18.x/reference/transformers/index.html) to do so.
Have you tried using that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org