You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "thuyhoang-gp (via GitHub)" <gi...@apache.org> on 2023/07/25 10:38:45 UTC
[GitHub] [camel-kafka-connector] thuyhoang-gp opened a new issue, #1545: Problems with CamelDDBStreamSourceConnector 3.20.3
thuyhoang-gp opened a new issue, #1545:
URL: https://github.com/apache/camel-kafka-connector/issues/1545
Hi,
I am struggling to use CamelDDBStreamSourceConnector for DynamoDB change capture. Could you please advice me how to get over them?
1. I have tried several common converters for Kafka Connect, but only one of them is 'usable'
- `io.confluent.connect.avro.AvroConverter` ❌
- `org.apache.kafka.connect.json.JsonConverter` ❌
- `org.apache.kafka.connect.converters.ByteArrayConverter` ❌
- `org.apache.kafka.connect.storage.StringConverter` ✅
The value of message in case of `StringConverter`, however, is address of `InputStreamCache`:
<img width="1174" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/122586888/14b11ac0-91c9-436c-895f-0d2753695193">
Could you provide me **a name of converter** I can use with this `value.converter`?
2. The topic.key is currently blank. Could you clone it from payload of DDB Stream, for example: `ID`.
<img width="1194" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/122586888/fee52f63-25e5-4761-b879-81b8ebf1d2ea">
Again, thanks for the work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] thuyhoang-gp commented on issue #1545: Problems with CamelDDBStreamSourceConnector 3.20.3
Posted by "thuyhoang-gp (via GitHub)" <gi...@apache.org>.
thuyhoang-gp commented on issue #1545:
URL: https://github.com/apache/camel-kafka-connector/issues/1545#issuecomment-1649596032
Here is my source connector configuration:
```
{
"connector.class": "org.apache.camel.kafkaconnector.awsddbstreamssource.CamelAwsddbstreamssourceSourceConnector",
"camel.kamelet.aws-ddb-streams-source.accessKey": "XXX",
"camel.kamelet.aws-ddb-streams-source.secretKey": "XXX",
"tasks.max": "1",
"camel.kamelet.aws-ddb-streams-source.uriEndpointOverride": "",
"camel.kamelet.aws-ddb-streams-source.useDefaultCredentialsProvider": "false",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"session.timeout.ms": "60000",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"camel.kamelet.aws-ddb-streams-source.region": "XXX",
"camel.kamelet.aws-ddb-streams-source.delay": "5000",
"topics": "ddb-test",
"database.history.kafka.bootstrap.servers": "kafka-headless.kafka-banzai:29092",
"camel.kamelet.aws-ddb-streams-source.streamIteratorType": "FROM_START",
"camel.kamelet.aws-ddb-streams-source.table": "ecgps_paymenttags",
"name": "ddb-consumer",
"camel.kamelet.aws-ddb-streams-source.overrideEndpoint": "false",
"snapshot.mode": "initial"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] jakubmalek commented on issue #1545: Problems with CamelDDBStreamSourceConnector 3.20.3
Posted by "jakubmalek (via GitHub)" <gi...@apache.org>.
jakubmalek commented on issue #1545:
URL: https://github.com/apache/camel-kafka-connector/issues/1545#issuecomment-1657835145
It looks like you have same problem as in here
https://github.com/apache/camel-kafka-connector/issues/1543
The source connector is returning value as `InputStream` java-type, which cannot be serialized with default converters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1545: Problems with CamelDDBStreamSourceConnector 3.20.3
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #1545:
URL: https://github.com/apache/camel-kafka-connector/issues/1545#issuecomment-1657836730
We're looking into the InputStream behavior. @valdar is looking at that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org