You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/05/25 13:32:25 UTC
[GitHub] [camel-kafka-connector] m4nuC opened a new issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
m4nuC opened a new issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186
Hi,
I am trying to use the kinesis source connector to copy kinesis data to a Kafka stream.
I copy the `camel-aws-kinesis-kafka-connector-0.7.1.jar` but the class does not seem to be available in the Kafka connect.
The documentation mentions adding:
```
<dependency>
<groupId>org.apache.camel.kafkaconnector</groupId>
<artifactId>camel-aws-kinesis-kafka-connector</artifactId>
<version>x.x.x</version>
<!-- use the same version as your Camel Kafka connector version -->
</dependency>
```
but not knowing much java I am unsure what to do with this. Am I missing one step ?
Thanka a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848537302
Same error with 0.10.0. I can create the connector without the transform with the following config:
```
connector.class=org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector
tasks.max=1
topics=_topic
camel.source.endpoint.region=eu-north-1
camel.source.endpoint.accessKey=hhhhhh
camel.source.endpoint.secretKey=hhhhhh
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
camel.source.path.streamName=kinesis_test
```
but if I add the transform it will faill with the following error:
`Invalid value org.apache.camel.kafkaconnector.KinesisRecordDataTransforms for configuration transforms.KinesisRecordDataTransforms.type: Class org.apache.camel.kafkaconnector.KinesisRecordDataTransforms could not be found`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848540963
So instead of
```
transforms=KinesisRecordDataTransforms
transforms.KinesisRecordDataTransforms.type=org.apache.camel.kafkaconnector.KinesisRecordDataTransforms
```
Now it is
```
transforms=KinesisRecordDataTransforms
transforms.KinesisRecordDataTransforms.type=org.apache.camel.kafkaconnector.aws2kinesis.transformers.KinesisRecordDataTransforms
```
Sorry for this, we need to update docs and examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar closed issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
valdar closed issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848531946
0.9.0 when adding the transform errored out saying it could not fine the `org.apache.camel.kafkaconnector.KinesisRecordDataTransforms` class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848549599
Ok sorry, there was a leftover byte encoder in the config. It is now working! Thanks a lot for this support @oscerd!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848652530
Ah, I should totally have found that on my own.. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848547361
Can you post your complete configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848545959
No worries. I can only imagine how much work is going into this. So very grateful for the support you provide here. The transform loads and now getting this error: `org.apache.kafka.connect.errors.DataException: Invalid schema type for ByteArrayConverter: STRING`. :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848538963
I guess the package is different now for the transform. I'll get back to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC edited a comment on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC edited a comment on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848531946
0.9.0 when adding the transform errored out saying it could not find the `org.apache.camel.kafkaconnector.KinesisRecordDataTransforms` class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-847874833
Hello,
Can you please report the steps you're following please?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848514920
@m4nuC you can safely ignore the ftp-extended part (a typo by the way).
The SMT for Kinesis is already included in the connector jar.
https://github.com/apache/camel-kafka-connector/blob/main/connectors/camel-aws2-kinesis-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2kinesis/transformers/KinesisRecordDataTransforms.java
Please use the latest released version 0.10.0, just follow the example, ignoring the maven archetype generation part, just use the connector you downloaded and the configuration provided.
Let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC edited a comment on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC edited a comment on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-847885175
Hi,
Thank you for the fast reply.
The steps are:
1. I download the [connector]( https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.9.0/camel-aws2-kinesis-kafka-connector-0.9.0-package.tar.gz)
2. I unzip it and copy the `camel-aws-kinesis-kafka-connecto`r folder inside kafka connect` class path` using a dockerfile:
`ADD ./services/kafka_connect/camel-aws-kinesis-kafka-connector/ /usr/share/confluent-hub-components/
`
3. I start the application and try to strat a connector with the class: `org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector` and I will get a class not found error
Am I missing anything ?
Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848529571
@oscerd Indeed I have not yet added the transform in the connector config. Let me look up 0.10.0 first then I will report back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC removed a comment on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC removed a comment on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848541754
Thanks, @oscerd. Otherwise, building the transform from the template that you referenced is OK should that be easier but I am unsure where to grab that template from
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848651661
https://github.com/apache/camel-kafka-connector/blob/main/connectors/camel-aws2-kinesis-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2kinesis/transformers/KinesisRecordDataTransforms.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-847885175
Hi,
Thank you for the fast reply.
The steps are:
1. I download the [connector]( https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.9.0/camel-aws2-kinesis-kafka-connector-0.9.0-package.tar.gz)
2. I unzip it and copy the `camel-aws-kinesis-kafka-connecto`r folder inside kafka connect` class path` using a dockerfile
3. I start the application and try to strat a connector with the class: `org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector` and I will get a class not found error
Am I missing anything ?
Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848526418
It has been released you can find it in maven central.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848646683
@oscerd would you have the source code for the `KinesisRecordDataTransforms` handy by any chance? I would like to adapt it to output byte array rather than string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848525474
@oscerd Thanks for the clarification. The installation works but when receiving the first message I get the following error:
`org.apache.kafka.connect.errors.DataException: ByteArrayConverter is not compatible with objects of type class software.amazon.awssdk.services.kinesis.model.Record`
This btw is version 0.9.0, I couldn't find a version 0.10.0 of this connector in the [connector list](https://camel.apache.org/camel-kafka-connector/latest/connectors.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848529995
It should work with 0.9.0 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848527011
You need to specify the transform in the configuration. You are using the byte array converter. can you please post your configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC edited a comment on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC edited a comment on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-847885175
Hi,
Thank you for the fast reply.
The steps are:
1. I download the [connector]( https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-kinesis-kafka-connector/0.9.0/camel-aws2-kinesis-kafka-connector-0.9.0-package.tar.gz)
2. I unzip it and copy the `camel-aws-kinesis-kafka-connecto`r folder inside kafka connect` class path` using a dockerfile:
`ADD ./services/kafka_connect/camel-aws-kinesis-kafka-connector/ /usr/share/confluent-hub-components/camel-aws-kinesis-kafka-connector
`
3. I start the application and try to strat a connector with the class: `org.apache.camel.kafkaconnector.aws2kinesis.CamelAws2kinesisSourceConnector` and I will get a class not found error
Am I missing anything ?
Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848463556
I personally don't have so much experience with confluent hub.
You can have a look at the examples we provide on plain kafka: https://github.com/apache/camel-kafka-connector-examples/tree/main/aws2-kinesis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848770378
If your use case could be a general one, you can even propose a PR and we could add the SMT in the connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848541754
Thanks, @oscerd. Otherwise, building the transform from the template that you referenced is OK should that be easier but I am unsure where to grab that template from
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] m4nuC commented on issue #1186: CAMEL-AWS-KINESIS-KAFKA-CONNECTOR SOURCE CONFIGURATION
Posted by GitBox <gi...@apache.org>.
m4nuC commented on issue #1186:
URL: https://github.com/apache/camel-kafka-connector/issues/1186#issuecomment-848512205
@oscerd thanks, that helps. In that example, do you know what is meant by: `import the ftp-extended project`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org