You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/12/02 01:23:40 UTC
[GitHub] [camel-kafka-connector] papillon88 opened a new issue #1293: Fails to create CamelAws2sqsSourceConnector
papillon88 opened a new issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293
I use below docker compose to spin up kafka cluster -
```
version: '2'
services:
# this is our kafka cluster.
kafka-cluster:
image: landoop/fast-data-dev:2.4.1
environment:
ADV_HOST: 127.0.0.1 # Change to 192.168.99.100 if using Docker Toolbox
RUNTESTS: 0 # Disable Running tests so the cluster starts faster
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
volumes:
- "//c/intellij_projects/kafka/kafka-connectors/source/kafka-connect-github-source:/connectors/kafka-connect-github-source"
- "//c/intellij_projects/kafka/kafka-connectors/source/camel-aws2-sqs-kafka-connector:/connectors/camel-aws2-sqs-kafka-connector"
- "//c/intellij_projects/kafka/kafka-connectors/source/camel-aws-sqs-kafka-connector:/connectors/camel-aws-sqs-kafka-connector"
```
I use (download) these connectors to volume mount them into the abv containers as shown in above docker compose -
```
https://maven.repository.redhat.com/ga/org/apache/camel/kafkaconnector/camel-aws2-sqs-kafka-connector/0.7.1.fuse-800004-redhat-00001/
https://maven.repository.redhat.com/ga/org/apache/camel/kafkaconnector/camel-aws-sqs-kafka-connector/0.7.1.fuse-800004-redhat-00001/
```
I then hit http://127.0.0.1:3030/kafka-connect-ui to create the connector in UI with this config -
```
name=CamelAws2-sqsSourceConnector
connector.class=org.apache.camel.kafkaconnector.aws2sqs.CamelAws2sqsSourceConnector
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
camel.source.maxPollDuration=10000
topics=sqs-1
camel.source.path.queueNameOrArn=camel-1
camel.source.endpoint.deleteAfterRead=false
camel.component.aws2-sqs.access-key=##########################
camel.component.aws2-sqs.secret-key=##########################
camel.component.aws2-sqs.region=us-east-1
topic.creation.enable=false
topic.creation.groups=inventory,sales
topic.creation.default.replication.factor=3
```
The connector creation fails with this error -
```
org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context
at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:144)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:198)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Option queueNameOrArn is required when creating endpoint uri with syntax aws2-sqs:queueNameOrArn
at org.apache.camel.support.component.EndpointUriFactorySupport.buildPathParameter(EndpointUriFactorySupport.java:53)
at org.apache.camel.component.aws2.sqs.Sqs2EndpointUriFactory.buildUri(Sqs2EndpointUriFactory.java:104)
at org.apache.camel.kafkaconnector.utils.TaskHelper.buildUrl(TaskHelper.java:68)
at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:111)
... 8 more
```
Screenshot -
Im already supplying the "queueNameOrArn" in the config as shown abv.
Not sure how to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293#issuecomment-984574771
1. The difference is the under the hood SDK version, for aws2-sqs is the SDK v2 and for aws-sqs is the v1. The suggestion is to switch to v2, because in the new ckc release you won't find aws-sqs, because the related camel component has been removed starting from camel 3.9.0, this means from ckc >= 0.9.0 you won't have aws-sqs connector
2. The camel.source.component options are meant to be used to set something at Camel component level, this means, in Camel ecosystem, the endpoints created from the component will have the same options set at component level (you set them one and they will be in all the endpoints). camel.source.endpoint are fine grained option for a single endpoint.
It's always better in the context of ckc to set everything at endpoint options level, instead of component.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] papillon88 closed issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
papillon88 closed issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] papillon88 commented on issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
papillon88 commented on issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293#issuecomment-984570796
Thanks. will give it a try.
Also ,
1. what is the diff between camel-aws2-sqs-kafka-connector and camel-aws-sqs-kafka-connector and which one to use when ?
2. diff between camel.source.component option and camel.source.endpoint option and which one to use when ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293#issuecomment-984335569
Hello,
This is the upstream repository for camel-kafka-connector, if you have trouble with this you need to contact Red Hat. My suggestion is to use the latest version released upstream and check with it.
Also please don't mix camel.source.component option and camel.source.endpoint option.
Move the accessKey, secretKey and region to camel.source.endpoint options and try again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] papillon88 commented on issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
papillon88 commented on issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293#issuecomment-985068794
thanks. the connector at https://camel.apache.org/camel-kafka-connector/0.11.0/reference/connectors/camel-aws2-sqs-kafka-source-connector.html works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] papillon88 edited a comment on issue #1293: Fails to create CamelAws2sqsSourceConnector
Posted by GitBox <gi...@apache.org>.
papillon88 edited a comment on issue #1293:
URL: https://github.com/apache/camel-kafka-connector/issues/1293#issuecomment-984570796
Thanks @oscerd . will give it a try.
Also ,
1. what is the diff between camel-aws2-sqs-kafka-connector and camel-aws-sqs-kafka-connector and which one to use when ?
2. diff between camel.source.component option and camel.source.endpoint option and which one to use when ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org