You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/06/22 05:38:12 UTC
[GitHub] [camel-kafka-connector] arjun180 opened a new issue #1216: AWSS2-S3-Kafka connector sink not working with Strimzi Kafka cluster
arjun180 opened a new issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216
Hello,
I am trying to get the Camel AWSS2-S3-Kafka connector sink to work with my Kafka cluster. I have a functioning Kafka cluster as well as a Kafka connect cluster up and running. My aim is to move messages from my Kafka topic to the s3 bucket I have created.
My KafkaConnect cluster is fine and has the following pulgins :
```
Status:
Conditions:
Last Transition Time: 2021-06-21T22:51:04.727332Z
Status: True
Type: Ready
Connector Plugins:
Class: org.apache.camel.kafkaconnector.CamelSinkConnector
Type: sink
Version: 0.8.0
Class: org.apache.camel.kafkaconnector.CamelSourceConnector
Type: source
Version: 0.8.0
Class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SinkConnector
Type: sink
Version: 0.8.0
Class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
Type: source
Version: 0.8.0
Class: org.apache.camel.kafkaconnector.timer.CamelTimerSourceConnector
Type: source
Version: 0.8.0
Class: org.apache.kafka.connect.file.FileStreamSinkConnector
Type: sink
Version: 2.8.0
Class: org.apache.kafka.connect.file.FileStreamSourceConnector
Type: source
Version: 2.8.0
Class: org.apache.kafka.connect.mirror.MirrorCheckpointConnector
Type: source
Version: 1
Class: org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
Type: source
Version: 1
Class: org.apache.kafka.connect.mirror.MirrorSourceConnector
Type: source
Version: 1
Label Selector: strimzi.io/cluster=my-connect-cluster,strimzi.io/name=my-connect-cluster-connect,strimzi.io/kind=KafkaConnect
Observed Generation: 1
Replicas: 1
URL: http://my-connect-cluster-connect-api.my-connect-kafka.svc:8083
```
My KafkaConnector looks like the following :
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: s3-sink-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: kafka-connect-topic
camel.sink.path.bucketNameOrArn: my-connect-bucket
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}
camel.component.aws2-s3.accessKey: <access_key>
camel.component.aws2-s3.secretKey: <secret_key>
camel.component.aws2-s3.region: <region>
```
After running the connector above I get the following error :
```
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://my-cluster-connect?keyName=20210622-xxxxxxxx-xxxxxxxxxxxxxxxxxxx-0000000000000000 due to: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=)
```
I imagine the 403 error is because of my s3 bucket not allowing the connector access. Are there some configuration changes I need to be making to the s3 bucket. I did try connecting to my s3 bucket through the AWS CLI and that worked fine. Currently the bucket is set to "Bucket and objects not public"
I'd appreciate any help! Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-866238303
Hi @oscerd - Thanks. So, I was following the thread : https://github.com/apache/camel-kafka-connector/issues/282 here. My access key and secret key were not working (possibly because I use temp credentials which involves the use of secret, access key as well as the session token)
I was trying to use the ```camel.component.aws2-s3.useIAMCredentials: true``` option instead. We made sure the default EC2 node group IAM has full s3 access. The YAML file for the connector looks like this
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: s3-sink-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: kafka-connect-topic
camel.sink.path.bucketNameOrArn: my-connect-bucket
camel.sink.endpoint.keyName: ${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}
camel.component.aws2-s3.useIAMCredentials : true
camel.component.aws2-s3.region: <region>
```
On running the connector, I end up getting this error :
```
Caused by: org.apache.camel.PropertyBindingException: Error binding property (camel.component.aws2-s3.useIAMCredentials=true) with name: useIAMCredentials on bean: org.apache.camel.component.aws2.s3.AWS2S3Component@6c7f1a7 with value: true
```
Any pointers on where we could be going wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-865645612
Thanks @oscerd . I thought the same, but my user has admin access and I can use AWS access key and secret key to upload files to the bucket using the AWS CLI. And I use the same access key and secret key in the Kafka connector yaml file. So, I was wondering if I was missing out on something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-866549716
This got resolved. Thank you for the help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-865645612
Thanks @oscerd . I thought the same, but my user has admin access and I can use AWS access key and secret key to upload files to the bucket using the AWS CLI. And I use the same access key and secret key in the Kafka connector yaml file. So, I was wondering if I was missing out on something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 closed issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 closed issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-865613424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-866241938
To add some more information, my Kafka connect YAML looks like this :
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
namespace : my-cluster
name: my-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true"
spec:
replicas: 1
bootstrapServers: kafka-bootstrap.my-cluster.nvidia.com:9093
tls:
trustedCertificates:
- secretName: my-secret
certificate: server.crt
authentication:
type: tls
certificateAndKey:
secretName: kafka-tls-client-credentials
certificate: user.crt
key: user.key
config:
group.id: my-connect-cluster
offset.storage.topic: my-connect-cluster-offsets
config.storage.topic: mya-connect-cluster-configs
status.storage.topic: my-connect-cluster-status
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: true
value.converter.schemas.enable: true
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
build:
output:
type: docker
image: image:latest
pushSecret: <secret>
plugins:
- name: camel-timer
artifacts:
- type: tgz
url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-timer-kafka-connector/0.8.0/camel-timer-kafka-connector-0.8.0-package.tar.gz
- name: awss2-kafka-connect
artifacts:
- type: tgz
url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-866553470
For completeness the correct property name was useDefaultCredentialsProvider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-865613424
I believe is a problem with permissions on the user role you're using
https://aws.amazon.com/it/premiumsupport/knowledge-center/s3-403-forbidden-error/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-865646618
Without access or information about the environment or role, it's really impossible to say something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1216: AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1216:
URL: https://github.com/apache/camel-kafka-connector/issues/1216#issuecomment-866241938
To add some more information, my Kafka connect YAML looks like this :
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
namespace : my-cluster
name: my-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true"
spec:
replicas: 1
bootstrapServers: kafka-bootstrap.my-cluster.com:9093
tls:
trustedCertificates:
- secretName: my-secret
certificate: server.crt
authentication:
type: tls
certificateAndKey:
secretName: kafka-tls-client-credentials
certificate: user.crt
key: user.key
config:
group.id: my-connect-cluster
offset.storage.topic: my-connect-cluster-offsets
config.storage.topic: mya-connect-cluster-configs
status.storage.topic: my-connect-cluster-status
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: true
value.converter.schemas.enable: true
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
build:
output:
type: docker
image: image:latest
pushSecret: <secret>
plugins:
- name: camel-timer
artifacts:
- type: tgz
url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-timer-kafka-connector/0.8.0/camel-timer-kafka-connector-0.8.0-package.tar.gz
- name: awss2-kafka-connect
artifacts:
- type: tgz
url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.8.0/camel-aws2-s3-kafka-connector-0.8.0-package.tar.gz
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org