You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/06/17 23:58:58 UTC
[GitHub] [camel-kafka-connector] saranyaeu2987 opened a new issue #282: Question: Does Camel looks for IAM roles from Env variables
saranyaeu2987 opened a new issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282
This is a followup of https://github.com/apache/camel-kafka-connector/issues/267.
We configured EKS k8s pod with serviceaccount with IAM role for S3 authentication (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) which will create 2 ENV variables inside pod
```
AWS_ROLE_ARN=arn:aws:iam::AWS_ACCOUNT_ID:role/IAM_ROLE_NAME
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
```
Does camel.component.aws-s3.useIAMCredentials: true looks for IAM role arn from those env variable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646874779
Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646253389
@valdar
I am not very successful in resolving access issue.
I tried both **camel-aws2-s3-kafka-connector-0.2.0.jar** and **camel-aws-s3-kafka-connector-0.2.0.jar** both gave me access issue.
Below are list of plugins used.
<img width="1248" alt="Screen Shot 2020-06-18 at 2 07 33 PM" src="https://user-images.githubusercontent.com/19850718/85061979-568f3980-b16d-11ea-8dbc-f93f858c6b89.png">
I am unsure if **camel-aws2-s3-kafka-connector-0.2.0.jar** and **camel-aws-s3-kafka-connector-0.2.0.jar** uses aws-sdk >= 2.10.11
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
I got all jar from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/ If its incorrect can you please give me maven repo link aws2-s3?
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646699151
@oscerd
I get this error now. If my S3Client class and yaml file right ?
**org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified**
Here is the kafkaconnector.yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcde-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.COMPANY.0
#camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.url: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
**camel.sink.endpoint.amazonS3Client:** **org.apache.camel.kafkaconnector.HEBS3Client**
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646050137
@valdar
Thanks for prompt response. I am currently using **camel-aws-s3-kafka-connector-0.2.0.jar**
https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/ has only 0.2.0 version.
Where can I find camel-aws2-2.10.11-kafka-connector jar/binaries ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
Can you please give me maven repo link?
There is are similar component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-645776652
The useiamcredentials option just looks for using the ec2 IAM credentials and use them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
Can you please give me maven repo link?
There is are similar component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in finding dependent jars
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
Can you please give me maven repo link aws2-s3?
There is are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646699151
@oscerd
I get this error now. If my S3Client class and yaml file right ?
**org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified**
Here is the kafkaconnector.yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcde-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.COMPANY.0
camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client:org.apache.camel.kafkaconnector.CustomS3Client
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646336477
> And you don't have to replace any jar in your plugin. Simply download the Aws2 s3 zipped connector and unzip it in your plugin path. Don't mix up dependencies.
Thats what I did in last run.
**Configurations**
1. K8S on EKS cluster. Pod instance config
2. Yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-abcd-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: DB.EMD.COMPANY.0
camel.sink.url: aws2-s3://<bucketname>?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
camel.component.aws2-s3.useIAMCredentials: true
```
3. Env variables property set in pod
```
[kafka@emd-kafka-cluster-connect-5fb8cdf44d-jf6s2 kafka]$ printenv |grep aws
AWS_ROLE_ARN=arn:aws:iam::<arn>:role/emd-kafka-cluster-connect
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
```
4. (K8s Pod Configuration)
Service Account which has access to bucket
```
ML-C02Z605SLVDQ:selumalai-k8s-s3-connect e192270$ aws-vault exec heb-dsol-foundational-projects-bld-staff -- kubectl get serviceaccounts emd-kafka-cluster-connect -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<xxxxxxxxx>:role/emd-kafka-cluster-connect
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::<xxxxxxxx>:role/emd-kafka-cluster-connect"},"creationTimestamp":"2020-06-17T21:48:25Z","labels":{"app.kubernetes.io/instance":"emd-kafka-cluster","app.kubernetes.io/managed-by":"strimzi-cluster-operator","app.kubernetes.io/name":"kafka-connect","app.kubernetes.io/part-of":"strimzi-emd-kafka-cluster","strimzi.io/cluster":"emd-kafka-cluster","strimzi.io/kind":"KafkaConnect","strimzi.io/name":"strimzi"},"name":"emd-kafka-cluster-connect","namespace":"default","resourceVersion":"855936","selfLink":"/api/v1/namespaces/default/serviceaccounts/emd-kafka-cluster-connect","uid":"24bdb5d3-1d43-4bf6-b883-b275b0f5397e"},"secrets":[{"name":"emd-kafka-cluster-connect-token-glprh"}]}
creationTimestamp: "2020-06-17T22:19:54Z"
labels:
app.kubernetes.io/instance: emd-kafka-cluster
app.kubernetes.io/managed-by: strimzi-cluster-operator
app.kubernetes.io/name: kafka-connect
app.kubernetes.io/part-of: strimzi-emd-kafka-cluster
strimzi.io/cluster: emd-kafka-cluster
strimzi.io/kind: KafkaConnect
strimzi.io/name: strimzi
name: emd-kafka-cluster-connect
namespace: default
resourceVersion: "859885"
selfLink: /api/v1/namespaces/default/serviceaccounts/emd-kafka-cluster-connect
uid: 528c4309-4581-4fcb-a66f-f64741299709
secrets:
- name: emd-kafka-cluster-connect-token-glprh
- name: emd-kafka-cluster-connect-token-bvq9n
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646283235
@oscerd
I tried 2 images having camel aws2 jars and both gave the same following error
**Image 1 with following plugins**
**Image 2 with following plugins**
```
Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
at org.apache.camel.main.Main.doInit(Main.java:109)
at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
```
I feel camel-aws2-s3-*.jar is giving me this error. Please guide me which jar Am I missing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646283235
@oscerd @valdar
I tried 2 images having camel aws2 jars and both gave the same following error
**Image 1 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.2.0/ and replaced 2 aws jars ( camel-aws-s3-3.2.0.jar, camel-aws-s3-kafka-connector-0.2.0.jar) with aws2 jar (camel-aws2-s3-3.3.0.jar, camel-aws2-s3-kafka-connector-0.2.0.jar)
**Image 2 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/
```
Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
at org.apache.camel.main.Main.doInit(Main.java:109)
at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
```
I feel camel-aws2-s3-<version>.jar is giving me this error, because if I have camel-aws-s3-<version>.jar doesnt through above error.
Please guide me which jar Am I missing !!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
1. I got all jars from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/camel-aws2-s3-kafka-connector-0.2.0-package.zip If its incorrect can you please give me maven repo link aws2-s3?
> The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
2. Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
3. Below is connector yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-s23456789-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.RETL_UNLD.0
camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
==>camel.component.aws-s3.configuration.autocloseBody: false. <-- Is this the right property name because of this error Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. I get this error only in aws2-s3 ??
camel.component.aws-s3.autoclose-body
camel.component.aws-s3.useIAMCredentials: true
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646462270
@oscerd
1. If I dont provide
` camel.component.aws2-s3.useIAMCredentials: true`
Camel expects static for value for camel.component.aws-s3.accessKey AND camel.component.aws-s3.secretKey right ? What value should I need to provide them?
2. Also, camel.sink.endpoint.amazonS3Client is a reference to a com.amazonaws.services.s3.AmazonS3 in the registry. I didnt understand why we need to set this config and what value goes into it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646320914
@oscerd @valdar
> You have all the information and download links here
> https://camel.apache.org/camel-kafka-connector/latest/connectors.html
Yes, thats where I am getting all jars from
I still getting access denied issues after using aws2-s3. TBH clueless
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] geota commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
geota commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646052472
@valdar is correct. This should be as easy as Camel using the correct SDK version that has support for the AWS_WEB_IDENTITY_TOKEN_FILE env var. This should just be crawled in the DefaultCredentialProvider chain that the SDK executes to find credentials.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646701601
I do have to try. I'll have a look next week, maybe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646693207
@oscerd
Do you mean something like below ? Please let me know if I am wrong
```
public class CustomS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(customS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public CustomS3Client() {
this.s3 = S3Client.builder() .region(region) .credentialsProvider(EnvironmentVariableCredentialsProvider.create()).build();;
}
@Override
public String serviceName() {
return "S3 Client with Env variable Auth";
}
@Override
public void close() {
}
public static void main(String args[]){
CustomS3Client customs3Client = new CustomS3Client();
log.info("customs3Client created!");
}
}
```
and in kafkaconnector.yaml
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-abcd-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: DB.EMD.COMPANY.0
camel.sink.url: aws2-s3://test1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client =#class:org.apache.camel.kafkaconnector.CustomS3Client. <-- class will be added in org.apache.camel.kafkaconnector
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646064968
@oscerd @valdar
Does Camel repo have camel-aws2-0.2.0-kafka-connector.jar ? does it support aws-sdk >= 2.10.11
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
Can you please give me maven repo link aws2-s3?
There is are similar component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-645776652
It's not supported. If you want to use IAM credentials you need to create your own EKS client instance and set it as config option. Useiamcredebtials is not listed in the documentation of this connector
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
1. I got all jars from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/camel-aws2-s3-kafka-connector-0.2.0-package.zip If its incorrect can you please give me maven repo link aws2-s3?
> The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
2. Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
3. Below is connector yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-s23456789-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.RETL_UNLD.0
camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws-s3.configuration.autocloseBody: false. <-- Is this the right property name because of this error Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3??
camel.component.aws-s3.autoclose-body
camel.component.aws-s3.useIAMCredentials: true
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646465575
1. useIamCredentials works by using the underline EC2 credentials used to create the ec2 instance, if you don't use that you need to provide static credentials.
2. You need to create an instance of the client and pass it to the configuration like we did for example here: https://github.com/apache/camel-kafka-connector/blob/master/examples/CamelAmqpSourceConnector.properties#L27 with the #class notation.
I'll try to create an example or something when I'll have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646062881
Anyway, it should possible when instancing your own client. Using the correct version
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646317912
You have all the information and download links here
https://camel.apache.org/camel-kafka-connector/latest/connectors.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646325562
And you don't have to replace any jar in your plugin. Simply download the Aws2 s3 zipped connector and unzip it in your plugin path. Don't mix up dependencies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646283235
@oscerd
I tried 2 images having camel aws2 jars and both gave the same following error
**Image 1 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.2.0/ and replaced 2 aws jars with aws2 jar.
**Image 2 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/
```
Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
at org.apache.camel.main.Main.doInit(Main.java:109)
at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
```
I feel camel-aws2-s3-<version>.jar is giving me this error, because if I have camel-aws-s3-<version>.jar doesnt through above error.
Please guide me which jar Am I missing !!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646699151
@oscerd
I get this error now. If my S3Client class and yaml file right ?
**org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified**
Here is the kafkaconnector.yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcde-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.COMPANY.0
#camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.url: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client:org.apache.camel.kafkaconnector.CustomS3Client
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
valdar commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646040584
@saranyaeu2987 there are more info here: https://github.com/aws/aws-sdk-java-v2/issues/1470 but you need sdk version >= `2.10.11`, so you need to use `camel-aws2-*-kafka-connector` versions...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646064968
@oscerd @valdar
Camel repo has only camel-aws2-0.2.0-kafka-connector.jar , Does it support aws-sdk >= 2.10.11 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646288542
The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646876501
> You need to instantiate the client and from the error it's clear that the client is not in the registry.
> Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.
--> ok, How to check if that my custom S3 client is in registry or not ?
This is how I have added - I have created a separate jar with one class **HEBS3Client** with following definition
```
public class HEBS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(HEBS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public static void main(String args[]){
HEBS3Client hebs3Client = new HEBS3Client();
hebs3Client.s3 = S3Client.builder() .region(hebs3Client.region) .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create()).build();
public String serviceName() {
return "";
}
public void close() {
}
```
with config
`camel.sink.endpoint.amazonS3Client: heb.dsol.fp.HEBS3Client`
and added jar in docker container which have other camel files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646693207
@oscerd
Do you mean something like below ? Please let me know if I am wrong
```
public class CustomS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(customS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public CustomS3Client() {
this.s3 = S3Client.builder() .region(region) .credentialsProvider(EnvironmentVariableCredentialsProvider.create()).build();;
}
@Override
public String serviceName() {
return "S3 Client with Env variable Auth";
}
@Override
public void close() {
}
public static void main(String args[]){
CustomS3Client customs3Client = new CustomS3Client();
log.info("customs3Client created!");
}
}
```
and in kafkaconnector.yaml
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcd-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: DB.EMD.COMPANY.0
camel.sink.url: aws2-s3://test1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client =#class:org.apache.camel.kafkaconnector.CustomS3Client. <-- class will be added in org.apache.camel.kafkaconnector
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646696410
yes, something like that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
I got all jars from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/camel-aws2-s3-kafka-connector-0.2.0-package.zip If its incorrect can you please give me maven repo link aws2-s3?
> The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646275043
Only the Aws2 connectors are based on sdk v2. The version used on 0.2.0 kafka connector of sdk v2 is 2.11.5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646054853
For the env variables for secret and access key, the standard client instance should check them out of the box
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646050137
@valdar
Thanks for prompt response.
https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/ has only 0.2.0 version.
Where can I find camel-aws2-2.10.11-kafka-connector jar/binaries ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646320914
@oscerd @valdar
> You have all the information and download links here
> https://camel.apache.org/camel-kafka-connector/latest/connectors.html
Yes, thats where I am getting all jars from
I still getting access denied issues after using aws2-s3. TBH I am clueless
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646873722
@oscerd @valdar
Is the above error is because `camel-aws2-s3` uses only StaticCredentialsProvider (accesskey and secret) and not WebIdentityTokenFileCredentialsProvider in case of no useIAMCredentials ?
https://github.com/apache/camel/blob/39cbd7f6866cb041d77f6fddc3c7c0721e4d0926/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/client/AWS2S3ClientFactory.java#L40
Is this a bug at camel-aws2-s3 because WebIdentityTokenFileCredentialsProvider is the one which looks for credentials in **Env variable:** **AWS_WEB_IDENTITY_TOKEN_FILE**?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646876501
1.
> You need to instantiate the client and from the error it's clear that the client is not in the registry.
> Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.
--> ok, How to check if that my custom S3 client is in registry or not ?
This is how I have added - I have created a separate jar with one class **HEBS3Client** with following definition
```
public class HEBS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(HEBS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public static void main(String args[]){
HEBS3Client hebs3Client = new HEBS3Client();
hebs3Client.s3 = S3Client.builder() .region(hebs3Client.region) .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create()).build();
public String serviceName() {
return "";
}
public void close() {
}
```
with config
`camel.sink.endpoint.amazonS3Client: heb.dsol.fp.HEBS3Client`
and added jar in docker container which have other camel files.
2. Any other config or setting needed to add S3client class in registry ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
I got all jars from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/camel-aws2-s3-kafka-connector-0.2.0-package If its incorrect can you please give me maven repo link aws2-s3?
> The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646033467
That's related to the underline instance. It's the IAM role with you created the ec2 instance. No the env variables are not taken into account. With the Aws2 components you can inject your own client instance and it will look to env variables if you don't provide a credential provider
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd closed issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd closed issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646874576
It's not a bug, not all the credential provider can be supported in the component out of the box. We have to choose and btw it's something else. You need to instantiate the client and from the error it's clear that the client is not in the registry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646699151
@oscerd
I get this error now. If my S3Client class and yaml file right ?
**org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified**
Here is the kafkaconnector.yaml file
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcde-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.COMPANY.0
#camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.url: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
**camel.sink.endpoint.amazonS3Client:** **org.apache.camel.kafkaconnector.HEBS3Client**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646876501
> You need to instantiate the client and from the error it's clear that the client is not in the registry.
> Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.
--> ok, How to check if that my custom S3 client is in registry or not ?
This is how I have added - I have created a separate jar with one class **HEBS3Client** with following definition
```
public class HEBS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(HEBS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public static void main(String args[]){
HEBS3Client hebs3Client = new HEBS3Client();
hebs3Client.s3 = S3Client.builder() .region(hebs3Client.region) .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create()).build();
public String serviceName() {
return "";
}
public void close() {
}
```
with config
`camel.sink.endpoint.amazonS3Client:org.apache.camel.kafkaconnector.CustomS3Client`
and added jar in docker container which have other camel files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646027770
@oscerd
Where do you look for IAM credentials in a EC2 instance? Are Env variables in credentials provider chain?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646876501
1.
> You need to instantiate the client and from the error it's clear that the client is not in the registry.
> Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.
--> ok, How to check if that my custom S3 client is in registry or not and what is this registry?
This is how I have added - I have created a separate jar with one class **HEBS3Client** with following definition
```
public class HEBS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(HEBS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public static void main(String args[]){
HEBS3Client hebs3Client = new HEBS3Client();
hebs3Client.s3 = S3Client.builder() .region(hebs3Client.region) .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create()).build();
public String serviceName() {
return "";
}
public void close() {
}
```
with config
`camel.sink.endpoint.amazonS3Client: heb.dsol.fp.HEBS3Client`
and added jar in docker container which have other camel files.
2. Any other config or setting needed to add S3client class in registry ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
I got all jar from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/ If its incorrect can you please give me maven repo link aws2-s3?
> so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
```
FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./camel-plugins/ /opt/kafka/plugins/camel-aws-s3/
USER 1001
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646699151
@oscerd
I get this error now. If my S3Client class and yaml file right ?
**org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified**
Here is the kafkaconnector.yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-abcde-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: DB.EMD.COMPANY.0
camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client:org.apache.camel.kafkaconnector.CustomS3Client
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646693207
@oscerd
Do you mean something like below ?
```
public class CustomS3Client implements S3Client{
private static Logger log = LoggerFactory.getLogger(customS3Client.class);
Region region = Region.US_EAST_1;
S3Client s3;
public CustomS3Client() {
this.s3 = S3Client.builder() .region(region) .credentialsProvider(EnvironmentVariableCredentialsProvider.create()).build();;
}
@Override
public String serviceName() {
return "S3 Client with Env variable Auth";
}
@Override
public void close() {
}
public static void main(String args[]){
CustomS3Client customs3Client = new CustomS3Client();
log.info("customs3Client created!");
}
}
```
and in kafkaconnector.yaml
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: heb-s3-abcd-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: DB.EMD.COMPANY.0
camel.sink.url: aws2-s3://test1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.component.aws2-s3.configuration.autocloseBody: false
camel.sink.endpoint.amazonS3Client =#class:org.apache.camel.kafkaconnector.CustomS3Client. <-- class will be added in org.apache.camel.kafkaconnector
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] geota commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
geota commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646058818
@oscerd we are trying to get EKS role based credentials working. To be clear, this is the official AWS supported way of providing role base credentials to Kubernetes pods. Our company does not allow access key/secret keys to avoid credential management overhead (securing, cycling, invalidating etc) and enforces all AWS access to use roles.
AWS added a new way to source credentials in their SDKs to support K8s pod scoped IAM roles. @valdar linked the correct issue discussing this need in the java-sdk. It was fxed via https://github.com/aws/aws-sdk-java-v2/pull/1501 and released as part of AWS Java SDK `2.10.11`
See:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 removed a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 removed a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646290314
@oscerd
1. I got all jars from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/camel-aws2-s3-kafka-connector-0.2.0-package.zip If its incorrect can you please give me maven repo link aws2-s3?
> The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...
There are few camel.component folders in https://repo1.maven.org/maven2/org/apache/camel/ and have hard time in all finding dependent jars
2. Is there a list of dependent jars thats need to be in **/opt/kafka/plugins/camel-aws-s3/** in docker container
3. Below is connector yaml file
```
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-s23456789-connector
labels:
strimzi.io/cluster: emd-kafka-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: HEB.DB.EMD.RETL_UNLD.0
camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
camel.sink.maxPollDuration: 1000
camel.sink.maxBatchPollSize: 10
==>camel.component.aws-s3.configuration.autocloseBody: false. <-- Is this the right property name because of this error Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. I get this error only in aws2-s3 ??
camel.component.aws-s3.autoclose-body
camel.component.aws-s3.useIAMCredentials: true
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646283235
@oscerd
I tried 2 images having camel aws2 jars and both gave the same following error
**Image 1 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.2.0/ and replaced 2 aws jars ( camel-aws-s3-3.2.0.jar, camel-aws-s3-kafka-connector-0.2.0.jar) with aws2 jar (camel-aws2-s3-3.3.0.jar, camel-aws2-s3-kafka-connector-0.2.0.jar)
**Image 2 with following plugins** obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/
```
Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
at org.apache.camel.main.Main.doInit(Main.java:109)
at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
```
I feel camel-aws2-s3-<version>.jar is giving me this error, because if I have camel-aws-s3-<version>.jar doesnt through above error.
Please guide me which jar Am I missing !!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646462270
@oscerd
1. If I dont provide
` camel.component.aws2-s3.useIAMCredentials: true`
Camel expects static for value for camel.component.aws-s3.accessKey AND camel.component.aws-s3.secretKey right ? What value should I need to provide them?
2. Also, camel.sink.endpoint.amazonS3Client is a reference to a com.amazonaws.services.s3.AmazonS3 in the registry.
I didnt understand why we need to set this config ? and what value goes into it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] saranyaeu2987 edited a comment on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
saranyaeu2987 edited a comment on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646064968
@oscerd @valdar
Camel repo has only camel-aws2-0.2.0-kafka-connector.jar ? does it support aws-sdk >= 2.10.11
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646324889
Please report your configuration, what you're setting, how is your client instance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #282: Question: Does Camel looks for IAM roles from Env variables
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #282:
URL: https://github.com/apache/camel-kafka-connector/issues/282#issuecomment-646450704
If you use the useIAMCredential parameter, the client won't check for env variables. The documentation is clear on this: "Set whether the S3 client should expect to load credentials on an EC2 instance or to expect static credentials to be passed in."
You need to provide your own client instance, like explained here: https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html
`
Region region = Region.US_WEST_2;
S3Client s3 = S3Client.builder()
.region(region)
.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.build();
`
And then reference this as camel.sink.endpoint.amazonS3Client
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org