You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/07/16 22:46:57 UTC
[GitHub] [camel-kafka-connector] arjun180 opened a new issue #1235: aws2-s3 prefix not working with Kafka source connector
arjun180 opened a new issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235
Hello,
I'm using a Kafka source connector to move publish json files from an s3 bucket to a Kafka topic. When my configuration looks like this - things work fine and I can use a consumer to pull down the messages. In this case, the file 'something.json' is present in the root directory of the bucket
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: s3-source-connector
namespace : my-kafka
labels:
strimzi.io/cluster: my-dev-kafka-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: my-connect-topic
camel.source.path.bucketNameOrArn: my-kakfa-connect
#camel.source.endpoint.useDefaultCredentials Provider: true
camel.component.aws2-s3.useDefaultCredentialsProvider: true
camel.component.aws2-s3.fileName: 'something.json'
camel.source.endpoint.region: us-west-2
```
However, when I use camel.component.aws2-s3.prefix option and create a folder and change my yaml file to
```
piVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: s3-source-connector
namespace : my-kafka
labels:
strimzi.io/cluster: my-dev-kafka-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: my-connect-topic
camel.source.path.bucketNameOrArn: my-kakfa-connect
#camel.source.endpoint.useDefaultCredentials Provider: true
camel.component.aws2-s3.useDefaultCredentialsProvider: true
camel.source.endpoint.region: <region>
camel.component.aws2-s3.prefix: 'some-folder/'
```
I get the message below In the logs and nothing gets published
```
2021-07-16 22:40:09,732 INFO WorkerSourceTask{id=s3-source-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
```
The structure of my bucket is
```
my-bucket
- - some-folder
--- something.json
```
I was wondering if I could help on this end?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-883884388
Thank you very much. I'll move this ticket to done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-883884388
Thank you very much. I'll close out this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
Thanks a lot @oscerd - that worked.
Another quick question - I don't think Kafka Connector works with regexes. Is there way to automate the connector to read from a folder structure in s3 such as this :
`bucket_name/year/day/hour/some.json `where the year/day/hour are subject to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 closed issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 closed issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882492550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
Thanks a lot @oscerd - that worked.
I don't think Kafka Connector works with regexes. Is there way to automate the connector to read from a folder structure in s3 such as this :
`bucket_name/year/day/hour/some.json `where the year/day/hour are subject to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-883069916
Not directly, if you need to do something like that, you may need to look at plain Camel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882492550
Please use:
camel.source.endpoint.prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
Thanks a lot @oscerd - that worked.
I don't think Kafka Connector works with regexes. Is there way to automate the connector to read from a folder structure in s3 such as this :
`bucket_name/year/day/hour/some.json `where the year/day/hour are subject to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 edited a comment on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 edited a comment on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
Thanks a lot @oscerd - that worked.
Another quick question - I don't think Kafka Connector works with regexes. Is there way to automate the connector to read from a folder structure which is dynamic in nature in s3 such as this :
`bucket_name/year/day/hour/some.json `where the year/day/hour are subject to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882492550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] arjun180 commented on issue #1235: aws2-s3 prefix not working with Kafka source connector
Posted by GitBox <gi...@apache.org>.
arjun180 commented on issue #1235:
URL: https://github.com/apache/camel-kafka-connector/issues/1235#issuecomment-882777769
Thanks a lot @oscerd - that worked.
I don't think Kafka Connector works with regexes. Is there way to automate the connector to read from a folder structure in s3 such as this :
`bucket_name/year/day/hour/some.json `where the year/day/hour are subject to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org