You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2020/06/11 14:06:51 UTC
[camel-kafka-connector-examples] branch Minio created (now 78c7b66)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch Minio
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git.
at 78c7b66 Added a MinIO Example with AWS2 S3 Source connector
This branch includes the following new commits:
new 78c7b66 Added a MinIO Example with AWS2 S3 Source connector
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-kafka-connector-examples] 01/01: Added a MinIO Example with
AWS2 S3 Source connector
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch Minio
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit 78c7b66bbeea1176f334fb15a1beac903e6b9764
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Jun 11 16:06:08 2020 +0200
Added a MinIO Example with AWS2 S3 Source connector
---
aws2-s3/aws2-s3-minio-source/README.md | 84 ++++++++++++++++++++++
.../config/CamelAWS2S3SourceConnector.properties | 33 +++++++++
2 files changed, 117 insertions(+)
diff --git a/aws2-s3/aws2-s3-minio-source/README.md b/aws2-s3/aws2-s3-minio-source/README.md
new file mode 100644
index 0000000..cbf7fe7
--- /dev/null
+++ b/aws2-s3/aws2-s3-minio-source/README.md
@@ -0,0 +1,84 @@
+# Camel-Kafka-connector AWS2 S3 Source
+
+## Introduction
+
+This is an example for Camel-Kafka-connector AW2-S3
+
+## What is needed
+
+- A MinIO instance running
+
+## Running Kafka
+
+```
+$KAFKA_HOME/bin/zookeeper-server-start.sh config/zookeeper.properties
+$KAFKA_HOME/bin/kafka-server-start.sh config/server.properties
+$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
+```
+
+## Setting up the needed bits and running the example
+
+You'll need to setup the plugin.path property in your kafka
+
+Open the `$KAFKA_HOME/config/connect-standalone.properties`
+
+and set the `plugin.path` property to your choosen location
+
+In this example we'll use `/home/oscerd/connectors/`
+
+```
+> cd /home/oscerd/connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/camel-aws2-s3-kafka-connector-0.2.0-package.zip
+> unzip camel-aws2-s3-kafka-connector-0.2.0-package.zip
+```
+
+Now we need to setup a MinIO instance
+
+```
+> wget https://dl.min.io/server/minio/release/linux-amd64/minio
+> chmod +x minio
+> export MINIO_ACCESS_KEY=minio
+> export MINIO_SECRET_KEY=miniostorage
+> mkdir data
+> ./minio server data/
+```
+
+Now it's time to setup the connectors
+
+Open the AWS2 S3 configuration file for MinIO
+
+```
+name=CamelAWS2S3SourceConnector
+connector.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+camel.source.maxPollDuration=10000
+
+topics=test1
+
+camel.source.path.bucketNameOrArn=bucket
+camel.source.endpoint.overrideEndpoint=true
+camel.source.endpoint.uriEndpointOverride=http://localhost:9000
+camel.component.aws2-s3.access-key=minio
+camel.component.aws2-s3.secret-key=miniostorage
+camel.component.aws2-s3.region=eu-west-1
+```
+
+and add the correct credentials for AWS.
+
+Now you can run the example
+
+```
+$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties config/CamelAWSS3SourceConnector.properties config/CamelAWS2SQSSourceConnector.properties
+```
+
+Just connect to your MinIO console and upload a file into your bucket.
+
+On a different terminal run the kafka-consumer and you should see messages from the MinIO S3 bucket arriving through Kafka Broker.
+
+```
+bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytopic --from-beginning
+S3 to Kafka through Camel
+```
+
diff --git a/aws2-s3/aws2-s3-minio-source/config/CamelAWS2S3SourceConnector.properties b/aws2-s3/aws2-s3-minio-source/config/CamelAWS2S3SourceConnector.properties
new file mode 100644
index 0000000..1d0826e
--- /dev/null
+++ b/aws2-s3/aws2-s3-minio-source/config/CamelAWS2S3SourceConnector.properties
@@ -0,0 +1,33 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name=CamelAWS2S3SourceConnector
+connector.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+camel.source.maxPollDuration=10000
+
+topics=test1
+
+camel.source.path.bucketNameOrArn=bucket
+camel.source.endpoint.overrideEndpoint=true
+camel.source.endpoint.uriEndpointOverride=http://localhost:9000
+camel.component.aws2-s3.access-key=minio
+camel.component.aws2-s3.secret-key=miniostorage
+camel.component.aws2-s3.region=eu-west-1
+