You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2021/03/02 12:10:32 UTC
[camel-kafka-connector-examples] branch master updated (c88d25a ->
a6963cc)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git.
from c88d25a Added Google PubSub to the examples list
new 17f83bc Added Google Pubsub sink connector example
new a6963cc Added Google Pubsub sink connector example
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../README.adoc | 41 ++++++++++++++++++----
.../CamelGooglepubsubSinkConnector.properties | 12 +++----
.../CamelGooglepubsubSourceConnector.properties | 2 +-
3 files changed, 41 insertions(+), 14 deletions(-)
copy google-pubsub/{google-pubsub-source => google-pubsub-sink}/README.adoc (65%)
copy cron/cron-source/config/CamelCronSourceConnector.properties => google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties (77%)
copy google-pubsub/{google-pubsub-source => google-pubsub-sink}/config/CamelGooglepubsubSourceConnector.properties (98%)
[camel-kafka-connector-examples] 01/02: Added Google Pubsub sink
connector example
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit 17f83bc3f3956ee1d853bfe47e89ab67376d9b65
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Tue Mar 2 13:06:33 2021 +0100
Added Google Pubsub sink connector example
---
google-pubsub/google-pubsub-sink/README.adoc | 132 +++++++++++++++++++++
.../CamelGooglepubsubSourceConnector.properties | 28 +++++
2 files changed, 160 insertions(+)
diff --git a/google-pubsub/google-pubsub-sink/README.adoc b/google-pubsub/google-pubsub-sink/README.adoc
new file mode 100644
index 0000000..4281d16
--- /dev/null
+++ b/google-pubsub/google-pubsub-sink/README.adoc
@@ -0,0 +1,132 @@
+= Camel-Kafka-connector Google Pubsub Sink Example
+
+This is an example for Camel-Kafka-connector Google PubSub Sink Source Example
+
+== Standalone
+
+=== What is needed
+
+- A google cloud account
+- A service account for PubSub service
+- A project in google cloud
+- A PubSub topic
+- A PubSub topic subscription
+
+=== Setting up PubSub
+
+You need to go here: https://console.cloud.google.com/cloudpubsub
+- Create a service account with permission to operate on PubSub service
+- Export a service account key file somewhere on your filesystem (for example /home/connectors/ServiceAccountPubsub.json)
+- Follow the instruction to create a topic (call it 'test' for example)
+- Follow the instruction to create a subscription (call it 'test-sub' for example)
+
+Take note of project id and subscription name. If for example your topic name is 'projects/testpubsub-306306/topics/test'
+the project id will be 'testpubsub-306306'.
+
+=== Running Kafka
+
+[source]
+----
+$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
+$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
+$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
+----
+
+=== Download the connector package
+
+Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
+
+[source]
+----
+> cd /home/oscerd/connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.8.0/camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
+> untar.gz camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
+----
+
+=== Configuring Kafka Connect
+
+You'll need to set up the `plugin.path` property in your kafka
+
+Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:
+
+[source]
+----
+...
+plugin.path=/home/oscerd/connectors
+...
+----
+
+=== Setup the connectors
+
+Open the Google Pubsub Sink connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties`
+
+[source]
+----
+name=CamelGoogle-pubsubSinkConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.sink.path.destinationName=test
+camel.sink.path.projectId=testpubsub-306306
+camel.sink.endpoint.synchronousPull=true
+----
+
+For this example we'll use a Google PubSub Source connector too to get the messages:
+
+Open the Google Pubsub source connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-source/config/CamelGooglepubsubSourceConnector.properties`
+
+[source]
+----
+name=CamelGoogle-pubsubSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+
+topics=mytopic-source
+
+camel.source.path.destinationName=test-sub
+camel.source.path.projectId=testpubsub-306306
+camel.source.endpoint.synchronousPull=true
+----
+
+
+=== Running the example
+
+On the same terminal you'll run the connector run the following command:
+
+[source]
+----
+export GOOGLE_APPLICATION_CREDENTIALS=/home/connectors/ServiceAccountPubsub.json
+----
+
+pointing to the service account key you created.
+
+Run the kafka connect with the Google Pubsub Source connector:
+
+[source]
+----
+$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+----
+
+In a terminal with kafkacat run the following command:
+[source]
+----
+> echo "Camel Kafka Connector is great!" | ./kafkacat -b localhost:9092 -t mytopic
+% Reached end of topic mytopic [0] at offset 1
+----
+
+On a different terminal run the kafkacat consumer
+
+[source]
+----
+> ./kafkacat -b localhost:9092 -t mytopic-source
+Camel Kafka Connector is great!
+% Reached end of topic mytopic [0] at offset 1
+----
+
+The message in the test topic has been consumed by the source connector now.
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
new file mode 100644
index 0000000..a040c4a
--- /dev/null
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name=CamelGoogle-pubsubSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+
+topics=mytopic
+
+camel.source.path.destinationName=test-sub
+camel.source.path.projectId=testpubsub-306306
+camel.source.endpoint.synchronousPull=true
[camel-kafka-connector-examples] 02/02: Added Google Pubsub sink
connector example
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit a6963cc914fb988c6e8330f2e9a0e8383f694e6e
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Tue Mar 2 13:09:07 2021 +0100
Added Google Pubsub sink connector example
---
...roperties => CamelGooglepubsubSinkConnector.properties} | 14 +++++++-------
.../config/CamelGooglepubsubSourceConnector.properties | 2 +-
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
similarity index 76%
copy from google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
copy to google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
index a040c4a..8fbdd0b 100644
--- a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
@@ -15,14 +15,14 @@
# limitations under the License.
#
-name=CamelGoogle-pubsubSourceConnector
-connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+name=CamelGoogle-pubsubSinkConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
-value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
-topics=mytopic
+topics=pubsub-topic
-camel.source.path.destinationName=test-sub
-camel.source.path.projectId=testpubsub-306306
-camel.source.endpoint.synchronousPull=true
+camel.sink.path.destinationName=test
+camel.sink.path.projectId=testpubsub-306306
+camel.sink.endpoint.synchronousPull=true
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
index a040c4a..2e7a8d2 100644
--- a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
@@ -21,7 +21,7 @@ tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
-topics=mytopic
+topics=mytopic-source
camel.source.path.destinationName=test-sub
camel.source.path.projectId=testpubsub-306306