You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2021/03/02 12:09:39 UTC
[camel-kafka-connector-examples] branch google-pubsub-sink created
(now 0e21f9b)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch google-pubsub-sink
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git.
at 0e21f9b Added Google Pubsub sink connector example
This branch includes the following new commits:
new 957250e Added Google Pubsub sink connector example
new 0e21f9b Added Google Pubsub sink connector example
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-kafka-connector-examples] 02/02: Added Google Pubsub sink
connector example
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch google-pubsub-sink
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit 0e21f9b99d67b0ad8b8fe4eb92caa8a9f07719b4
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Tue Mar 2 13:09:07 2021 +0100
Added Google Pubsub sink connector example
---
...roperties => CamelGooglepubsubSinkConnector.properties} | 14 +++++++-------
.../config/CamelGooglepubsubSourceConnector.properties | 2 +-
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
similarity index 76%
copy from google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
copy to google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
index a040c4a..8fbdd0b 100644
--- a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties
@@ -15,14 +15,14 @@
# limitations under the License.
#
-name=CamelGoogle-pubsubSourceConnector
-connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+name=CamelGoogle-pubsubSinkConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
-value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
-topics=mytopic
+topics=pubsub-topic
-camel.source.path.destinationName=test-sub
-camel.source.path.projectId=testpubsub-306306
-camel.source.endpoint.synchronousPull=true
+camel.sink.path.destinationName=test
+camel.sink.path.projectId=testpubsub-306306
+camel.sink.endpoint.synchronousPull=true
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
index a040c4a..2e7a8d2 100644
--- a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
@@ -21,7 +21,7 @@ tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
-topics=mytopic
+topics=mytopic-source
camel.source.path.destinationName=test-sub
camel.source.path.projectId=testpubsub-306306
[camel-kafka-connector-examples] 01/02: Added Google Pubsub sink
connector example
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch google-pubsub-sink
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit 957250e7920f6da479877d1d98cb2693a662929c
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Tue Mar 2 13:06:33 2021 +0100
Added Google Pubsub sink connector example
---
google-pubsub/google-pubsub-sink/README.adoc | 132 +++++++++++++++++++++
.../CamelGooglepubsubSourceConnector.properties | 28 +++++
2 files changed, 160 insertions(+)
diff --git a/google-pubsub/google-pubsub-sink/README.adoc b/google-pubsub/google-pubsub-sink/README.adoc
new file mode 100644
index 0000000..4281d16
--- /dev/null
+++ b/google-pubsub/google-pubsub-sink/README.adoc
@@ -0,0 +1,132 @@
+= Camel-Kafka-connector Google Pubsub Sink Example
+
+This is an example for Camel-Kafka-connector Google PubSub Sink Source Example
+
+== Standalone
+
+=== What is needed
+
+- A google cloud account
+- A service account for PubSub service
+- A project in google cloud
+- A PubSub topic
+- A PubSub topic subscription
+
+=== Setting up PubSub
+
+You need to go here: https://console.cloud.google.com/cloudpubsub
+- Create a service account with permission to operate on PubSub service
+- Export a service account key file somewhere on your filesystem (for example /home/connectors/ServiceAccountPubsub.json)
+- Follow the instruction to create a topic (call it 'test' for example)
+- Follow the instruction to create a subscription (call it 'test-sub' for example)
+
+Take note of project id and subscription name. If for example your topic name is 'projects/testpubsub-306306/topics/test'
+the project id will be 'testpubsub-306306'.
+
+=== Running Kafka
+
+[source]
+----
+$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
+$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
+$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
+----
+
+=== Download the connector package
+
+Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
+
+[source]
+----
+> cd /home/oscerd/connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.8.0/camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
+> untar.gz camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
+----
+
+=== Configuring Kafka Connect
+
+You'll need to set up the `plugin.path` property in your kafka
+
+Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:
+
+[source]
+----
+...
+plugin.path=/home/oscerd/connectors
+...
+----
+
+=== Setup the connectors
+
+Open the Google Pubsub Sink connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties`
+
+[source]
+----
+name=CamelGoogle-pubsubSinkConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.sink.path.destinationName=test
+camel.sink.path.projectId=testpubsub-306306
+camel.sink.endpoint.synchronousPull=true
+----
+
+For this example we'll use a Google PubSub Source connector too to get the messages:
+
+Open the Google Pubsub source connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-source/config/CamelGooglepubsubSourceConnector.properties`
+
+[source]
+----
+name=CamelGoogle-pubsubSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+
+topics=mytopic-source
+
+camel.source.path.destinationName=test-sub
+camel.source.path.projectId=testpubsub-306306
+camel.source.endpoint.synchronousPull=true
+----
+
+
+=== Running the example
+
+On the same terminal you'll run the connector run the following command:
+
+[source]
+----
+export GOOGLE_APPLICATION_CREDENTIALS=/home/connectors/ServiceAccountPubsub.json
+----
+
+pointing to the service account key you created.
+
+Run the kafka connect with the Google Pubsub Source connector:
+
+[source]
+----
+$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
+----
+
+In a terminal with kafkacat run the following command:
+[source]
+----
+> echo "Camel Kafka Connector is great!" | ./kafkacat -b localhost:9092 -t mytopic
+% Reached end of topic mytopic [0] at offset 1
+----
+
+On a different terminal run the kafkacat consumer
+
+[source]
+----
+> ./kafkacat -b localhost:9092 -t mytopic-source
+Camel Kafka Connector is great!
+% Reached end of topic mytopic [0] at offset 1
+----
+
+The message in the test topic has been consumed by the source connector now.
diff --git a/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
new file mode 100644
index 0000000..a040c4a
--- /dev/null
+++ b/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name=CamelGoogle-pubsubSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
+
+topics=mytopic
+
+camel.source.path.destinationName=test-sub
+camel.source.path.projectId=testpubsub-306306
+camel.source.endpoint.synchronousPull=true