[camel-kafka-connector-examples] branch sheets-stream created (now 8b85e93)

[ac...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

acosentino pushed a change to branch sheets-stream
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git.


      at 8b85e93  Added Google Sheets Stream Source connector example

This branch includes the following new commits:

     new 8b85e93  Added Google Sheets Stream Source connector example

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[camel-kafka-connector-examples] 01/01: Added Google Sheets Stream Source connector example

[ac...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

acosentino pushed a commit to branch sheets-stream
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git

commit 8b85e9343c0fb4666ee25012dda1899fdbb4b7b1
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Nov 12 08:15:26 2020 +0100

    Added Google Sheets Stream Source connector example
---
 .../google-sheets-stream-source/README.adoc        | 104 +++++++++++++++++++++
 ...melGoogleSheetsStreamSourceConnector.properties |  33 +++++++
 2 files changed, 137 insertions(+)

diff --git a/google-sheets-stream/google-sheets-stream-source/README.adoc b/google-sheets-stream/google-sheets-stream-source/README.adoc
new file mode 100644
index 0000000..d71bfb2
--- /dev/null
+++ b/google-sheets-stream/google-sheets-stream-source/README.adoc
@@ -0,0 +1,104 @@
+= Camel-Kafka-connector Calendar Stream Source Example
+
+This is an example for Camel-Kafka-connector Calendar Stream Source Example
+
+== Standalone
+
+=== What is needed
+
+- A google account
+- A google application authorized to deal with calendar API
+
+=== Setting up Google Application
+
+The step-by-step guide is a bit long, so follow it with attention:
+
+1. Create a project
+2. Note down clientSecret and clientId
+2. Create an OAuth consent screen
+3. Create OAuth Client credentials (OAuth for WebApp)
+4. Add https://developers.google.com/oauthplayground to Authorized redirect URIs
+5. Setup https://developers.google.com/oauthplayground using credentials from above
+6. Authorize APIs for Sheets API v4 scope https://www.googleapis.com/auth/spreadsheets
+7. Exchange authorization code for tokens
+8. Note down Access token & Refresh token
+9. Create a google sheets document and add two column in A1 and B1, named username and city
+10. In A2 and B2, add "John" and "New York"
+10. Note down the sheets document Id
+
+=== Running Kafka
+
+[source]
+----
+$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
+$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
+$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
+----
+
+=== Download the connector package
+
+Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
+
+[source]
+----
+> cd /home/oscerd/connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-sheets-stream-kafka-connector/0.6.0/camel-google-sheets-stream-kafka-connector-0.6.0-package.zip
+> unzip camel-google-sheets-stream-kafka-connector-0.6.0-package.zip
+----
+
+=== Configuring Kafka Connect
+
+You'll need to set up the `plugin.path` property in your kafka
+
+Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:
+
+[source]
+----
+...
+plugin.path=/home/oscerd/connectors
+...
+----
+
+=== Setup the connectors
+
+Open the Git configuration file at `$EXAMPLES/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties`
+
+[source]
+----
+name=CamelGoogle-sheets-streamSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlesheetsstream.CamelGooglesheetsstreamSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.source.path.apiName=data
+camel.source.endpoint.clientId=<client_id>
+camel.source.endpoint.spreadsheetId=<spreadsheet_id>
+camel.source.endpoint.accessToken=<access_token>
+camel.source.endpoint.refreshToken=<refresh_token>
+camel.source.endpoint.clientSecret=<client_secret>
+camel.source.endpoint.applicationName=ckc
+camel.source.endpoint.delay=60000
+camel.component.google-sheets-stream.range=A1:B2
+----
+
+=== Running the example
+
+Run the kafka connect with the Google Sheets Stream Source connector:
+
+[source]
+----
+$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties
+----
+
+On a different terminal run the kafkacat consumer
+
+[source]
+----
+> ./kafkacat -b localhost:9092 -t db899 -f 'Headers: %h: Message value: %s\n'
+% Auto-selecting Consumer mode (use -P or -C to override)
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B2,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelProperty.CamelBatchSize=1,CamelProperty.CamelBatchComplete=true,CamelProperty.CamelBatchIndex=0,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&pollingConsumerQueueSize=1000: Message value: { [...]
+% Reached end of topic db899 [0] at offset 1
+----
diff --git a/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties b/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties
new file mode 100644
index 0000000..6c6b4a5
--- /dev/null
+++ b/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties
@@ -0,0 +1,33 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name=CamelGoogle-sheets-streamSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlesheetsstream.CamelGooglesheetsstreamSourceConnector
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.source.path.apiName=data
+camel.source.endpoint.clientId=513252201330-ddsi05kcp44b1fgt2tdffleurpkrb2jt.apps.googleusercontent.com
+camel.source.endpoint.accessToken=ya29.A0AfH6SMCA3VfeHn-zMabAvPQmKvakDf8Gw3O8MOXP7WYfo7Ki8nWg3-AJHJA1tA6_ww6eeoBTwjf8wuumNEm0WEfshsmgLudkkegdgYF99ME2H5toVUl-XnqivY6iHaafCACc54v9vha9vQdu_BtU9GXo9miyGkTjp2mCilKA98E
+camel.source.endpoint.refreshToken=1//04q_xA-KU1yrSCgYIARAAGAQSNwF-L9Ir27zVsuKZ5GEhXDD3YEFXT80Vnd6z8X3NDQckQX8yUb2ujHcCIkmQh3_hKjISix4k1Po
+camel.source.endpoint.clientSecret=8_R8cCH0tGlDlv39gcASagSp
+camel.source.endpoint.spreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI
+camel.source.endpoint.applicationName=ckc127
+camel.source.endpoint.delay=60000
+camel.component.google-sheets-stream.range=A1:B2