You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2020/11/12 17:36:33 UTC
[camel-kafka-connector-examples] branch google-sheets-with-split
created (now a56adc9)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch google-sheets-with-split
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git.
at a56adc9 Added Google Sheets Streams Source Example with split
This branch includes the following new commits:
new a56adc9 Added Google Sheets Streams Source Example with split
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-kafka-connector-examples] 01/01: Added Google Sheets Streams
Source Example with split
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch google-sheets-with-split
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit a56adc93e59a1ea12c3f01e5a1c7b54ffffa3f33
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Nov 12 18:36:05 2020 +0100
Added Google Sheets Streams Source Example with split
---
.../README.adoc | 114 +++++++++++++++++++++
...melGoogleSheetsStreamSourceConnector.properties | 35 +++++++
2 files changed, 149 insertions(+)
diff --git a/google-sheets-stream/google-sheets-stream-with-split-source/README.adoc b/google-sheets-stream/google-sheets-stream-with-split-source/README.adoc
new file mode 100644
index 0000000..cbcfa16
--- /dev/null
+++ b/google-sheets-stream/google-sheets-stream-with-split-source/README.adoc
@@ -0,0 +1,114 @@
+= Camel-Kafka-connector Sheets Stream Source with Split Example
+
+This is an example for Camel-Kafka-connector Sheets Stream Source with Split Example
+
+== Standalone
+
+=== What is needed
+
+- A google account
+- A google application authorized to deal with Sheets API
+
+=== Setting up Google Application
+
+The step-by-step guide is a bit long, so follow it with attention:
+
+1. Create a project
+2. Note down clientSecret and clientId
+2. Create an OAuth consent screen
+3. Create OAuth Client credentials (OAuth for WebApp)
+4. Add https://developers.google.com/oauthplayground to Authorized redirect URIs
+5. Setup https://developers.google.com/oauthplayground using credentials from above
+6. Authorize APIs for Sheets API v4 scope https://www.googleapis.com/auth/spreadsheets
+7. Exchange authorization code for tokens
+8. Note down Access token & Refresh token
+9. Create a google sheets document and add two column in A1 and B1, named username and city
+10. In A2 and B2, add "John" and "New York"
+11. In A3 and B3, add "Andrea" and "Roma"
+12. In A4 and B4, add "Andrea" and "Milan"
+13. In A5 and B5, add "Jack" and "London"
+14. Note down the sheets document Id
+
+=== Running Kafka
+
+[source]
+----
+$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
+$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
+$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
+----
+
+=== Download the connector package
+
+Download the connector package zip and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`
+
+[source]
+----
+> cd /home/oscerd/connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-sheets-stream-kafka-connector/0.6.0/camel-google-sheets-stream-kafka-connector-0.6.0-package.zip
+> unzip camel-google-sheets-stream-kafka-connector-0.6.0-package.zip
+----
+
+=== Configuring Kafka Connect
+
+You'll need to set up the `plugin.path` property in your kafka
+
+Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:
+
+[source]
+----
+...
+plugin.path=/home/oscerd/connectors
+...
+----
+
+=== Setup the connectors
+
+Open the Git configuration file at `$EXAMPLES/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties`
+
+[source]
+----
+name=CamelGoogle-sheets-streamSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlesheetsstream.CamelGooglesheetsstreamSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.source.path.apiName=data
+camel.source.endpoint.clientId=<client_id>
+camel.source.endpoint.spreadsheetId=<spreadsheet_id>
+camel.source.endpoint.accessToken=<access_token>
+camel.source.endpoint.refreshToken=<refresh_token>
+camel.source.endpoint.clientSecret=<client_secret>
+camel.source.endpoint.applicationName=ckc
+camel.source.endpoint.delay=60000
+camel.component.google-sheets-stream.range=A1:B5
+camel.source.endpoint.splitResults=true
+----
+
+=== Running the example
+
+Run the kafka connect with the Google Sheets Stream Source connector:
+
+[source]
+----
+$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-sheets-stream/google-sheets-stream-source/config/CamelGoogleSheetsStreamSourceConnector.properties
+----
+
+On a different terminal run the kafkacat consumer
+
+[source]
+----
+> ./kafkacat -b localhost:9092 -t db903 -f 'Headers: %h: Message value: %s\n'
+% Auto-selecting Consumer mode (use -P or -C to override)
+% Reached end of topic db903 [0] at offset 0
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B5,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelHeader.CamelGoogleSheetsValueIndex=1,CamelProperty.CamelBatchSize=5,CamelProperty.CamelBatchComplete=false,CamelProperty.CamelBatchIndex=0,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&poll [...]
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B5,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelHeader.CamelGoogleSheetsValueIndex=2,CamelProperty.CamelBatchSize=5,CamelProperty.CamelBatchComplete=false,CamelProperty.CamelBatchIndex=1,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&poll [...]
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B5,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelHeader.CamelGoogleSheetsValueIndex=3,CamelProperty.CamelBatchSize=5,CamelProperty.CamelBatchComplete=false,CamelProperty.CamelBatchIndex=2,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&poll [...]
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B5,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelHeader.CamelGoogleSheetsValueIndex=4,CamelProperty.CamelBatchSize=5,CamelProperty.CamelBatchComplete=false,CamelProperty.CamelBatchIndex=3,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&poll [...]
+Headers: CamelHeader.CamelGoogleSheetsMajorDimension=ROWS,CamelHeader.CamelGoogleSheetsRange=Foglio1!A1:B5,CamelHeader.CamelGoogleSheetsRangeIndex=1,CamelHeader.CamelGoogleSheetsSpreadsheetId=1rkX3YNc0IEbIouNsQEzOVu0s5QmeyOlyiqjwQFK_hmI,CamelHeader.CamelGoogleSheetsValueIndex=5,CamelProperty.CamelBatchSize=5,CamelProperty.CamelBatchComplete=true,CamelProperty.CamelBatchIndex=4,CamelProperty.CamelToEndpoint=direct://end?pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true&polli [...]
+% Reached end of topic db903 [0] at offset 5
+
+----
diff --git a/google-sheets-stream/google-sheets-stream-with-split-source/config/CamelGoogleSheetsStreamSourceConnector.properties b/google-sheets-stream/google-sheets-stream-with-split-source/config/CamelGoogleSheetsStreamSourceConnector.properties
new file mode 100644
index 0000000..82fb374
--- /dev/null
+++ b/google-sheets-stream/google-sheets-stream-with-split-source/config/CamelGoogleSheetsStreamSourceConnector.properties
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name=CamelGoogle-sheets-streamSourceConnector
+connector.class=org.apache.camel.kafkaconnector.googlesheetsstream.CamelGooglesheetsstreamSourceConnector
+tasks.max=1
+key.converter=org.apache.kafka.connect.storage.StringConverter
+value.converter=org.apache.kafka.connect.storage.StringConverter
+
+topics=mytopic
+
+camel.source.path.apiName=data
+camel.source.endpoint.clientId=<client_id>
+camel.source.endpoint.spreadsheetId=<spreadsheet_id>
+camel.source.endpoint.accessToken=<access_token>
+camel.source.endpoint.refreshToken=<refresh_token>
+camel.source.endpoint.clientSecret=<client_secret>
+camel.source.endpoint.applicationName=ckc
+camel.source.endpoint.delay=60000
+camel.component.google-sheets-stream.range=A1:B5
+camel.source.endpoint.splitResults=true