You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2020/10/26 11:08:19 UTC
[camel-kafka-connector-examples] 01/01: Telegram Source example:
Added Openshift instructions
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch telegram-openshift
in repository https://gitbox.apache.org/repos/asf/camel-kafka-connector-examples.git
commit 60c6d8cfc284a11100f92848b399c3a54dd60f7d
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Mon Oct 26 12:07:49 2020 +0100
Telegram Source example: Added Openshift instructions
---
telegram/telegram-source/README.adoc | 195 +++++++++++++++++++++
.../config/openshift/telegram-source.yaml | 16 ++
.../config/openshift/telegram-token.properties | 1 +
3 files changed, 212 insertions(+)
diff --git a/telegram/telegram-source/README.adoc b/telegram/telegram-source/README.adoc
index d88880d..971d96b 100644
--- a/telegram/telegram-source/README.adoc
+++ b/telegram/telegram-source/README.adoc
@@ -71,3 +71,198 @@ IncomingMessage{messageId=7, date=2020-10-21T12:13:10Z, from=User{id=217879046,
% Reached end of topic mytopic [0] at offset 1
```
+## Openshift
+
+### What is needed
+
+- A Slack App
+- A Slack channel
+- An Openshift instance
+
+### Running Kafka using Strimzi Operator
+
+First we install the Strimzi operator and use it to deploy the Kafka broker and Kafka Connect into our OpenShift project.
+We need to create security objects as part of installation so it is necessary to switch to admin user.
+If you use Minishift, you can do it with the following command:
+
+[source,bash,options="nowrap"]
+----
+oc login -u system:admin
+----
+
+We will use OpenShift project `myproject`.
+If it doesn't exist yet, you can create it using following command:
+
+[source,bash,options="nowrap"]
+----
+oc new-project myproject
+----
+
+If the project already exists, you can switch to it with:
+
+[source,bash,options="nowrap"]
+----
+oc project myproject
+----
+
+We can now install the Strimzi operator into this project:
+
+[source,bash,options="nowrap",subs="attributes"]
+----
+oc apply -f https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.19.0/strimzi-cluster-operator-0.19.0.yaml
+----
+
+Next we will deploy a Kafka broker cluster and a Kafka Connect cluster and then create a Kafka Connect image with the Debezium connectors installed:
+
+[source,bash,options="nowrap",subs="attributes"]
+----
+# Deploy a single node Kafka broker
+oc apply -f https://github.com/strimzi/strimzi-kafka-operator/raw/0.19.0/examples/kafka/kafka-persistent-single.yaml
+
+# Deploy a single instance of Kafka Connect with no plug-in installed
+oc apply -f https://github.com/strimzi/strimzi-kafka-operator/raw/0.19.0/examples/connect/kafka-connect-s2i-single-node-kafka.yaml
+----
+
+Optionally enable the possibility to instantiate Kafka Connectors through specific custom resource:
+[source,bash,options="nowrap"]
+----
+oc annotate kafkaconnects2is my-connect-cluster strimzi.io/use-connector-resources=true
+----
+
+### Add Camel Kafka connector binaries
+
+Strimzi uses `Source2Image` builds to allow users to add their own connectors to the existing Strimzi Docker images.
+We now need to build the connectors and add them to the image,
+if you have built the whole project (`mvn clean package`) decompress the connectors you need in a folder (i.e. like `my-connectors/`)
+so that each one is in its own subfolder
+(alternatively you can download the latest officially released and packaged connectors from maven):
+
+So we need to do something like this:
+
+```
+> cd my-connectors/
+> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-telegram-kafka-connector/0.5.0/camel-telegram-kafka-connector-0.5.0-package.zip
+> unzip camel-telegram-kafka-connector-0.5.0-package.zip
+```
+
+Now we can start the build
+
+[source,bash,options="nowrap"]
+----
+oc start-build my-connect-cluster-connect --from-dir=./my-connectors/ --follow
+----
+
+We should now wait for the rollout of the new image to finish and the replica set with the new connector to become ready.
+Once it is done, we can check that the connectors are available in our Kafka Connect cluster.
+Strimzi is running Kafka Connect in a distributed mode.
+
+To check the available connector plugins, you can run the following command:
+
+[source,bash,options="nowrap"]
+----
+oc exec -i `oc get pods --field-selector status.phase=Running -l strimzi.io/name=my-connect-cluster-connect -o=jsonpath='{.items[0].metadata.name}'` -- curl -s http://my-connect-cluster-connect-api:8083/connector-plugins
+----
+
+You should see something like this:
+
+[source,json,options="nowrap"]
+----
+[{"class":"org.apache.camel.kafkaconnector.CamelSinkConnector","type":"sink","version":"0.5.0"},{"class":"org.apache.camel.kafkaconnector.CamelSourceConnector","type":"source","version":"0.5.0"},{"class":"org.apache.camel.kafkaconnector.slack.CamelTelegramSinkConnector","type":"sink","version":"0.5.0"},{"class":"org.apache.camel.kafkaconnector.slack.CamelTelegramSourceConnector","type":"source","version":"0.5.0"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"si [...]
+----
+
+### Set the Authorization Token as secret (optional)
+
+You can also set the token option as secret, you'll need to edit the file config/slack-token.properties with the correct credentials and then execute the following command
+
+[source,bash,options="nowrap"]
+----
+oc create secret generic telegram-token --from-file=config/openshift/telegram-token.properties
+----
+
+Now we need to edit KafkaConnectS2I custom resource to reference the secret. For example:
+
+[source,bash,options="nowrap"]
+----
+spec:
+ # ...
+ config:
+ config.providers: file
+ config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
+ #...
+ externalConfiguration:
+ volumes:
+ - name: telegram-token
+ secret:
+ secretName: telegram-token
+----
+
+In this way the secret telegram-token will be mounted as volume with path /opt/kafka/external-configuration/telegram-token/
+
+### Create connector instance
+
+Now we can create some instance of the Telegram source connector:
+
+[source,bash,options="nowrap"]
+----
+oc exec -i `oc get pods --field-selector status.phase=Running -l strimzi.io/name=my-connect-cluster-connect -o=jsonpath='{.items[0].metadata.name}'` -- curl -X POST \
+ -H "Accept:application/json" \
+ -H "Content-Type:application/json" \
+ http://my-connect-cluster-connect-api:8083/connectors -d @- <<'EOF'
+{
+ "name": "telegram-source-connector",
+ "config": {
+ "connector.class": "org.apache.camel.kafkaconnector.telegram.CamelTelegramSourceConnector",
+ "tasks.max": "1",
+ "key.converter": "org.apache.kafka.connect.storage.StringConverter",
+ "value.converter": "org.apache.kafka.connect.storage.StringConverter",
+ "topics": "telegram-topic",
+ "camel.source.endpoint.authorizationToken": "<bot_authorization_token>",
+ "camel.source.path.type": "bots"
+ }
+}
+EOF
+----
+
+Altenatively, if have enabled `use-connector-resources`, you can create the connector instance by creating a specific custom resource:
+
+[source,bash,options="nowrap"]
+----
+oc apply -f - << EOF
+apiVersion: kafka.strimzi.io/v1alpha1
+kind: KafkaConnector
+metadata:
+ name: telegram-source-connector
+ namespace: myproject
+ labels:
+ strimzi.io/cluster: my-connect-cluster
+spec:
+ class: org.apache.camel.kafkaconnector.telegram.CamelTelegramSourceConnector
+ tasksMax: 1
+ config:
+ key.converter: org.apache.kafka.connect.storage.StringConverter
+ value.converter: org.apache.kafka.connect.storage.StringConverter
+ topics: telegram-topic
+ camel.source.endpoint.authorizationToken: <bot_auth_token>
+ camel.source.path.type: bots
+EOF
+----
+
+If you followed the optional step for secret webhook you can run the following command:
+
+[source,bash,options="nowrap"]
+----
+oc apply -f config/openshift/telegram-source.yaml
+----
+
+Send message to your BOT
+
+In another terminal, using kafkacat, you can consume the messages
+
+You can also run the Kafka console consumer to see the messages received from the topic:
+
+[source,bash,options="nowrap"]
+----
+oc exec -i -c kafka my-cluster-kafka-0 -- bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic telegram-topic --from-beginning
+Hello
+----
+
diff --git a/telegram/telegram-source/config/openshift/telegram-source.yaml b/telegram/telegram-source/config/openshift/telegram-source.yaml
new file mode 100644
index 0000000..049bbf7
--- /dev/null
+++ b/telegram/telegram-source/config/openshift/telegram-source.yaml
@@ -0,0 +1,16 @@
+apiVersion: kafka.strimzi.io/v1alpha1
+kind: KafkaConnector
+metadata:
+ name: telegram-source-connector
+ namespace: myproject
+ labels:
+ strimzi.io/cluster: my-connect-cluster
+spec:
+ class: org.apache.camel.kafkaconnector.telegram.CamelSlackTelegramConnector
+ tasksMax: 1
+ config:
+ key.converter: org.apache.kafka.connect.storage.StringConverter
+ value.converter: org.apache.kafka.connect.storage.StringConverter
+ topics: telegram-topic
+ camel.source.path.type: bots
+ camel.source.endpoint.authorizationToken: ${file:/opt/kafka/external-configuration/telegram-token/telegram-token.properties:auth_token}
diff --git a/telegram/telegram-source/config/openshift/telegram-token.properties b/telegram/telegram-source/config/openshift/telegram-token.properties
new file mode 100644
index 0000000..b5f08e8
--- /dev/null
+++ b/telegram/telegram-source/config/openshift/telegram-token.properties
@@ -0,0 +1 @@
+auth_token=xxx