[PLC4X] [Kafka Connect] Help with Kafka Connect configuration?

[Christofer Dutz <ch...@c-ware.de>]

Hi,

we are currently working on improving the PLC4X Kafka Connect plugin.
While the current version offered a pretty simple configuration, this is not quite suitable for production scenarios.

In contrast to normal Kafka Connect sources/sinks the PLC4X source and sink should distribute the load that a connection to an industrial controller is only initiated by one Kafka connect node at a time
(The reason for this is, that industrial controllers are only able to accept a very limited number of connections)
So the Kafka Connect system should distribute these sources to it’s nodes.

For each source we also are able to run multiple jobs which collect given sets of parameters in individual intervals and push the data to individual Kafka topics.

We came up with the following configuration:

name=plc-source-test
connector.class=org.apache.plc4x.kafka.Plc4xSourceConnector

defaults.topic=some/default

sources.machineA.connectionString=s7://1.2.3.4/1/1
sources.machineA.jobReferences.s7-dashboard.enabled=true
sources.machineA.jobReferences.s7-heartbeat.enabled=true
sources.machineA.jobReferences.s7-heartbeat.topic=heartbeat

sources.machineB.connectionString=s7://10.20.30.40/1/1
sources.machineB.topic=heartbeat
sources.machineB.jobReferences.s7-heartbeat.enabled=true

sources.machineC.connectionString=ads://1.2.3.4.5.6
sources.machineC.topic=heartbeat
sources.machineC.jobReferences.ads-heartbeat.enabled=true

jobs.s7-dashboard.interval=500
jobs.s7-dashboard.fields.inputPreasure=%DB.DB1.4:INT
jobs.s7-dashboard.fields.outputPreasure=%Q1:BYT
jobs.s7-dashboard.fields.temperature=%I3:INT

jobs.s7-heartbeat.interval=1000
jobs.s7-heartbeat.fields.active=%I0.2:BOOL

jobs.ads-heartbeat.interval=1000
jobs.ads-heartbeat.fields.active=Main.running

So we define the individual sources which map to PLC connections and each connection references a set of jobs.
“Source” and “Job” are PLC4X terms, so if they collide with Kafka Connect terms, please don’t be confused.

We are able to successfully process this into a working configuration.
The problem is, that we would like to certify the driver and therefore are required to use the ConfigDef objects to describe the configuration structure.
What would be the best way to do this? Or is our approach completely wrong? As far as I can see it, I can only configure my Kafka Connect plugins via property structures (Even if they’re transferred via JAVA in the distributed mode)

We would greatly appreciate some help here.

Chris