You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Christofer Dutz <ch...@c-ware.de> on 2021/06/30 17:53:20 UTC
[DISCUSS] Discover and Browse API for PLC4J?
Hi all,
as you all know, I have threatened to bring API additions from plc4go
back to plc4j and now the time has come. But I want to involve you all
in it as we all have to live with it ;-)
So in general in go I did it the following way:
- I split the topic up into "Discovery" and "Browse"
"Browse" I'll focus on as soon ad "Discovery" is defined. In general
this should produce a stream of PlcBrowseEvents which provide a stream
of PlcField entries that can be used to query information. So this is
inside an established connection to a device. It won't find devices
(However this term is slightly ambiguous because in KNX for example we
consider a KNX devices GroupAddress an address, even if in the physical
world it actually is a device)
Discovery is the operation of finding out which devices we can connect
to. The result is a list of PlcDiscoveryEvents. These contain:
- protocolCode (like s7, modbus, ...)
- transportCode (like tcp, udp, serial, ...)
- transportUrl (which is the part after the "://" but before the "?"
- option (which is a map of key-value pairs making up the part after the
"?" .... the part which Lukazs hates so much ;-) )
In go I extended the DriverManager to have a discovery function which
simply iterates over all drivers it knows and if they support discovery,
the functionality is executed.
In the past we usually returned a Future and this became complete as
soon as the operation was finished. However Discovery can sometimes take
quite long. Some protocols support a direct "Please all devices
supporting this report to me" functionality, but some we will need to
potentially probe by trying all IP addresses in a given range. Therefore
I think such a future approach is sub-ideal.
In go I gave the discover function a callback argument where you set a
handler that gets notified directly as soon as a device is found. I am a
bit unsure if this should be the way to go.
What I would love to do, would be to have the option to add one
(possibly multiple) handlers to a DiscoveryRequest. Whenever something
is found, then every handler is called and at the end the result is put
into a list. As soon as the operation is complete, the user could then
use this list, just like our normal read-results and be happy with it,
or he could use the callbacks if he wants to be informed directly.
Unfortunately the Futures are so totally all-or-nothing. I would love to
return something that knows the progress of the operation. In a
Broadcast search it could simply be the progress toward the timeout or
when scanning an IP range it could be the percentage of the addresses
that were probed. This could help tools to show a nice progress on the
auto-discovery.
What do you think? As I need this for my work on the PROFINET, I'm
working on this on the feature/profinet-chris branch.
Chris
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Ben,
I guess as a first way, using such a discovery server sounds like a good
plan.
In the future I even imagine extending the query options we use so you
could for example provide an IP range and a set of protocols to check
and then the system probes every IP in that range for connections.
But that's future stuff ... now I think that this discovery server is
exactly what we need for OPC-UA.
And I agree, we can't provide user-names and passwords ... however it
would definitely be fun, if we could ;)
in this case I guess we can output {username} and {password} or alike in
the connection string. If we follow a standard format here, tools like
Streampipes can probably process it.
Regarding filtering: I agree. It would be cool to limit which devices
discovery is run on how many vlan hops should be allowed and in case of
active ip scans, we definitely need to limit the ip ranges and port
ranges. But I think we'll concentrate on the immediate tasks of finding
stuff and deal with filtering afterwards.
Chris
On 04.07.21 13:02, Ben Hutcheson wrote:
> Hi,
>
> Just commenting on the OPCUA side of things,
> Your description of discovery matches what is used for OPCUA, it is
> described here.
> https://reference.opcfoundation.org/v104/Core/docs/Part4/5.4.1/
> I would expect that we pass a url for the discovery server and a list of
> endpoints is returned using the format you described.
>
> Also as we use certificates/usernames/passwords that are defined in the
> connection string, we probably can't assume that we are going to be able to
> return a valid connection string for each endpoint as this information
> would also need to be included as parameters.
>
> For browsing, it is similar and is described here,
> https://reference.opcfoundation.org/v104/Core/docs/Part4/5.8.2/
> One concern I have is we probaby should have someway of limiting/filtering
> the fields returned, as having a large number of fields is kind of expected.
>
> Kind Regards
>
> Ben
>
>
>
> On Fri, Jul 2, 2021 at 3:02 PM Otto Fowler <ot...@gmail.com> wrote:
>
>> Enip and Profinet Discovery and configuration protocol may ‘switch’ after
>> discovery.
>>
>> Like, you send ENIP Identify on udp, and then connect on tcp to do other
>> commands I believe.
>> Also, you may use DCP to discover things, and then use s7 to talk to them (
>> the DCP identity response is over link layer, but has ip address as a field
>> ).
>>
>>
>>
>> From: Łukasz Dywicki <lu...@code-house.org> <lu...@code-house.org>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Date: July 2, 2021 at 13:17:14
>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: Re: [DISCUSS] Discover and Browse API for PLC4J?
>>
>> Hey Chris,
>> Indeed - discovery is about finding interfaces/devices while browsing is
>> already interacting with them. I believe that in most of cases once
>> discovery is done browsing uses same transport as PLC connection itself.
>> At least this would be logical consequence. If someone knows other case
>> then it is right time to speak! :-)
>>
>> The AMS discovery answer returns you AMS net id of a node, so you know
>> IP of PLC and also its logical network identifier. As you pointed
>> usually it is just an <ip>.1.1. I actually use this convention when
>> making discovery caller ams. It is just my-ip + 1.1.
>> What we miss is a ams port (by default there is 851, but there are
>> others and it can vary). The user and password is needed only for route
>> setup, if you have route set up on PLC then this step is not necessary.
>> Anyhow, I think we need to have a look on OPC UA discovery (which is a
>> separate sub-spec) to see if we're set on discovery inputs. Given its
>> verbosity I do expect additional set of requirements coming from that end.
>>
>> For KNX I see indeed a similarity between CANopen (and profinet) how
>> node discovery could work. All these are "bus systems" thus connection
>> is not a point to point. In their case we see multiple participants
>> which are listening to a single communication channel. As soon as we
>> acquire link we can discover and browse through it.
>>
>> A short note. I am not sure if we could automate plain CAN discovery,
>> cause then we have 11 bit identifier which can be anything. Role of
>> these 11 bits is known only if we assume CANopen.
>> In worst case basic CAN could lead to us to discovery of up to 4095 CAN
>> nodes. ;-)
>>
>> Anyhow, I believe what you have sketched is pretty clear.
>>
>> Best,
>> Łukasz
>>
>>
>> On 02.07.2021 18:23, Christofer Dutz wrote:
>>> Hi Lukasz,
>>> br/>> so from what I read I sort of think you also aggree that
>> "Discovery"
>>> should find devices and the result is connection information we can then
>>> pass to the DriverManager to get a connection. And Browse will result
>>> with fields (Or parsed address strings) for a given connection.
>>> br/>> I think in the ADS Case ... we shouldn't do thee setting up of ADS
>> Routes
>>> as it involves sending a servers Admin password in clear text. However I
>>> think we are able to find an ADS device. As with ADS you also have to
>>> provide the source and targer ams net id, it will be challenging to
>>> guess these (Or does the discovery handle that?). However we could
>>> assume the default of "appending ".1.1" to the local IP which seems to
>>> be quite established. If however the discovery allows the listing up of
>>> AMSNetIds a device running on a given IP has, this would be awesome.
>>> br/>> I think KNX and CAN can be considered a bit simmilar regarding
>> what's a
>>> device and what's a ressouce. For KNX we currently consider the KNXNetIP
>>> Gateway the device (and therefore the connection string only has the
>>> coordinates to connect to that) and then the KNX Devices on the KNXBus
>>> are considered Resources which are found by a Browse operation.
>>> When thinking of KNX I would consider the Device resources sort of like
>>> with S7 ... the KNX Device on the KNX Bus is the DataBlock and the
>>> resource on the device is the address relative to that DataBlock. So if
>>> I want to read a parameter of a KNX device, then I add the parmeter
>>> coordinates to the knx address which makes up the field address in PLC4X.
>>> br/>> It looks like it should be possible to bring thhe CAN driver into
>>> supporting the same mechanisms here too.
>>> br/>> Chris <
>>> br/>> On 02.07.21 17:53, &##321;ukasz Dywicki wrote:
>>>> Hey Chris,
>>>> I do agree that browse and discovery is different and it might be either
>>>> expanded or narrowed.
>>>>
>>>> For example openHAB stuff I did based on CANopen driver does following:
>>>> - Discovery of CAN enabled interface
>>>> For linux it is quite straight cause you can use either a "net" part of
>>>> sysfs or even a dbus [1] in order to find interesting network nodes.
>>>> - Discvoery of CANopen nodes
>>>> Depending on strategy this could be a scan with predefined address range
>>>> or simply listen who talks to the bus. For CANopen scan is quite
>>>> straight, cause there is up to 127 nodes. You can submit a read out
>>>> request for well known register address, ie. manufacturer code.
>>>> Listening is also possible cause each node should (in theory) emit a
>>>> heartbeat.
>>>> - Discovery of CANopen fields readable or writable via driver.
>>>> This is scan of node registers within predefined address ranges or
>>>> ranges retrieved from initial node read out.
>>>> I don't have this part (nor didn't plan to work on it anytime soon).
>>>> This is probably a thing which is most interesting in case of data
>>>> acquisition.
>>>>
>>>> Process is following:
>>>> - [discovery] Physical interface [0..N]
>>>> - [discovery] Discovery connection string [1:1] (?)
>>>> - [discovery/browse] Finding accessible nodes [0..N]
>>>> - [browse] Fields for a specific node [0..N]
>>>>
>>>> Discovery of Beckhoff stuff I did works in a following way:
>>>> - Find network interfaces
>>>> - Send broadcast over network interface
>>>> - For each answered node
>>>> - Try setting up a ADS route (requires PLC username and password)
>>>> - Connect
>>>> - Retrieve symbols (I did not implement this)
>>>> Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
>>>> you can clearly see that there are two different transports used. I
>>>> believe KNX is same.
>>>> For Profinet-DCP what I managed to complete with Adrian is focused only
>>>> on two parts:
>>>> - Finding accessible profinet devices
>>>> Since they do answer with their own datas we can get their MAC which is
>>>> later on used to communicate with them. Initial mapping of DCP ident
>>>> response I did mapped that response to PlcStruct. Mainly because there
>>>> are multiple fields which can come, including device name, DHCP settings
>>>> or its ip address.
>>>>
>>>> Given the nature of node discovery it should by default stick with
>>>> callbacks and not request/response modes. Browse is something we briefly
>>>> spooked before, but it also fails into same category. We never know if
>>>> discovery or browse is finite or not. For that reason API should rather
>>>> assume later than earlier and let driver decide.
>>>> For address range queries we indeed can have a progress indication,
>>>> while for PN-DCP I don't think we should make any assumptions. I am not
>>>> quire sure how big PN network can be, so I'll let others speak.
>>>>
>>>> In my understanding, also based on above openHAB experiences, discovery
>>>> and browse APIs are kind of related, yet have different syntax.
>>>> For Beckhoff definitely we get some parts for transportUrl options such
>>>> as node AMS id which is advertised in identification response.
>>>> For AMS there is also an authentication, however I don't think we're
>>>> able to scrap this. In new versions I think there might be also a device
>>>> certificate signature.
>>>>
>>>> Anyhow, these would be my 0.02€ of input.
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>> On 02.07.2021 17:02, Otto Fowler wrote:
>>>>> Some protocols the discovery process returns information about the
>> device
>>>>> (DCP, ENIP) , that might be important as well. How would this be
>> handled /
>>>>> modeled ?
>>>>>
>>>>> The idea here is that plc4x doesn’t store any of this right? You’d have
>> to
>>>>> handle the callback and store the information yourself?
>>>>>
>>>>> From: Christofer Dutz <ch...@c-ware.de>
>>>>> <ch...@c-ware.de>
>>>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
>> dev@plc4x.apache.org>
>>
>>>>> Date: June 30, 2021 at 13:53:32
>>>>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>>>> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>>>>>
>>>>> Hi all,
>>>>>
>>>>> as you all know, I have threatened to bring API additions from plc4go
>>>>> br/>back to plc4j and now the time has come. But I want to invollve you
>> all
>>>>> br/>in it as we all have to live with it ;-) <
>>>>>
>>>>> So in general in go I did it the following way:
>>>>>
>>>>> - I split the topic up into "Discovery" and "Browse"
>>>>>
>>>>> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
>>>>> br/>this should produce a stream of PlcBrowseEvents whichh provide a
>> stream
>>>>> br/>of PlcFField entries that can be used to query information. So this
>> is
>>>>> br/>inside an established connection to a device. It won't find devices
>>>>> br/>(However this term is slightly ambbiguous because in KNX for
>> example we
>>>>> br/>consider a KNX devices GrouupAddress an address, even if in the
>>>>> physical br/>world it actually iis a device)
>>>>>
>>>>> Discovery is the operation of finding out which devices we can connect
>>>>> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
>>>>> - protocolCode (like s7, modbus, ...)
>>>>> - transportCode (like tcp, udp, serial, ...)
>>>>> - transportUrl (which is the part after the "://" but before the "?"
>>>>> - option (which is a map of key-value pairs making up the part after
>> the
>>>>> br/>""?" .... the part which Lukazs hates so much ;-) )
>>>>>
>>>>> In go I extended the DriverManager to have a discovery function which
>>>>> br/>simply iterates over all drivers it knows and if they supporrt
>>>>> discovery, br/>the functionality is executed. <
>>>>>
>>>>> In the past we usually returned a Future and this became complete as
>>>>> br/>soon as the operation was finished. However Discovery can
>> ssometimes
>>>>> take br/>quite long. Some protocols support a direct ""Please all
>> devices
>>>>> br/>supporting this report to me"" functionality, but some we will need
>> to
>>>>> br/>potentially probe by trying all IP addreesses in a given range.
>>>>> Therefore br/>I think such a future approach is sub-ideal.
>>>>>
>>>>> In go I gave the discover function a callback argument where you set a
>>>>> br/>handler that gets notified directly as soon as a device is found. I
>> am
>>>>> a br/>bit unsure if this should be the way to go. <
>>>>>
>>>>> What I would love to do, would be to have the option to add one
>>>>> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
>>>>> br/>is found, then every handler is called and at the end the resuult
>> is
>>>>> put br/>into a list. As soon as the operation is complete, thee user
>> could
>>>>> then br/>use this list, just like our normal read-resultts and be happy
>>>>> with it, br/>or he could use the callbacks if he wantts to be informed
>>>>> directly.
>>>>>
>>>>> Unfortunately the Futures are so totally all-or-nothing. I would love
>> to
>>>>> br/>return something that knows the progress of the operatiion. In a
>>>>> br/>Broadcast search it could simply be the progress towardd the
>> timeout or
>>>>> br/>when scanning an IP range it could be the percenntage of the
>> addresses
>>>>> br/>that were probed. This could help tools too show a nice progress on
>> the
>>>>> br/>auto-discovery. <
>>>>>
>>>>> What do you think? As I need this for my work on the PROFINET, I'm
>>>>> br/>working on this on the feature/profinet-chris branch. <
>>>>>
>>>>> Chris
>>>>>
>>
>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Ben Hutcheson <be...@gmail.com>.
Hi,
Just commenting on the OPCUA side of things,
Your description of discovery matches what is used for OPCUA, it is
described here.
https://reference.opcfoundation.org/v104/Core/docs/Part4/5.4.1/
I would expect that we pass a url for the discovery server and a list of
endpoints is returned using the format you described.
Also as we use certificates/usernames/passwords that are defined in the
connection string, we probably can't assume that we are going to be able to
return a valid connection string for each endpoint as this information
would also need to be included as parameters.
For browsing, it is similar and is described here,
https://reference.opcfoundation.org/v104/Core/docs/Part4/5.8.2/
One concern I have is we probaby should have someway of limiting/filtering
the fields returned, as having a large number of fields is kind of expected.
Kind Regards
Ben
On Fri, Jul 2, 2021 at 3:02 PM Otto Fowler <ot...@gmail.com> wrote:
> Enip and Profinet Discovery and configuration protocol may ‘switch’ after
> discovery.
>
> Like, you send ENIP Identify on udp, and then connect on tcp to do other
> commands I believe.
> Also, you may use DCP to discover things, and then use s7 to talk to them (
> the DCP identity response is over link layer, but has ip address as a field
> ).
>
>
>
> From: Łukasz Dywicki <lu...@code-house.org> <lu...@code-house.org>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: July 2, 2021 at 13:17:14
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: [DISCUSS] Discover and Browse API for PLC4J?
>
> Hey Chris,
> Indeed - discovery is about finding interfaces/devices while browsing is
> already interacting with them. I believe that in most of cases once
> discovery is done browsing uses same transport as PLC connection itself.
> At least this would be logical consequence. If someone knows other case
> then it is right time to speak! :-)
>
> The AMS discovery answer returns you AMS net id of a node, so you know
> IP of PLC and also its logical network identifier. As you pointed
> usually it is just an <ip>.1.1. I actually use this convention when
> making discovery caller ams. It is just my-ip + 1.1.
> What we miss is a ams port (by default there is 851, but there are
> others and it can vary). The user and password is needed only for route
> setup, if you have route set up on PLC then this step is not necessary.
> Anyhow, I think we need to have a look on OPC UA discovery (which is a
> separate sub-spec) to see if we're set on discovery inputs. Given its
> verbosity I do expect additional set of requirements coming from that end.
>
> For KNX I see indeed a similarity between CANopen (and profinet) how
> node discovery could work. All these are "bus systems" thus connection
> is not a point to point. In their case we see multiple participants
> which are listening to a single communication channel. As soon as we
> acquire link we can discover and browse through it.
>
> A short note. I am not sure if we could automate plain CAN discovery,
> cause then we have 11 bit identifier which can be anything. Role of
> these 11 bits is known only if we assume CANopen.
> In worst case basic CAN could lead to us to discovery of up to 4095 CAN
> nodes. ;-)
>
> Anyhow, I believe what you have sketched is pretty clear.
>
> Best,
> Łukasz
>
>
> On 02.07.2021 18:23, Christofer Dutz wrote:
> > Hi Lukasz,
> > br/>> so from what I read I sort of think you also aggree that
> "Discovery"
> > should find devices and the result is connection information we can then
> > pass to the DriverManager to get a connection. And Browse will result
> > with fields (Or parsed address strings) for a given connection.
> > br/>> I think in the ADS Case ... we shouldn't do thee setting up of ADS
> Routes
> > as it involves sending a servers Admin password in clear text. However I
> > think we are able to find an ADS device. As with ADS you also have to
> > provide the source and targer ams net id, it will be challenging to
> > guess these (Or does the discovery handle that?). However we could
> > assume the default of "appending ".1.1" to the local IP which seems to
> > be quite established. If however the discovery allows the listing up of
> > AMSNetIds a device running on a given IP has, this would be awesome.
> > br/>> I think KNX and CAN can be considered a bit simmilar regarding
> what's a
> > device and what's a ressouce. For KNX we currently consider the KNXNetIP
> > Gateway the device (and therefore the connection string only has the
> > coordinates to connect to that) and then the KNX Devices on the KNXBus
> > are considered Resources which are found by a Browse operation.
> > When thinking of KNX I would consider the Device resources sort of like
> > with S7 ... the KNX Device on the KNX Bus is the DataBlock and the
> > resource on the device is the address relative to that DataBlock. So if
> > I want to read a parameter of a KNX device, then I add the parmeter
> > coordinates to the knx address which makes up the field address in PLC4X.
> > br/>> It looks like it should be possible to bring thhe CAN driver into
> > supporting the same mechanisms here too.
> > br/>> Chris <
> > br/>> On 02.07.21 17:53, &##321;ukasz Dywicki wrote:
> >> Hey Chris,
> >> I do agree that browse and discovery is different and it might be either
> >> expanded or narrowed.
> >>
> >> For example openHAB stuff I did based on CANopen driver does following:
> >> - Discovery of CAN enabled interface
> >> For linux it is quite straight cause you can use either a "net" part of
> >> sysfs or even a dbus [1] in order to find interesting network nodes.
> >> - Discvoery of CANopen nodes
> >> Depending on strategy this could be a scan with predefined address range
> >> or simply listen who talks to the bus. For CANopen scan is quite
> >> straight, cause there is up to 127 nodes. You can submit a read out
> >> request for well known register address, ie. manufacturer code.
> >> Listening is also possible cause each node should (in theory) emit a
> >> heartbeat.
> >> - Discovery of CANopen fields readable or writable via driver.
> >> This is scan of node registers within predefined address ranges or
> >> ranges retrieved from initial node read out.
> >> I don't have this part (nor didn't plan to work on it anytime soon).
> >> This is probably a thing which is most interesting in case of data
> >> acquisition.
> >>
> >> Process is following:
> >> - [discovery] Physical interface [0..N]
> >> - [discovery] Discovery connection string [1:1] (?)
> >> - [discovery/browse] Finding accessible nodes [0..N]
> >> - [browse] Fields for a specific node [0..N]
> >>
> >> Discovery of Beckhoff stuff I did works in a following way:
> >> - Find network interfaces
> >> - Send broadcast over network interface
> >> - For each answered node
> >> - Try setting up a ADS route (requires PLC username and password)
> >> - Connect
> >> - Retrieve symbols (I did not implement this)
> >> Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
> >> you can clearly see that there are two different transports used. I
> >> believe KNX is same.
> >> For Profinet-DCP what I managed to complete with Adrian is focused only
> >> on two parts:
> >> - Finding accessible profinet devices
> >> Since they do answer with their own datas we can get their MAC which is
> >> later on used to communicate with them. Initial mapping of DCP ident
> >> response I did mapped that response to PlcStruct. Mainly because there
> >> are multiple fields which can come, including device name, DHCP settings
> >> or its ip address.
> >>
> >> Given the nature of node discovery it should by default stick with
> >> callbacks and not request/response modes. Browse is something we briefly
> >> spooked before, but it also fails into same category. We never know if
> >> discovery or browse is finite or not. For that reason API should rather
> >> assume later than earlier and let driver decide.
> >> For address range queries we indeed can have a progress indication,
> >> while for PN-DCP I don't think we should make any assumptions. I am not
> >> quire sure how big PN network can be, so I'll let others speak.
> >>
> >> In my understanding, also based on above openHAB experiences, discovery
> >> and browse APIs are kind of related, yet have different syntax.
> >> For Beckhoff definitely we get some parts for transportUrl options such
> >> as node AMS id which is advertised in identification response.
> >> For AMS there is also an authentication, however I don't think we're
> >> able to scrap this. In new versions I think there might be also a device
> >> certificate signature.
> >>
> >> Anyhow, these would be my 0.02€ of input.
> >>
> >> Best,
> >> Łukasz
> >>
> >> On 02.07.2021 17:02, Otto Fowler wrote:
> >>> Some protocols the discovery process returns information about the
> device
> >>> (DCP, ENIP) , that might be important as well. How would this be
> handled /
> >>> modeled ?
> >>>
> >>> The idea here is that plc4x doesn’t store any of this right? You’d have
> to
> >>> handle the callback and store the information yourself?
> >>>
> >>> From: Christofer Dutz <ch...@c-ware.de>
> >>> <ch...@c-ware.de>
> >>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <
> dev@plc4x.apache.org>
>
> >>> Date: June 30, 2021 at 13:53:32
> >>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> >>> Subject: [DISCUSS] Discover and Browse API for PLC4J?
> >>>
> >>> Hi all,
> >>>
> >>> as you all know, I have threatened to bring API additions from plc4go
> >>> br/>back to plc4j and now the time has come. But I want to invollve you
> all
> >>> br/>in it as we all have to live with it ;-) <
> >>>
> >>> So in general in go I did it the following way:
> >>>
> >>> - I split the topic up into "Discovery" and "Browse"
> >>>
> >>> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
> >>> br/>this should produce a stream of PlcBrowseEvents whichh provide a
> stream
> >>> br/>of PlcFField entries that can be used to query information. So this
> is
> >>> br/>inside an established connection to a device. It won't find devices
> >>> br/>(However this term is slightly ambbiguous because in KNX for
> example we
> >>> br/>consider a KNX devices GrouupAddress an address, even if in the
> >>> physical br/>world it actually iis a device)
> >>>
> >>> Discovery is the operation of finding out which devices we can connect
> >>> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
> >>> - protocolCode (like s7, modbus, ...)
> >>> - transportCode (like tcp, udp, serial, ...)
> >>> - transportUrl (which is the part after the "://" but before the "?"
> >>> - option (which is a map of key-value pairs making up the part after
> the
> >>> br/>""?" .... the part which Lukazs hates so much ;-) )
> >>>
> >>> In go I extended the DriverManager to have a discovery function which
> >>> br/>simply iterates over all drivers it knows and if they supporrt
> >>> discovery, br/>the functionality is executed. <
> >>>
> >>> In the past we usually returned a Future and this became complete as
> >>> br/>soon as the operation was finished. However Discovery can
> ssometimes
> >>> take br/>quite long. Some protocols support a direct ""Please all
> devices
> >>> br/>supporting this report to me"" functionality, but some we will need
> to
> >>> br/>potentially probe by trying all IP addreesses in a given range.
> >>> Therefore br/>I think such a future approach is sub-ideal.
> >>>
> >>> In go I gave the discover function a callback argument where you set a
> >>> br/>handler that gets notified directly as soon as a device is found. I
> am
> >>> a br/>bit unsure if this should be the way to go. <
> >>>
> >>> What I would love to do, would be to have the option to add one
> >>> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
> >>> br/>is found, then every handler is called and at the end the resuult
> is
> >>> put br/>into a list. As soon as the operation is complete, thee user
> could
> >>> then br/>use this list, just like our normal read-resultts and be happy
> >>> with it, br/>or he could use the callbacks if he wantts to be informed
> >>> directly.
> >>>
> >>> Unfortunately the Futures are so totally all-or-nothing. I would love
> to
> >>> br/>return something that knows the progress of the operatiion. In a
> >>> br/>Broadcast search it could simply be the progress towardd the
> timeout or
> >>> br/>when scanning an IP range it could be the percenntage of the
> addresses
> >>> br/>that were probed. This could help tools too show a nice progress on
> the
> >>> br/>auto-discovery. <
> >>>
> >>> What do you think? As I need this for my work on the PROFINET, I'm
> >>> br/>working on this on the feature/profinet-chris branch. <
> >>>
> >>> Chris
> >>>
>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Otto Fowler <ot...@gmail.com>.
Enip and Profinet Discovery and configuration protocol may ‘switch’ after
discovery.
Like, you send ENIP Identify on udp, and then connect on tcp to do other
commands I believe.
Also, you may use DCP to discover things, and then use s7 to talk to them (
the DCP identity response is over link layer, but has ip address as a field
).
From: Łukasz Dywicki <lu...@code-house.org> <lu...@code-house.org>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: July 2, 2021 at 13:17:14
To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: [DISCUSS] Discover and Browse API for PLC4J?
Hey Chris,
Indeed - discovery is about finding interfaces/devices while browsing is
already interacting with them. I believe that in most of cases once
discovery is done browsing uses same transport as PLC connection itself.
At least this would be logical consequence. If someone knows other case
then it is right time to speak! :-)
The AMS discovery answer returns you AMS net id of a node, so you know
IP of PLC and also its logical network identifier. As you pointed
usually it is just an <ip>.1.1. I actually use this convention when
making discovery caller ams. It is just my-ip + 1.1.
What we miss is a ams port (by default there is 851, but there are
others and it can vary). The user and password is needed only for route
setup, if you have route set up on PLC then this step is not necessary.
Anyhow, I think we need to have a look on OPC UA discovery (which is a
separate sub-spec) to see if we're set on discovery inputs. Given its
verbosity I do expect additional set of requirements coming from that end.
For KNX I see indeed a similarity between CANopen (and profinet) how
node discovery could work. All these are "bus systems" thus connection
is not a point to point. In their case we see multiple participants
which are listening to a single communication channel. As soon as we
acquire link we can discover and browse through it.
A short note. I am not sure if we could automate plain CAN discovery,
cause then we have 11 bit identifier which can be anything. Role of
these 11 bits is known only if we assume CANopen.
In worst case basic CAN could lead to us to discovery of up to 4095 CAN
nodes. ;-)
Anyhow, I believe what you have sketched is pretty clear.
Best,
Łukasz
On 02.07.2021 18:23, Christofer Dutz wrote:
> Hi Lukasz,
> br/>> so from what I read I sort of think you also aggree that
"Discovery"
> should find devices and the result is connection information we can then
> pass to the DriverManager to get a connection. And Browse will result
> with fields (Or parsed address strings) for a given connection.
> br/>> I think in the ADS Case ... we shouldn't do thee setting up of ADS
Routes
> as it involves sending a servers Admin password in clear text. However I
> think we are able to find an ADS device. As with ADS you also have to
> provide the source and targer ams net id, it will be challenging to
> guess these (Or does the discovery handle that?). However we could
> assume the default of "appending ".1.1" to the local IP which seems to
> be quite established. If however the discovery allows the listing up of
> AMSNetIds a device running on a given IP has, this would be awesome.
> br/>> I think KNX and CAN can be considered a bit simmilar regarding
what's a
> device and what's a ressouce. For KNX we currently consider the KNXNetIP
> Gateway the device (and therefore the connection string only has the
> coordinates to connect to that) and then the KNX Devices on the KNXBus
> are considered Resources which are found by a Browse operation.
> When thinking of KNX I would consider the Device resources sort of like
> with S7 ... the KNX Device on the KNX Bus is the DataBlock and the
> resource on the device is the address relative to that DataBlock. So if
> I want to read a parameter of a KNX device, then I add the parmeter
> coordinates to the knx address which makes up the field address in PLC4X.
> br/>> It looks like it should be possible to bring thhe CAN driver into
> supporting the same mechanisms here too.
> br/>> Chris <
> br/>> On 02.07.21 17:53, &##321;ukasz Dywicki wrote:
>> Hey Chris,
>> I do agree that browse and discovery is different and it might be either
>> expanded or narrowed.
>>
>> For example openHAB stuff I did based on CANopen driver does following:
>> - Discovery of CAN enabled interface
>> For linux it is quite straight cause you can use either a "net" part of
>> sysfs or even a dbus [1] in order to find interesting network nodes.
>> - Discvoery of CANopen nodes
>> Depending on strategy this could be a scan with predefined address range
>> or simply listen who talks to the bus. For CANopen scan is quite
>> straight, cause there is up to 127 nodes. You can submit a read out
>> request for well known register address, ie. manufacturer code.
>> Listening is also possible cause each node should (in theory) emit a
>> heartbeat.
>> - Discovery of CANopen fields readable or writable via driver.
>> This is scan of node registers within predefined address ranges or
>> ranges retrieved from initial node read out.
>> I don't have this part (nor didn't plan to work on it anytime soon).
>> This is probably a thing which is most interesting in case of data
>> acquisition.
>>
>> Process is following:
>> - [discovery] Physical interface [0..N]
>> - [discovery] Discovery connection string [1:1] (?)
>> - [discovery/browse] Finding accessible nodes [0..N]
>> - [browse] Fields for a specific node [0..N]
>>
>> Discovery of Beckhoff stuff I did works in a following way:
>> - Find network interfaces
>> - Send broadcast over network interface
>> - For each answered node
>> - Try setting up a ADS route (requires PLC username and password)
>> - Connect
>> - Retrieve symbols (I did not implement this)
>> Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
>> you can clearly see that there are two different transports used. I
>> believe KNX is same.
>> For Profinet-DCP what I managed to complete with Adrian is focused only
>> on two parts:
>> - Finding accessible profinet devices
>> Since they do answer with their own datas we can get their MAC which is
>> later on used to communicate with them. Initial mapping of DCP ident
>> response I did mapped that response to PlcStruct. Mainly because there
>> are multiple fields which can come, including device name, DHCP settings
>> or its ip address.
>>
>> Given the nature of node discovery it should by default stick with
>> callbacks and not request/response modes. Browse is something we briefly
>> spooked before, but it also fails into same category. We never know if
>> discovery or browse is finite or not. For that reason API should rather
>> assume later than earlier and let driver decide.
>> For address range queries we indeed can have a progress indication,
>> while for PN-DCP I don't think we should make any assumptions. I am not
>> quire sure how big PN network can be, so I'll let others speak.
>>
>> In my understanding, also based on above openHAB experiences, discovery
>> and browse APIs are kind of related, yet have different syntax.
>> For Beckhoff definitely we get some parts for transportUrl options such
>> as node AMS id which is advertised in identification response.
>> For AMS there is also an authentication, however I don't think we're
>> able to scrap this. In new versions I think there might be also a device
>> certificate signature.
>>
>> Anyhow, these would be my 0.02€ of input.
>>
>> Best,
>> Łukasz
>>
>> On 02.07.2021 17:02, Otto Fowler wrote:
>>> Some protocols the discovery process returns information about the
device
>>> (DCP, ENIP) , that might be important as well. How would this be
handled /
>>> modeled ?
>>>
>>> The idea here is that plc4x doesn’t store any of this right? You’d have
to
>>> handle the callback and store the information yourself?
>>>
>>> From: Christofer Dutz <ch...@c-ware.de>
>>> <ch...@c-ware.de>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Date: June 30, 2021 at 13:53:32
>>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>>>
>>> Hi all,
>>>
>>> as you all know, I have threatened to bring API additions from plc4go
>>> br/>back to plc4j and now the time has come. But I want to invollve you
all
>>> br/>in it as we all have to live with it ;-) <
>>>
>>> So in general in go I did it the following way:
>>>
>>> - I split the topic up into "Discovery" and "Browse"
>>>
>>> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
>>> br/>this should produce a stream of PlcBrowseEvents whichh provide a
stream
>>> br/>of PlcFField entries that can be used to query information. So this
is
>>> br/>inside an established connection to a device. It won't find devices
>>> br/>(However this term is slightly ambbiguous because in KNX for
example we
>>> br/>consider a KNX devices GrouupAddress an address, even if in the
>>> physical br/>world it actually iis a device)
>>>
>>> Discovery is the operation of finding out which devices we can connect
>>> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
>>> - protocolCode (like s7, modbus, ...)
>>> - transportCode (like tcp, udp, serial, ...)
>>> - transportUrl (which is the part after the "://" but before the "?"
>>> - option (which is a map of key-value pairs making up the part after
the
>>> br/>""?" .... the part which Lukazs hates so much ;-) )
>>>
>>> In go I extended the DriverManager to have a discovery function which
>>> br/>simply iterates over all drivers it knows and if they supporrt
>>> discovery, br/>the functionality is executed. <
>>>
>>> In the past we usually returned a Future and this became complete as
>>> br/>soon as the operation was finished. However Discovery can
ssometimes
>>> take br/>quite long. Some protocols support a direct ""Please all
devices
>>> br/>supporting this report to me"" functionality, but some we will need
to
>>> br/>potentially probe by trying all IP addreesses in a given range.
>>> Therefore br/>I think such a future approach is sub-ideal.
>>>
>>> In go I gave the discover function a callback argument where you set a
>>> br/>handler that gets notified directly as soon as a device is found. I
am
>>> a br/>bit unsure if this should be the way to go. <
>>>
>>> What I would love to do, would be to have the option to add one
>>> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
>>> br/>is found, then every handler is called and at the end the resuult
is
>>> put br/>into a list. As soon as the operation is complete, thee user
could
>>> then br/>use this list, just like our normal read-resultts and be happy
>>> with it, br/>or he could use the callbacks if he wantts to be informed
>>> directly.
>>>
>>> Unfortunately the Futures are so totally all-or-nothing. I would love
to
>>> br/>return something that knows the progress of the operatiion. In a
>>> br/>Broadcast search it could simply be the progress towardd the
timeout or
>>> br/>when scanning an IP range it could be the percenntage of the
addresses
>>> br/>that were probed. This could help tools too show a nice progress on
the
>>> br/>auto-discovery. <
>>>
>>> What do you think? As I need this for my work on the PROFINET, I'm
>>> br/>working on this on the feature/profinet-chris branch. <
>>>
>>> Chris
>>>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Łukasz Dywicki <lu...@code-house.org>.
Hey Chris,
Indeed - discovery is about finding interfaces/devices while browsing is
already interacting with them. I believe that in most of cases once
discovery is done browsing uses same transport as PLC connection itself.
At least this would be logical consequence. If someone knows other case
then it is right time to speak! :-)
The AMS discovery answer returns you AMS net id of a node, so you know
IP of PLC and also its logical network identifier. As you pointed
usually it is just an <ip>.1.1. I actually use this convention when
making discovery caller ams. It is just my-ip + 1.1.
What we miss is a ams port (by default there is 851, but there are
others and it can vary). The user and password is needed only for route
setup, if you have route set up on PLC then this step is not necessary.
Anyhow, I think we need to have a look on OPC UA discovery (which is a
separate sub-spec) to see if we're set on discovery inputs. Given its
verbosity I do expect additional set of requirements coming from that end.
For KNX I see indeed a similarity between CANopen (and profinet) how
node discovery could work. All these are "bus systems" thus connection
is not a point to point. In their case we see multiple participants
which are listening to a single communication channel. As soon as we
acquire link we can discover and browse through it.
A short note. I am not sure if we could automate plain CAN discovery,
cause then we have 11 bit identifier which can be anything. Role of
these 11 bits is known only if we assume CANopen.
In worst case basic CAN could lead to us to discovery of up to 4095 CAN
nodes. ;-)
Anyhow, I believe what you have sketched is pretty clear.
Best,
Łukasz
On 02.07.2021 18:23, Christofer Dutz wrote:
> Hi Lukasz,
>
> so from what I read I sort of think you also agree that "Discovery"
> should find devices and the result is connection information we can then
> pass to the DriverManager to get a connection. And Browse will result
> with fields (Or parsed address strings) for a given connection.
>
> I think in the ADS Case ... we shouldn't do the setting up of ADS Routes
> as it involves sending a servers Admin password in clear text. However I
> think we are able to find an ADS device. As with ADS you also have to
> provide the source and targer ams net id, it will be challenging to
> guess these (Or does the discovery handle that?). However we could
> assume the default of "appending ".1.1" to the local IP which seems to
> be quite established. If however the discovery allows the listing up of
> AMSNetIds a device running on a given IP has, this would be awesome.
>
> I think KNX and CAN can be considered a bit similar regarding what's a
> device and what's a ressouce. For KNX we currently consider the KNXNetIP
> Gateway the device (and therefore the connection string only has the
> coordinates to connect to that) and then the KNX Devices on the KNXBus
> are considered Resources which are found by a Browse operation.
> When thinking of KNX I would consider the Device resources sort of like
> with S7 ... the KNX Device on the KNX Bus is the DataBlock and the
> resource on the device is the address relative to that DataBlock. So if
> I want to read a parameter of a KNX device, then I add the parmeter
> coordinates to the knx address which makes up the field address in PLC4X.
>
> It looks like it should be possible to bring the CAN driver into
> supporting the same mechanisms here too.
>
> Chris
>
> On 02.07.21 17:53, Łukasz Dywicki wrote:
>> Hey Chris,
>> I do agree that browse and discovery is different and it might be either
>> expanded or narrowed.
>>
>> For example openHAB stuff I did based on CANopen driver does following:
>> - Discovery of CAN enabled interface
>> For linux it is quite straight cause you can use either a "net" part of
>> sysfs or even a dbus [1] in order to find interesting network nodes.
>> - Discvoery of CANopen nodes
>> Depending on strategy this could be a scan with predefined address range
>> or simply listen who talks to the bus. For CANopen scan is quite
>> straight, cause there is up to 127 nodes. You can submit a read out
>> request for well known register address, ie. manufacturer code.
>> Listening is also possible cause each node should (in theory) emit a
>> heartbeat.
>> - Discovery of CANopen fields readable or writable via driver.
>> This is scan of node registers within predefined address ranges or
>> ranges retrieved from initial node read out.
>> I don't have this part (nor didn't plan to work on it anytime soon).
>> This is probably a thing which is most interesting in case of data
>> acquisition.
>>
>> Process is following:
>> - [discovery] Physical interface [0..N]
>> - [discovery] Discovery connection string [1:1] (?)
>> - [discovery/browse] Finding accessible nodes [0..N]
>> - [browse] Fields for a specific node [0..N]
>>
>> Discovery of Beckhoff stuff I did works in a following way:
>> - Find network interfaces
>> - Send broadcast over network interface
>> - For each answered node
>> - Try setting up a ADS route (requires PLC username and password)
>> - Connect
>> - Retrieve symbols (I did not implement this)
>> Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
>> you can clearly see that there are two different transports used. I
>> believe KNX is same.
>> For Profinet-DCP what I managed to complete with Adrian is focused only
>> on two parts:
>> - Finding accessible profinet devices
>> Since they do answer with their own datas we can get their MAC which is
>> later on used to communicate with them. Initial mapping of DCP ident
>> response I did mapped that response to PlcStruct. Mainly because there
>> are multiple fields which can come, including device name, DHCP settings
>> or its ip address.
>>
>> Given the nature of node discovery it should by default stick with
>> callbacks and not request/response modes. Browse is something we briefly
>> spooked before, but it also fails into same category. We never know if
>> discovery or browse is finite or not. For that reason API should rather
>> assume later than earlier and let driver decide.
>> For address range queries we indeed can have a progress indication,
>> while for PN-DCP I don't think we should make any assumptions. I am not
>> quire sure how big PN network can be, so I'll let others speak.
>>
>> In my understanding, also based on above openHAB experiences, discovery
>> and browse APIs are kind of related, yet have different syntax.
>> For Beckhoff definitely we get some parts for transportUrl options such
>> as node AMS id which is advertised in identification response.
>> For AMS there is also an authentication, however I don't think we're
>> able to scrap this. In new versions I think there might be also a device
>> certificate signature.
>>
>> Anyhow, these would be my 0.02€ of input.
>>
>> Best,
>> Łukasz
>>
>> On 02.07.2021 17:02, Otto Fowler wrote:
>>> Some protocols the discovery process returns information about the device
>>> (DCP, ENIP) , that might be important as well. How would this be handled /
>>> modeled ?
>>>
>>> The idea here is that plc4x doesn’t store any of this right? You’d have to
>>> handle the callback and store the information yourself?
>>>
>>> From: Christofer Dutz <ch...@c-ware.de>
>>> <ch...@c-ware.de>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Date: June 30, 2021 at 13:53:32
>>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>>>
>>> Hi all,
>>>
>>> as you all know, I have threatened to bring API additions from plc4go
>>> br/>back to plc4j and now the time has come. But I want to invollve you all
>>> br/>in it as we all have to live with it ;-) <
>>>
>>> So in general in go I did it the following way:
>>>
>>> - I split the topic up into "Discovery" and "Browse"
>>>
>>> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
>>> br/>this should produce a stream of PlcBrowseEvents whichh provide a stream
>>> br/>of PlcFField entries that can be used to query information. So this is
>>> br/>inside an established connection to a device. It won't find devices
>>> br/>(However this term is slightly ambbiguous because in KNX for example we
>>> br/>consider a KNX devices GrouupAddress an address, even if in the
>>> physical br/>world it actually iis a device)
>>>
>>> Discovery is the operation of finding out which devices we can connect
>>> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
>>> - protocolCode (like s7, modbus, ...)
>>> - transportCode (like tcp, udp, serial, ...)
>>> - transportUrl (which is the part after the "://" but before the "?"
>>> - option (which is a map of key-value pairs making up the part after the
>>> br/>""?" .... the part which Lukazs hates so much ;-) )
>>>
>>> In go I extended the DriverManager to have a discovery function which
>>> br/>simply iterates over all drivers it knows and if they supporrt
>>> discovery, br/>the functionality is executed. <
>>>
>>> In the past we usually returned a Future and this became complete as
>>> br/>soon as the operation was finished. However Discovery can ssometimes
>>> take br/>quite long. Some protocols support a direct ""Please all devices
>>> br/>supporting this report to me"" functionality, but some we will need to
>>> br/>potentially probe by trying all IP addreesses in a given range.
>>> Therefore br/>I think such a future approach is sub-ideal.
>>>
>>> In go I gave the discover function a callback argument where you set a
>>> br/>handler that gets notified directly as soon as a device is found. I am
>>> a br/>bit unsure if this should be the way to go. <
>>>
>>> What I would love to do, would be to have the option to add one
>>> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
>>> br/>is found, then every handler is called and at the end the resuult is
>>> put br/>into a list. As soon as the operation is complete, thee user could
>>> then br/>use this list, just like our normal read-resultts and be happy
>>> with it, br/>or he could use the callbacks if he wantts to be informed
>>> directly.
>>>
>>> Unfortunately the Futures are so totally all-or-nothing. I would love to
>>> br/>return something that knows the progress of the operatiion. In a
>>> br/>Broadcast search it could simply be the progress towardd the timeout or
>>> br/>when scanning an IP range it could be the percenntage of the addresses
>>> br/>that were probed. This could help tools too show a nice progress on the
>>> br/>auto-discovery. <
>>>
>>> What do you think? As I need this for my work on the PROFINET, I'm
>>> br/>working on this on the feature/profinet-chris branch. <
>>>
>>> Chris
>>>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Lukasz,
so from what I read I sort of think you also agree that "Discovery"
should find devices and the result is connection information we can then
pass to the DriverManager to get a connection. And Browse will result
with fields (Or parsed address strings) for a given connection.
I think in the ADS Case ... we shouldn't do the setting up of ADS Routes
as it involves sending a servers Admin password in clear text. However I
think we are able to find an ADS device. As with ADS you also have to
provide the source and targer ams net id, it will be challenging to
guess these (Or does the discovery handle that?). However we could
assume the default of "appending ".1.1" to the local IP which seems to
be quite established. If however the discovery allows the listing up of
AMSNetIds a device running on a given IP has, this would be awesome.
I think KNX and CAN can be considered a bit similar regarding what's a
device and what's a ressouce. For KNX we currently consider the KNXNetIP
Gateway the device (and therefore the connection string only has the
coordinates to connect to that) and then the KNX Devices on the KNXBus
are considered Resources which are found by a Browse operation.
When thinking of KNX I would consider the Device resources sort of like
with S7 ... the KNX Device on the KNX Bus is the DataBlock and the
resource on the device is the address relative to that DataBlock. So if
I want to read a parameter of a KNX device, then I add the parmeter
coordinates to the knx address which makes up the field address in PLC4X.
It looks like it should be possible to bring the CAN driver into
supporting the same mechanisms here too.
Chris
On 02.07.21 17:53, Łukasz Dywicki wrote:
> Hey Chris,
> I do agree that browse and discovery is different and it might be either
> expanded or narrowed.
>
> For example openHAB stuff I did based on CANopen driver does following:
> - Discovery of CAN enabled interface
> For linux it is quite straight cause you can use either a "net" part of
> sysfs or even a dbus [1] in order to find interesting network nodes.
> - Discvoery of CANopen nodes
> Depending on strategy this could be a scan with predefined address range
> or simply listen who talks to the bus. For CANopen scan is quite
> straight, cause there is up to 127 nodes. You can submit a read out
> request for well known register address, ie. manufacturer code.
> Listening is also possible cause each node should (in theory) emit a
> heartbeat.
> - Discovery of CANopen fields readable or writable via driver.
> This is scan of node registers within predefined address ranges or
> ranges retrieved from initial node read out.
> I don't have this part (nor didn't plan to work on it anytime soon).
> This is probably a thing which is most interesting in case of data
> acquisition.
>
> Process is following:
> - [discovery] Physical interface [0..N]
> - [discovery] Discovery connection string [1:1] (?)
> - [discovery/browse] Finding accessible nodes [0..N]
> - [browse] Fields for a specific node [0..N]
>
> Discovery of Beckhoff stuff I did works in a following way:
> - Find network interfaces
> - Send broadcast over network interface
> - For each answered node
> - Try setting up a ADS route (requires PLC username and password)
> - Connect
> - Retrieve symbols (I did not implement this)
> Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
> you can clearly see that there are two different transports used. I
> believe KNX is same.
> For Profinet-DCP what I managed to complete with Adrian is focused only
> on two parts:
> - Finding accessible profinet devices
> Since they do answer with their own datas we can get their MAC which is
> later on used to communicate with them. Initial mapping of DCP ident
> response I did mapped that response to PlcStruct. Mainly because there
> are multiple fields which can come, including device name, DHCP settings
> or its ip address.
>
> Given the nature of node discovery it should by default stick with
> callbacks and not request/response modes. Browse is something we briefly
> spooked before, but it also fails into same category. We never know if
> discovery or browse is finite or not. For that reason API should rather
> assume later than earlier and let driver decide.
> For address range queries we indeed can have a progress indication,
> while for PN-DCP I don't think we should make any assumptions. I am not
> quire sure how big PN network can be, so I'll let others speak.
>
> In my understanding, also based on above openHAB experiences, discovery
> and browse APIs are kind of related, yet have different syntax.
> For Beckhoff definitely we get some parts for transportUrl options such
> as node AMS id which is advertised in identification response.
> For AMS there is also an authentication, however I don't think we're
> able to scrap this. In new versions I think there might be also a device
> certificate signature.
>
> Anyhow, these would be my 0.02€ of input.
>
> Best,
> Łukasz
>
> On 02.07.2021 17:02, Otto Fowler wrote:
>> Some protocols the discovery process returns information about the device
>> (DCP, ENIP) , that might be important as well. How would this be handled /
>> modeled ?
>>
>> The idea here is that plc4x doesn’t store any of this right? You’d have to
>> handle the callback and store the information yourself?
>>
>> From: Christofer Dutz <ch...@c-ware.de>
>> <ch...@c-ware.de>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Date: June 30, 2021 at 13:53:32
>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>>
>> Hi all,
>>
>> as you all know, I have threatened to bring API additions from plc4go
>> br/>back to plc4j and now the time has come. But I want to invollve you all
>> br/>in it as we all have to live with it ;-) <
>>
>> So in general in go I did it the following way:
>>
>> - I split the topic up into "Discovery" and "Browse"
>>
>> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
>> br/>this should produce a stream of PlcBrowseEvents whichh provide a stream
>> br/>of PlcFField entries that can be used to query information. So this is
>> br/>inside an established connection to a device. It won't find devices
>> br/>(However this term is slightly ambbiguous because in KNX for example we
>> br/>consider a KNX devices GrouupAddress an address, even if in the
>> physical br/>world it actually iis a device)
>>
>> Discovery is the operation of finding out which devices we can connect
>> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
>> - protocolCode (like s7, modbus, ...)
>> - transportCode (like tcp, udp, serial, ...)
>> - transportUrl (which is the part after the "://" but before the "?"
>> - option (which is a map of key-value pairs making up the part after the
>> br/>""?" .... the part which Lukazs hates so much ;-) )
>>
>> In go I extended the DriverManager to have a discovery function which
>> br/>simply iterates over all drivers it knows and if they supporrt
>> discovery, br/>the functionality is executed. <
>>
>> In the past we usually returned a Future and this became complete as
>> br/>soon as the operation was finished. However Discovery can ssometimes
>> take br/>quite long. Some protocols support a direct ""Please all devices
>> br/>supporting this report to me"" functionality, but some we will need to
>> br/>potentially probe by trying all IP addreesses in a given range.
>> Therefore br/>I think such a future approach is sub-ideal.
>>
>> In go I gave the discover function a callback argument where you set a
>> br/>handler that gets notified directly as soon as a device is found. I am
>> a br/>bit unsure if this should be the way to go. <
>>
>> What I would love to do, would be to have the option to add one
>> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
>> br/>is found, then every handler is called and at the end the resuult is
>> put br/>into a list. As soon as the operation is complete, thee user could
>> then br/>use this list, just like our normal read-resultts and be happy
>> with it, br/>or he could use the callbacks if he wantts to be informed
>> directly.
>>
>> Unfortunately the Futures are so totally all-or-nothing. I would love to
>> br/>return something that knows the progress of the operatiion. In a
>> br/>Broadcast search it could simply be the progress towardd the timeout or
>> br/>when scanning an IP range it could be the percenntage of the addresses
>> br/>that were probed. This could help tools too show a nice progress on the
>> br/>auto-discovery. <
>>
>> What do you think? As I need this for my work on the PROFINET, I'm
>> br/>working on this on the feature/profinet-chris branch. <
>>
>> Chris
>>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Łukasz Dywicki <lu...@code-house.org>.
Hey Chris,
I do agree that browse and discovery is different and it might be either
expanded or narrowed.
For example openHAB stuff I did based on CANopen driver does following:
- Discovery of CAN enabled interface
For linux it is quite straight cause you can use either a "net" part of
sysfs or even a dbus [1] in order to find interesting network nodes.
- Discvoery of CANopen nodes
Depending on strategy this could be a scan with predefined address range
or simply listen who talks to the bus. For CANopen scan is quite
straight, cause there is up to 127 nodes. You can submit a read out
request for well known register address, ie. manufacturer code.
Listening is also possible cause each node should (in theory) emit a
heartbeat.
- Discovery of CANopen fields readable or writable via driver.
This is scan of node registers within predefined address ranges or
ranges retrieved from initial node read out.
I don't have this part (nor didn't plan to work on it anytime soon).
This is probably a thing which is most interesting in case of data
acquisition.
Process is following:
- [discovery] Physical interface [0..N]
- [discovery] Discovery connection string [1:1] (?)
- [discovery/browse] Finding accessible nodes [0..N]
- [browse] Fields for a specific node [0..N]
Discovery of Beckhoff stuff I did works in a following way:
- Find network interfaces
- Send broadcast over network interface
- For each answered node
- Try setting up a ADS route (requires PLC username and password)
- Connect
- Retrieve symbols (I did not implement this)
Beckhoff uses a UDP for discovery and connection to PLC is over TCP thus
you can clearly see that there are two different transports used. I
believe KNX is same.
For Profinet-DCP what I managed to complete with Adrian is focused only
on two parts:
- Finding accessible profinet devices
Since they do answer with their own datas we can get their MAC which is
later on used to communicate with them. Initial mapping of DCP ident
response I did mapped that response to PlcStruct. Mainly because there
are multiple fields which can come, including device name, DHCP settings
or its ip address.
Given the nature of node discovery it should by default stick with
callbacks and not request/response modes. Browse is something we briefly
spooked before, but it also fails into same category. We never know if
discovery or browse is finite or not. For that reason API should rather
assume later than earlier and let driver decide.
For address range queries we indeed can have a progress indication,
while for PN-DCP I don't think we should make any assumptions. I am not
quire sure how big PN network can be, so I'll let others speak.
In my understanding, also based on above openHAB experiences, discovery
and browse APIs are kind of related, yet have different syntax.
For Beckhoff definitely we get some parts for transportUrl options such
as node AMS id which is advertised in identification response.
For AMS there is also an authentication, however I don't think we're
able to scrap this. In new versions I think there might be also a device
certificate signature.
Anyhow, these would be my 0.02€ of input.
Best,
Łukasz
On 02.07.2021 17:02, Otto Fowler wrote:
> Some protocols the discovery process returns information about the device
> (DCP, ENIP) , that might be important as well. How would this be handled /
> modeled ?
>
> The idea here is that plc4x doesn’t store any of this right? You’d have to
> handle the callback and store the information yourself?
>
> From: Christofer Dutz <ch...@c-ware.de>
> <ch...@c-ware.de>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: June 30, 2021 at 13:53:32
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>
> Hi all,
>
> as you all know, I have threatened to bring API additions from plc4go
> br/>back to plc4j and now the time has come. But I want to invollve you all
> br/>in it as we all have to live with it ;-) <
>
> So in general in go I did it the following way:
>
> - I split the topic up into "Discovery" and "Browse"
>
> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
> br/>this should produce a stream of PlcBrowseEvents whichh provide a stream
> br/>of PlcFField entries that can be used to query information. So this is
> br/>inside an established connection to a device. It won't find devices
> br/>(However this term is slightly ambbiguous because in KNX for example we
> br/>consider a KNX devices GrouupAddress an address, even if in the
> physical br/>world it actually iis a device)
>
> Discovery is the operation of finding out which devices we can connect
> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
> - protocolCode (like s7, modbus, ...)
> - transportCode (like tcp, udp, serial, ...)
> - transportUrl (which is the part after the "://" but before the "?"
> - option (which is a map of key-value pairs making up the part after the
> br/>""?" .... the part which Lukazs hates so much ;-) )
>
> In go I extended the DriverManager to have a discovery function which
> br/>simply iterates over all drivers it knows and if they supporrt
> discovery, br/>the functionality is executed. <
>
> In the past we usually returned a Future and this became complete as
> br/>soon as the operation was finished. However Discovery can ssometimes
> take br/>quite long. Some protocols support a direct ""Please all devices
> br/>supporting this report to me"" functionality, but some we will need to
> br/>potentially probe by trying all IP addreesses in a given range.
> Therefore br/>I think such a future approach is sub-ideal.
>
> In go I gave the discover function a callback argument where you set a
> br/>handler that gets notified directly as soon as a device is found. I am
> a br/>bit unsure if this should be the way to go. <
>
> What I would love to do, would be to have the option to add one
> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
> br/>is found, then every handler is called and at the end the resuult is
> put br/>into a list. As soon as the operation is complete, thee user could
> then br/>use this list, just like our normal read-resultts and be happy
> with it, br/>or he could use the callbacks if he wantts to be informed
> directly.
>
> Unfortunately the Futures are so totally all-or-nothing. I would love to
> br/>return something that knows the progress of the operatiion. In a
> br/>Broadcast search it could simply be the progress towardd the timeout or
> br/>when scanning an IP range it could be the percenntage of the addresses
> br/>that were probed. This could help tools too show a nice progress on the
> br/>auto-discovery. <
>
> What do you think? As I need this for my work on the PROFINET, I'm
> br/>working on this on the feature/profinet-chris branch. <
>
> Chris
>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Otto,
the idea was, that if for example a device (like a S7) supports S7,
Modbus and OpcUA, that the result of the Discover would return 3
connection strings ... one for each supported protocol. <
Chris
On 02.07.21 17:02, Otto Fowler wrote:
> Some protocols the discovery process returns information about the device
> (DCP, ENIP) , that might be important as well. How would this be handled /
> modeled ?
>
> The idea here is that plc4x doesn’t store any of this right? You’d have to
> handle the callback and store the information yourself?
>
> From: Christofer Dutz <ch...@c-ware.de>
> <ch...@c-ware.de>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: June 30, 2021 at 13:53:32
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: [DISCUSS] Discover and Browse API for PLC4J?
>
> Hi all,
>
> as you all know, I have threatened to bring API additions from plc4go
> br/>back to plc4j and now the time has come. But I want to invollve you all
> br/>in it as we all have to live with it ;-) <
>
> So in general in go I did it the following way:
>
> - I split the topic up into "Discovery" and "Browse"
>
> "Browse" I'll focus on as soon ad "Discovery" is defined. In general
> br/>this should produce a stream of PlcBrowseEvents whichh provide a stream
> br/>of PlcFField entries that can be used to query information. So this is
> br/>inside an established connection to a device. It won't find devices
> br/>(However this term is slightly ambbiguous because in KNX for example we
> br/>consider a KNX devices GrouupAddress an address, even if in the
> physical br/>world it actually iis a device)
>
> Discovery is the operation of finding out which devices we can connect
> br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
> - protocolCode (like s7, modbus, ...)
> - transportCode (like tcp, udp, serial, ...)
> - transportUrl (which is the part after the "://" but before the "?"
> - option (which is a map of key-value pairs making up the part after the
> br/>""?" .... the part which Lukazs hates so much ;-) )
>
> In go I extended the DriverManager to have a discovery function which
> br/>simply iterates over all drivers it knows and if they supporrt
> discovery, br/>the functionality is executed. <
>
> In the past we usually returned a Future and this became complete as
> br/>soon as the operation was finished. However Discovery can ssometimes
> take br/>quite long. Some protocols support a direct ""Please all devices
> br/>supporting this report to me"" functionality, but some we will need to
> br/>potentially probe by trying all IP addreesses in a given range.
> Therefore br/>I think such a future approach is sub-ideal.
>
> In go I gave the discover function a callback argument where you set a
> br/>handler that gets notified directly as soon as a device is found. I am
> a br/>bit unsure if this should be the way to go. <
>
> What I would love to do, would be to have the option to add one
> (possibly multiple) handlers to a DiscoveryRequest. Whenever something
> br/>is found, then every handler is called and at the end the resuult is
> put br/>into a list. As soon as the operation is complete, thee user could
> then br/>use this list, just like our normal read-resultts and be happy
> with it, br/>or he could use the callbacks if he wantts to be informed
> directly.
>
> Unfortunately the Futures are so totally all-or-nothing. I would love to
> br/>return something that knows the progress of the operatiion. In a
> br/>Broadcast search it could simply be the progress towardd the timeout or
> br/>when scanning an IP range it could be the percenntage of the addresses
> br/>that were probed. This could help tools too show a nice progress on the
> br/>auto-discovery. <
>
> What do you think? As I need this for my work on the PROFINET, I'm
> br/>working on this on the feature/profinet-chris branch. <
>
> Chris
>
Re: [DISCUSS] Discover and Browse API for PLC4J?
Posted by Otto Fowler <ot...@gmail.com>.
Some protocols the discovery process returns information about the device
(DCP, ENIP) , that might be important as well. How would this be handled /
modeled ?
The idea here is that plc4x doesn’t store any of this right? You’d have to
handle the callback and store the information yourself?
From: Christofer Dutz <ch...@c-ware.de>
<ch...@c-ware.de>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: June 30, 2021 at 13:53:32
To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: [DISCUSS] Discover and Browse API for PLC4J?
Hi all,
as you all know, I have threatened to bring API additions from plc4go
br/>back to plc4j and now the time has come. But I want to invollve you all
br/>in it as we all have to live with it ;-) <
So in general in go I did it the following way:
- I split the topic up into "Discovery" and "Browse"
"Browse" I'll focus on as soon ad "Discovery" is defined. In general
br/>this should produce a stream of PlcBrowseEvents whichh provide a stream
br/>of PlcFField entries that can be used to query information. So this is
br/>inside an established connection to a device. It won't find devices
br/>(However this term is slightly ambbiguous because in KNX for example we
br/>consider a KNX devices GrouupAddress an address, even if in the
physical br/>world it actually iis a device)
Discovery is the operation of finding out which devices we can connect
br/>to. The result is a list of PlcDiscoveryEvents. These contaain:
- protocolCode (like s7, modbus, ...)
- transportCode (like tcp, udp, serial, ...)
- transportUrl (which is the part after the "://" but before the "?"
- option (which is a map of key-value pairs making up the part after the
br/>""?" .... the part which Lukazs hates so much ;-) )
In go I extended the DriverManager to have a discovery function which
br/>simply iterates over all drivers it knows and if they supporrt
discovery, br/>the functionality is executed. <
In the past we usually returned a Future and this became complete as
br/>soon as the operation was finished. However Discovery can ssometimes
take br/>quite long. Some protocols support a direct ""Please all devices
br/>supporting this report to me"" functionality, but some we will need to
br/>potentially probe by trying all IP addreesses in a given range.
Therefore br/>I think such a future approach is sub-ideal.
In go I gave the discover function a callback argument where you set a
br/>handler that gets notified directly as soon as a device is found. I am
a br/>bit unsure if this should be the way to go. <
What I would love to do, would be to have the option to add one
(possibly multiple) handlers to a DiscoveryRequest. Whenever something
br/>is found, then every handler is called and at the end the resuult is
put br/>into a list. As soon as the operation is complete, thee user could
then br/>use this list, just like our normal read-resultts and be happy
with it, br/>or he could use the callbacks if he wantts to be informed
directly.
Unfortunately the Futures are so totally all-or-nothing. I would love to
br/>return something that knows the progress of the operatiion. In a
br/>Broadcast search it could simply be the progress towardd the timeout or
br/>when scanning an IP range it could be the percenntage of the addresses
br/>that were probed. This could help tools too show a nice progress on the
br/>auto-discovery. <
What do you think? As I need this for my work on the PROFINET, I'm
br/>working on this on the feature/profinet-chris branch. <
Chris