You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@plc4x.apache.org by Andreas Vogler <an...@me.com.INVALID> on 2021/03/05 20:02:43 UTC

Source Time of PLC Value?

Hi,

How can I get the source time of a PlcValue? 

a) from a subscription PlcSubscriptionEvent
b) from a read request  PlcReadResponse

Best regards,
Andreas

Re: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Brief proposal I see is below.
Please forgive coding via mailing, its fastest way to verify whether
there is an agreement.

interface PlcResponse {
  // new operation
  default PlcResponseMetadata getResponseMetadata() {
    return Empty_Response_Metadata;
  }
}

interface PlcFieldResponse {
  // new operation
  default PlcFieldResponseMetadata getFieldMetadata() {
    return Empty_Field_Metadata;
  }
}

interface PlcFieldResponseMetadata {
  <T> Optional<FieldMetadataKey<T>> getMetadata(String field,
Class<FieldMetadataKey<T>> key);

  Set<FieldMetadataKey<?>> getFieldMetadata(String field);
}

interface MetadataKey<T> {
  // just for informative/debug purposes, not for keys
  String getType();
}

class PlcLatencyMetadataKey implements MetadataKey<Long> {
  String getType() { return "latency"; }
}


interface FieldMetadataKey<T> extends MetadataKey<T> {
  // shall we propagate field name/instance?
}

class PlcResponseCodeMetadataKey implements
FieldMetadataKey<PlcResponseCode> {
  String getType() { return "response_code"; }
}

class ReadTiemstampMetadataKey implements FieldMetadataKey<Long> {
  String getType() { return "read_timestamp"; }
}

class MachineTiemstampMetadataKey implements FieldMetadataKey<Long> {
  String getType() { return "machine_timestamp"; }
}

Best,
Łukasz

On 08.03.2021 23:15, Matthias Milan Strljic wrote:
> Hi all,
> 
> i see there a point of how useful this meta information could be in some
> scenarios. But i would rather go the route of chris. Because I see there is
> more of a risk of an overcomplicated inhomogeneous api structure. I would
> not say that all API should have the same java structure on each platform
> but if you have for all the different platforms other featuresets you could
> also just use partially the code generation attempt to allow a
> separate meta info layer.
> 
> And there would be the question if that is worth the additional feature?
> I mean I know we could add especially on the OPC UA side some meta
> information but is there more than the  actual source timestamp? And if
> there is none, would it be not better to use it as the actual timestamp of
> the value?
> 
> 
> Einen schönen Abend ;)
> Greetings Matthias
> 
> 
> 
> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
> christofer.dutz@c-ware.de>:
> 
>> Hi,
>>
>> I'm generally more concerned about users expecting us to deliver feature
>> that driver X has for driver Y too.
>>
>> I won't object, if you think it's worth doing.
>>
>> Perhaps if you could whip up an example in a feature branch? I think
>> perhaps I was still not understanding what you propose.
>>
>> Would that be ok?
>>
>>
>> Chris
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Lukas Ott <ot...@gmail.com>
>> Gesendet: Montag, 8. März 2021 14:20
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> +1 to Lukasz
>>
>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>> luke@code-house.org>:
>>
>>> Hold on for a second.
>>>
>>> Making options pushed over connection string is a long term recipe for
>>> disaster. Apache Camel is a prime example of what could happen if you
>>> starting with configuration with URIs and dynamic parameters. Most of
>>> components distinguish producer (writer) and consumer (receiver)
>>> options, most complicated components ended up with 40+ options
>>> settable via URI. Having a mix of PlcValues which are timestamped and
>> 'qualified'
>>> and not will complicate encoding logic (how we push above info to IEC
>>> encoder/handler or decorate it?) and how we keep custom types
>>> compatible with above?
>>>
>>> That is why I rather opt for extending API in a way which does not
>>> force existing drivers to do anything. When any of community members,
>>> regardless if its developer or user, will find it necessary to be
>>> ported in another driver, she or he have right to do it. It is not
>>> only yours duty to keep all drivers and languages the same.
>>>
>>> Speaking of which I am also not concerned about making all languages
>>> having the same functionality as languages tend to offer different
>>> options which might be hard to be ported. Lets stick to basics which
>>> are same (connection strings, field definitions, read/write/subscribe
>>> syntax) and let various runtime lead their own ways toward
>> implementation.
>>>
>>> Key point is simple - we won't be able to maintain all languages
>>> equally. We should not hold moving ie. go or java forward because we
>>> can't port new API functionality to C/Python/C#/whenever. If there is
>>> someone who *does* use above in production and *wish* to have new API
>>> parts then that person/organization duty is to do it.
>>>
>>> Best,
>>> Łukasz
>>>
>>>
>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>> Hi,
>>>>
>>>> I must admit that I would be in favor of keeping it as simple as
>>> possible (max age in the connection string) and to implement more of
>>> the missing parts in plc4x (like the subscription simulation layer)
>>> and hereby getting the drivers we have a bit more aligned, than to
>>> implement more and more special API features, that only one or two
>>> drivers support. Cause this way we're running away form the
>>> project-promise of running a given program at any type of PLC with any
>> protocol.
>>>>
>>>> I mean ... at the current velocity the project is underway, I
>>>> wouldn't
>>> really want to add a new MetaData layer that probably in the end only
>>> 1-2 people will be implementing and which we must not only find out
>>> how to fill with life in the first place, but also port to all of our
>>> supported languages.
>>>>
>>>> If someone's willing to bring in some serious man/womenpower, I'm
>>>> fine
>>> with that ... or we add it to some wish-list for future extensions.
>>> But I wouldn't want to start something like this right now.
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>> Gesendet: Montag, 8. März 2021 13:07
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Source Time of PLC Value?
>>>>
>>>> I have not mentioned metadata term in my answer, so credits for
>>>> mining
>>> it API go to you. :-)
>>>>
>>>> Now since you brought it I remember that JDBC has a "result set
>>> metadata". That might be thing we still miss. The result code is most
>>> important and available instantly, other information is supplemental
>>> and can be read additionally.
>>>>
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>> future.
>>>>>
>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
>>>>> <ch...@c-ware.de>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> sorry for being late to the party ... KNX is currently consuming
>>>>>> all my cycles.
>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>
>>>>>> Initially I thought about adding this sort of information to the
>>>>>> API, but then I thought that we have so little protocols
>>>>>> supporting this sort of concept that blowing up every PLCValue
>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>
>>>>>> When it comes to simulating subscriptions by polling in the
>>>>>> background, I thought it would be a valid solution to provide a
>>>>>> default age a value can have in the cache and to make it possible
>>>>>> to override this in the connection string options. If a value is
>>>>>> too old,
>>> a return code of:
>>>>>> STALE_DATA could be returned.
>>>>>>
>>>>>> Also, we do have an option of adding an additional set of PlcValue
>>> types.
>>>>>> If a driver supports this quality-of-service type of data, we
>>>>>> could return special versions of PlcValues, that have these
>> properties?
>>>>>>
>>>>>> This way we don't have to waste the CPU time and Memory for
>>>>>> protocols that don't support this concept (We could even implement
>>>>>> the methods on the default PlvValues to simply return some
>>>>>> constants to the PlcValues all have the same API.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>
>>>>>> Yet looking at present state of API we are unlikely to make each
>>>>>> and every PlcValue in each and every variant (qualified and
>>>>>> timestamped, timestamped, qualified). It would force quite a lot of
>> wrapping.
>>>>>> Safest option I see is to extend subscription/response API with
>>>>>> additional method to retrieve all markers. For most of drivers it
>>>>>> would return just an empty set. It leaves us also an open door for
>>>>>> future drivers (or devices) to ship additional piece of metadata
>>>>>> which
>>> does not fit into present API.
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>>
>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>
>>>>>>> uint64_t timestamp;
>>>>>>> typedef enum {
>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>       SOURCE,
>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>       GENERATED,
>>>>>>> } TimestampQuality;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> We do have some error codes, but it would just need to be
>>>>>>>> extended a bit. I don't think we have one for BAD (we still
>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>
>>>>>>>> Timestamps I think are definitely a good idea, especially if we
>>>>>>>> eventually DNP3.
>>>>>>>>
>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
>>>>>>>> <lu...@code-house.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You have response code for each individual field which allows
>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same
>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Łukasz
>>>>>>>>>
>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>> indicator of a plc value?
>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>
>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andreas
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>

AW: AW: AW: AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Lukas,

I hope you don't have the feeling that I'm blocking this ... I'm not ... if you want to scratch that itch, do it.

I'm just not going to add this to my list of things to do, which has become way too long anyway.

The only things I ask anyone working on this is, to not break things, not make things too complicated for normal users or to have a significant impact on performance.

Chris



-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Mittwoch, 10. März 2021 17:28
An: dev@plc4x.apache.org
Betreff: Re: AW: AW: AW: Source Time of PLC Value?

On technical side of things - addressing Chris and Otto comments:

> And coming back to your proposal: The way you are proposing to 
> implement this with some generic stuff ... it would make it difficult 
> to provide the same functionality on other languages. Right now, the 
> API is so simple that it is easy to port.

I see your point. Actual trap I was trying to solve with generics is basic - avoid usage of string keys, cause that is recipe for disaster.
This also let us to tie key and its value type. Without that we can easily fall into situation where one driver reports latency as Long, another as Duration and third one as int.
Having said that, I can't speak of other languages and their capabilities. I hope other platforms have own ways of enforcing type safety during compilation. I was trying to abstain from enums cause they can quickly be insufficient mechanism.

> I think it is a Good Thing™ to have.  My only ( possibly academic ) 
> concern is conflating data that comes from the device with data that 
> is generated or concerned with PLC4x’s workings.  I think a good 
> separation of concerns there would good to think about.

That's fairly good point which drives us into.. MetadataValue with "computed" flag or whatever else indicator we will find valid.
This is due to fact that metadata value can be delivered from different places still having same key. Unless we make key mutable or find another way which keeps API compact.

Chhhhers,
Łukasz


On 10.03.2021 15:30, Christofer Dutz wrote:
> Hi Lukaz,
> 
> I would say: If there's a NEED from the industry. They could show how much they need it, by spending a few bucks on such features I would call enterprise features (I'm not talking about paying for the full implementation, but at least part of it). And especially, we shouldn't be guessing for them, what they might be needing. If they want something, they should speak up and tell us what they really need.
> 
> Personally, I have stopped thinking what the industry might be needing as the industry never started thinking about what the people behind an open-source project might be needing (Being able to talk about what they did or paid gigs, for example). I know of several cases where big companies would love to have something. Like I know that Siemens would love to have an ADS driver in C ... I also know about several other cases where companies would love to have certain drivers in certain languages. But staying at the Siemens example: As long as I have to pay 3000€ for a TIA portal license and there's just no way to provide me with a free (educational) license or there's not a single € in the budget for a paid gig ... this will be the last driver version I'll work on. 
> 
> For this reason, I will not continue implementing such enterprise features until I can see the industry openly admitting that they are using PLC4X or need something and that at least some of our community members can at least pay some of their bills from their hard work on this great project (not talking about me here).
> 
> And coming back to your proposal: The way you are proposing to 
> implement this with some generic stuff ... it would make it difficult 
> to provide the same functionality on other languages. Right now, the 
> API is so simple that it is easy to port. If you really want to tie me 
> to when I say: "Sort of like JDBC for PLCs" and because of that want 
> to have the MetaData - then consistently we should be allowed to run 
> Stored Procedures on PLCs, Have a query language that allows data 
> joining and aggregation as well as mathematical operations, have 
> cursors for large result sets, also should we be able to use PLC4X to 
> open terminals on PLCs and execute arbitrary code on the remote 
> device. Cause this is all stuff I have been doing using JDBC ;-)
> 
> Chris
>  
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org>
> Gesendet: Mittwoch, 10. März 2021 15:05
> An: dev@plc4x.apache.org
> Betreff: Re: AW: AW: Source Time of PLC Value?
> 
> Chris,
> What we discuss is whether there is a need and use for result set 
> metadata or not. Given initial statement of "being a for of JDBC for 
> industrial devices" and looking at JDBC itself we should provide it. 
> If we don't do it then we lave our OPC UA users with unanswered 
> trouble which they need to handle on their own. If we postpone it 
> until implementation need arise in non-opc drivers (let say 
> Ethernet/IP,
> AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use even standard UA capabilities.
> 
> I do agree that simplicity is a key, however what you compare are two completely different things. MQTT is an thin overlay over TCP transport without any real application layer (it has even transport in the name).
> OPC UA does both with much more diverse application layer causing much bigger overhead for both ends of communications. Both MQTT and OPC UA bring completely different approach and create problems in distinct places.
> 
> Neither I nor anybody else want to sacrifice time on pull request which will not be accepted.
> 
> Kind regards,
> Łukasz
> 
> 
> On 10.03.2021 14:34, Christofer Dutz wrote:
>> Yes,
>>
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>>
>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>>
>> Just my thoughts on this.
>>
>> Chris
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
>> / 
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/d
>> e
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> / 
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/p
>> l
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> / 
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/p
>> l c4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I 
>>>> think perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only 
>>>>> one or two drivers support. Cause this way we're running away form 
>>>>> the project-promise of running a given program at any type of PLC 
>>>>> with any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find 
>>>>> out how to fill with life in the first place, but also port to all 
>>>>> of our supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently 
>>>>>>>> consuming all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information to 
>>>>>>>> the API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If 
>>>>>>>> a value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make 
>>>>>>>> each and every PlcValue in each and every variant (qualified 
>>>>>>>> and timestamped, timestamped, qualified). It would force quite 
>>>>>>>> a lot of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece 
>>>>>>>> of metadata which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson 
>>>>>>>>>> <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field which 
>>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>

Re: AW: AW: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

On technical side of things - addressing Chris and Otto comments:

> And coming back to your proposal: The way you are proposing to
> implement this with some generic stuff ... it would make it difficult
> to provide the same functionality on other languages. Right now, the
> API is so simple that it is easy to port.

I see your point. Actual trap I was trying to solve with generics is
basic - avoid usage of string keys, cause that is recipe for disaster.
This also let us to tie key and its value type. Without that we can
easily fall into situation where one driver reports latency as Long,
another as Duration and third one as int.
Having said that, I can't speak of other languages and their
capabilities. I hope other platforms have own ways of enforcing type
safety during compilation. I was trying to abstain from enums cause they
can quickly be insufficient mechanism.

> I think it is a Good Thing™ to have.  My only ( possibly academic )
> concern is conflating data that comes from the device with data that
> is generated or concerned with PLC4x’s workings.  I think a good
> separation of concerns there would good to think about.

That's fairly good point which drives us into.. MetadataValue with
"computed" flag or whatever else indicator we will find valid.
This is due to fact that metadata value can be delivered from different
places still having same key. Unless we make key mutable or find another
way which keeps API compact.

Chhhhers,
Łukasz


On 10.03.2021 15:30, Christofer Dutz wrote:
> Hi Lukaz,
> 
> I would say: If there's a NEED from the industry. They could show how much they need it, by spending a few bucks on such features I would call enterprise features (I'm not talking about paying for the full implementation, but at least part of it). And especially, we shouldn't be guessing for them, what they might be needing. If they want something, they should speak up and tell us what they really need.
> 
> Personally, I have stopped thinking what the industry might be needing as the industry never started thinking about what the people behind an open-source project might be needing (Being able to talk about what they did or paid gigs, for example). I know of several cases where big companies would love to have something. Like I know that Siemens would love to have an ADS driver in C ... I also know about several other cases where companies would love to have certain drivers in certain languages. But staying at the Siemens example: As long as I have to pay 3000€ for a TIA portal license and there's just no way to provide me with a free (educational) license or there's not a single € in the budget for a paid gig ... this will be the last driver version I'll work on. 
> 
> For this reason, I will not continue implementing such enterprise features until I can see the industry openly admitting that they are using PLC4X or need something and that at least some of our community members can at least pay some of their bills from their hard work on this great project (not talking about me here).
> 
> And coming back to your proposal: The way you are proposing to implement this with some generic stuff ... it would make it difficult to provide the same functionality on other languages. Right now, the API is so simple that it is easy to port. If you really want to tie me to when I say: "Sort of like JDBC for PLCs" and because of that want to have the MetaData - then consistently we should be allowed to run Stored Procedures on PLCs, Have a query language that allows data joining and aggregation as well as mathematical operations, have cursors for large result sets, also should we be able to use PLC4X to open terminals on PLCs and execute arbitrary code on the remote device. Cause this is all stuff I have been doing using JDBC ;-) 
> 
> Chris
>  
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org> 
> Gesendet: Mittwoch, 10. März 2021 15:05
> An: dev@plc4x.apache.org
> Betreff: Re: AW: AW: Source Time of PLC Value?
> 
> Chris,
> What we discuss is whether there is a need and use for result set metadata or not. Given initial statement of "being a for of JDBC for industrial devices" and looking at JDBC itself we should provide it. If we don't do it then we lave our OPC UA users with unanswered trouble which they need to handle on their own. If we postpone it until implementation need arise in non-opc drivers (let say Ethernet/IP,
> AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use even standard UA capabilities.
> 
> I do agree that simplicity is a key, however what you compare are two completely different things. MQTT is an thin overlay over TCP transport without any real application layer (it has even transport in the name).
> OPC UA does both with much more diverse application layer causing much bigger overhead for both ends of communications. Both MQTT and OPC UA bring completely different approach and create problems in distinct places.
> 
> Neither I nor anybody else want to sacrifice time on pull request which will not be accepted.
> 
> Kind regards,
> Łukasz
> 
> 
> On 10.03.2021 14:34, Christofer Dutz wrote:
>> Yes,
>>
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>>
>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>>
>> Just my thoughts on this.
>>
>> Chris
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
>> c4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I think 
>>>> perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation 
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only one 
>>>>> or two drivers support. Cause this way we're running away form the 
>>>>> project-promise of running a given program at any type of PLC with 
>>>>> any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find out 
>>>>> how to fill with life in the first place, but also port to all of 
>>>>> our supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>>> all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If a 
>>>>>>>> value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>>> of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece of 
>>>>>>>> metadata which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>

Re: AW: AW: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Hey Chris, hope others still follow this thread!

Whether its fault of tight pockets of industry, decade long production
cycles or something else we can't wait indefinitely for them. Maybe,
just maybe OPC UA will become WSDL/SOAP of industry over time. Slow,
over-bloated, painful in development but in the end common data exchange
layer. (After several cycles of WebService Interoperability/WS-I releases!)

I will try to answer in a bit friday-ish attitude to address the JDBC
argument. The stored procedures and other not-so-great things JDBC
brings into table are in their API because they are in use. Maybe not
always, not by everyone but enough of times to be there. This also keeps
focus of database vendors to stick with it cause it saves them from
doing whole new abstraction layer just to cover own product features.
We are leaving space for custom types in our API/SPI. We also do
optimizations on reading data (S7 batching). We do cover some of JDBC
features even if they are named differently. I could also argue if our
field expressions are less or more complicated than basic SQL selects. ;-)

Regardless of industry or big pharma paying or not we are the ones who
lead the edge. So far mainly serving our small goals. Not the others
cause they are gone and quiet.
Apparently Ben have clear intention to ship better support for OPC UA.
He is paying by his own time to do such. I feel myself obligated to help
him cause I might be another person who will indirectly benefit from his
work.

Best,
Łukasz

On 10.03.2021 15:30, Christofer Dutz wrote:
> Hi Lukaz,
> 
> I would say: If there's a NEED from the industry. They could show how much they need it, by spending a few bucks on such features I would call enterprise features (I'm not talking about paying for the full implementation, but at least part of it). And especially, we shouldn't be guessing for them, what they might be needing. If they want something, they should speak up and tell us what they really need.
> 
> Personally, I have stopped thinking what the industry might be needing as the industry never started thinking about what the people behind an open-source project might be needing (Being able to talk about what they did or paid gigs, for example). I know of several cases where big companies would love to have something. Like I know that Siemens would love to have an ADS driver in C ... I also know about several other cases where companies would love to have certain drivers in certain languages. But staying at the Siemens example: As long as I have to pay 3000€ for a TIA portal license and there's just no way to provide me with a free (educational) license or there's not a single € in the budget for a paid gig ... this will be the last driver version I'll work on. 
> 
> For this reason, I will not continue implementing such enterprise features until I can see the industry openly admitting that they are using PLC4X or need something and that at least some of our community members can at least pay some of their bills from their hard work on this great project (not talking about me here).
> 
> And coming back to your proposal: The way you are proposing to implement this with some generic stuff ... it would make it difficult to provide the same functionality on other languages. Right now, the API is so simple that it is easy to port. If you really want to tie me to when I say: "Sort of like JDBC for PLCs" and because of that want to have the MetaData - then consistently we should be allowed to run Stored Procedures on PLCs, Have a query language that allows data joining and aggregation as well as mathematical operations, have cursors for large result sets, also should we be able to use PLC4X to open terminals on PLCs and execute arbitrary code on the remote device. Cause this is all stuff I have been doing using JDBC ;-) 
> 
> Chris
>  
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org> 
> Gesendet: Mittwoch, 10. März 2021 15:05
> An: dev@plc4x.apache.org
> Betreff: Re: AW: AW: Source Time of PLC Value?
> 
> Chris,
> What we discuss is whether there is a need and use for result set metadata or not. Given initial statement of "being a for of JDBC for industrial devices" and looking at JDBC itself we should provide it. If we don't do it then we lave our OPC UA users with unanswered trouble which they need to handle on their own. If we postpone it until implementation need arise in non-opc drivers (let say Ethernet/IP,
> AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use even standard UA capabilities.
> 
> I do agree that simplicity is a key, however what you compare are two completely different things. MQTT is an thin overlay over TCP transport without any real application layer (it has even transport in the name).
> OPC UA does both with much more diverse application layer causing much bigger overhead for both ends of communications. Both MQTT and OPC UA bring completely different approach and create problems in distinct places.
> 
> Neither I nor anybody else want to sacrifice time on pull request which will not be accepted.
> 
> Kind regards,
> Łukasz
> 
> 
> On 10.03.2021 14:34, Christofer Dutz wrote:
>> Yes,
>>
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>>
>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>>
>> Just my thoughts on this.
>>
>> Chris
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
>> c4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I think 
>>>> perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation 
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only one 
>>>>> or two drivers support. Cause this way we're running away form the 
>>>>> project-promise of running a given program at any type of PLC with 
>>>>> any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find out 
>>>>> how to fill with life in the first place, but also port to all of 
>>>>> our supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>>> all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If a 
>>>>>>>> value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>>> of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece of 
>>>>>>>> metadata which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>

AW: AW: AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi Lukaz,

I would say: If there's a NEED from the industry. They could show how much they need it, by spending a few bucks on such features I would call enterprise features (I'm not talking about paying for the full implementation, but at least part of it). And especially, we shouldn't be guessing for them, what they might be needing. If they want something, they should speak up and tell us what they really need.

Personally, I have stopped thinking what the industry might be needing as the industry never started thinking about what the people behind an open-source project might be needing (Being able to talk about what they did or paid gigs, for example). I know of several cases where big companies would love to have something. Like I know that Siemens would love to have an ADS driver in C ... I also know about several other cases where companies would love to have certain drivers in certain languages. But staying at the Siemens example: As long as I have to pay 3000€ for a TIA portal license and there's just no way to provide me with a free (educational) license or there's not a single € in the budget for a paid gig ... this will be the last driver version I'll work on. 

For this reason, I will not continue implementing such enterprise features until I can see the industry openly admitting that they are using PLC4X or need something and that at least some of our community members can at least pay some of their bills from their hard work on this great project (not talking about me here).

And coming back to your proposal: The way you are proposing to implement this with some generic stuff ... it would make it difficult to provide the same functionality on other languages. Right now, the API is so simple that it is easy to port. If you really want to tie me to when I say: "Sort of like JDBC for PLCs" and because of that want to have the MetaData - then consistently we should be allowed to run Stored Procedures on PLCs, Have a query language that allows data joining and aggregation as well as mathematical operations, have cursors for large result sets, also should we be able to use PLC4X to open terminals on PLCs and execute arbitrary code on the remote device. Cause this is all stuff I have been doing using JDBC ;-) 

Chris
 

-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Mittwoch, 10. März 2021 15:05
An: dev@plc4x.apache.org
Betreff: Re: AW: AW: Source Time of PLC Value?

Chris,
What we discuss is whether there is a need and use for result set metadata or not. Given initial statement of "being a for of JDBC for industrial devices" and looking at JDBC itself we should provide it. If we don't do it then we lave our OPC UA users with unanswered trouble which they need to handle on their own. If we postpone it until implementation need arise in non-opc drivers (let say Ethernet/IP,
AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use even standard UA capabilities.

I do agree that simplicity is a key, however what you compare are two completely different things. MQTT is an thin overlay over TCP transport without any real application layer (it has even transport in the name).
OPC UA does both with much more diverse application layer causing much bigger overhead for both ends of communications. Both MQTT and OPC UA bring completely different approach and create problems in distinct places.

Neither I nor anybody else want to sacrifice time on pull request which will not be accepted.

Kind regards,
Łukasz


On 10.03.2021 14:34, Christofer Dutz wrote:
> Yes,
> 
> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
> 
> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
> 
> Just my thoughts on this.
> 
> Chris
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org>
> Gesendet: Mittwoch, 10. März 2021 12:58
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Source Time of PLC Value?
> 
> Matthias,
> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
> 
> Two major points I see:
> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
> 
> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
> 
> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
> 
> Best,
> Łukasz
> 
> [1]
> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
> corator
> [2]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
> c4x/decorator/retry/ReadRetry.java
> [3]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
> c4x/decorator/retry/RetryDecoratorReadTest.java
> 
> 
> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>> Hi all,
>>
>> i see there a point of how useful this meta information could be in 
>> some scenarios. But i would rather go the route of chris. Because I 
>> see there is more of a risk of an overcomplicated inhomogeneous api 
>> structure. I would not say that all API should have the same java 
>> structure on each platform but if you have for all the different 
>> platforms other featuresets you could also just use partially the 
>> code generation attempt to allow a separate meta info layer.
>>
>> And there would be the question if that is worth the additional feature?
>> I mean I know we could add especially on the OPC UA side some meta 
>> information but is there more than the  actual source timestamp? And 
>> if there is none, would it be not better to use it as the actual 
>> timestamp of the value?
>>
>>
>> Einen schönen Abend ;)
>> Greetings Matthias
>>
>>
>>
>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>> christofer.dutz@c-ware.de>:
>>
>>> Hi,
>>>
>>> I'm generally more concerned about users expecting us to deliver 
>>> feature that driver X has for driver Y too.
>>>
>>> I won't object, if you think it's worth doing.
>>>
>>> Perhaps if you could whip up an example in a feature branch? I think 
>>> perhaps I was still not understanding what you propose.
>>>
>>> Would that be ok?
>>>
>>>
>>> Chris
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Lukas Ott <ot...@gmail.com>
>>> Gesendet: Montag, 8. März 2021 14:20
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>>
>>> +1 to Lukasz
>>>
>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>> luke@code-house.org>:
>>>
>>>> Hold on for a second.
>>>>
>>>> Making options pushed over connection string is a long term recipe 
>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>> if you starting with configuration with URIs and dynamic parameters.
>>>> Most of components distinguish producer (writer) and consumer
>>>> (receiver) options, most complicated components ended up with 40+ 
>>>> options settable via URI. Having a mix of PlcValues which are 
>>>> timestamped and
>>> 'qualified'
>>>> and not will complicate encoding logic (how we push above info to 
>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>> compatible with above?
>>>>
>>>> That is why I rather opt for extending API in a way which does not 
>>>> force existing drivers to do anything. When any of community 
>>>> members, regardless if its developer or user, will find it 
>>>> necessary to be ported in another driver, she or he have right to 
>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>
>>>> Speaking of which I am also not concerned about making all 
>>>> languages having the same functionality as languages tend to offer 
>>>> different options which might be hard to be ported. Lets stick to 
>>>> basics which are same (connection strings, field definitions, 
>>>> read/write/subscribe
>>>> syntax) and let various runtime lead their own ways toward
>>> implementation.
>>>>
>>>> Key point is simple - we won't be able to maintain all languages 
>>>> equally. We should not hold moving ie. go or java forward because 
>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>> there is someone who *does* use above in production and *wish* to 
>>>> have new API parts then that person/organization duty is to do it.
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>>
>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>> Hi,
>>>>>
>>>>> I must admit that I would be in favor of keeping it as simple as
>>>> possible (max age in the connection string) and to implement more 
>>>> of the missing parts in plc4x (like the subscription simulation 
>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>> than to implement more and more special API features, that only one 
>>>> or two drivers support. Cause this way we're running away form the 
>>>> project-promise of running a given program at any type of PLC with 
>>>> any
>>> protocol.
>>>>>
>>>>> I mean ... at the current velocity the project is underway, I 
>>>>> wouldn't
>>>> really want to add a new MetaData layer that probably in the end 
>>>> only
>>>> 1-2 people will be implementing and which we must not only find out 
>>>> how to fill with life in the first place, but also port to all of 
>>>> our supported languages.
>>>>>
>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>> fine
>>>> with that ... or we add it to some wish-list for future extensions.
>>>> But I wouldn't want to start something like this right now.
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>
>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>> mining
>>>> it API go to you. :-)
>>>>>
>>>>> Now since you brought it I remember that JDBC has a "result set
>>>> metadata". That might be thing we still miss. The result code is 
>>>> most important and available instantly, other information is 
>>>> supplemental and can be read additionally.
>>>>>
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>> +create
>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>> the
>>>> future.
>>>>>>
>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>> <ch...@c-ware.de>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>> all my cycles.
>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>
>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>
>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>> default age a value can have in the cache and to make it 
>>>>>>> possible to override this in the connection string options. If a 
>>>>>>> value is too old,
>>>> a return code of:
>>>>>>> STALE_DATA could be returned.
>>>>>>>
>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>> PlcValue
>>>> types.
>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>> could return special versions of PlcValues, that have these
>>> properties?
>>>>>>>
>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>> protocols that don't support this concept (We could even 
>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>>
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>
>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>
>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>> of
>>> wrapping.
>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>> door for future drivers (or devices) to ship additional piece of 
>>>>>>> metadata which
>>>> does not fit into present API.
>>>>>>>
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>>
>>>>>>>
>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>
>>>>>>>> uint64_t timestamp;
>>>>>>>> typedef enum {
>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>       SOURCE,
>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>       GENERATED,
>>>>>>>> } TimestampQuality;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>
>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>> we eventually DNP3.
>>>>>>>>>
>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>> <lu...@code-house.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Łukasz
>>>>>>>>>>
>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>
>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>
>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Andreas
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>

Re: AW: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Chris,
What we discuss is whether there is a need and use for result set
metadata or not. Given initial statement of "being a for of JDBC for
industrial devices" and looking at JDBC itself we should provide it. If
we don't do it then we lave our OPC UA users with unanswered trouble
which they need to handle on their own. If we postpone it until
implementation need arise in non-opc drivers (let say Ethernet/IP,
AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use
even standard UA capabilities.

I do agree that simplicity is a key, however what you compare are two
completely different things. MQTT is an thin overlay over TCP transport
without any real application layer (it has even transport in the name).
OPC UA does both with much more diverse application layer causing much
bigger overhead for both ends of communications. Both MQTT and OPC UA
bring completely different approach and create problems in distinct places.

Neither I nor anybody else want to sacrifice time on pull request which
will not be accepted.

Kind regards,
Łukasz


On 10.03.2021 14:34, Christofer Dutz wrote:
> Yes,
> 
> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
> 
> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
> 
> Just my thoughts on this.
> 
> Chris
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org> 
> Gesendet: Mittwoch, 10. März 2021 12:58
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Source Time of PLC Value?
> 
> Matthias,
> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
> 
> Two major points I see:
> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
> 
> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
> 
> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
> 
> Best,
> Łukasz
> 
> [1]
> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
> [2]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
> [3]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java
> 
> 
> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>> Hi all,
>>
>> i see there a point of how useful this meta information could be in 
>> some scenarios. But i would rather go the route of chris. Because I 
>> see there is more of a risk of an overcomplicated inhomogeneous api 
>> structure. I would not say that all API should have the same java 
>> structure on each platform but if you have for all the different 
>> platforms other featuresets you could also just use partially the code 
>> generation attempt to allow a separate meta info layer.
>>
>> And there would be the question if that is worth the additional feature?
>> I mean I know we could add especially on the OPC UA side some meta 
>> information but is there more than the  actual source timestamp? And 
>> if there is none, would it be not better to use it as the actual 
>> timestamp of the value?
>>
>>
>> Einen schönen Abend ;)
>> Greetings Matthias
>>
>>
>>
>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>> christofer.dutz@c-ware.de>:
>>
>>> Hi,
>>>
>>> I'm generally more concerned about users expecting us to deliver 
>>> feature that driver X has for driver Y too.
>>>
>>> I won't object, if you think it's worth doing.
>>>
>>> Perhaps if you could whip up an example in a feature branch? I think 
>>> perhaps I was still not understanding what you propose.
>>>
>>> Would that be ok?
>>>
>>>
>>> Chris
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Lukas Ott <ot...@gmail.com>
>>> Gesendet: Montag, 8. März 2021 14:20
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>>
>>> +1 to Lukasz
>>>
>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>> luke@code-house.org>:
>>>
>>>> Hold on for a second.
>>>>
>>>> Making options pushed over connection string is a long term recipe 
>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>> if you starting with configuration with URIs and dynamic parameters. 
>>>> Most of components distinguish producer (writer) and consumer 
>>>> (receiver) options, most complicated components ended up with 40+ 
>>>> options settable via URI. Having a mix of PlcValues which are 
>>>> timestamped and
>>> 'qualified'
>>>> and not will complicate encoding logic (how we push above info to 
>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>> compatible with above?
>>>>
>>>> That is why I rather opt for extending API in a way which does not 
>>>> force existing drivers to do anything. When any of community 
>>>> members, regardless if its developer or user, will find it necessary 
>>>> to be ported in another driver, she or he have right to do it. It is 
>>>> not only yours duty to keep all drivers and languages the same.
>>>>
>>>> Speaking of which I am also not concerned about making all languages 
>>>> having the same functionality as languages tend to offer different 
>>>> options which might be hard to be ported. Lets stick to basics which 
>>>> are same (connection strings, field definitions, 
>>>> read/write/subscribe
>>>> syntax) and let various runtime lead their own ways toward
>>> implementation.
>>>>
>>>> Key point is simple - we won't be able to maintain all languages 
>>>> equally. We should not hold moving ie. go or java forward because we 
>>>> can't port new API functionality to C/Python/C#/whenever. If there 
>>>> is someone who *does* use above in production and *wish* to have new 
>>>> API parts then that person/organization duty is to do it.
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>>
>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>> Hi,
>>>>>
>>>>> I must admit that I would be in favor of keeping it as simple as
>>>> possible (max age in the connection string) and to implement more of 
>>>> the missing parts in plc4x (like the subscription simulation layer) 
>>>> and hereby getting the drivers we have a bit more aligned, than to 
>>>> implement more and more special API features, that only one or two 
>>>> drivers support. Cause this way we're running away form the 
>>>> project-promise of running a given program at any type of PLC with 
>>>> any
>>> protocol.
>>>>>
>>>>> I mean ... at the current velocity the project is underway, I 
>>>>> wouldn't
>>>> really want to add a new MetaData layer that probably in the end 
>>>> only
>>>> 1-2 people will be implementing and which we must not only find out 
>>>> how to fill with life in the first place, but also port to all of 
>>>> our supported languages.
>>>>>
>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>> fine
>>>> with that ... or we add it to some wish-list for future extensions.
>>>> But I wouldn't want to start something like this right now.
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>
>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>> mining
>>>> it API go to you. :-)
>>>>>
>>>>> Now since you brought it I remember that JDBC has a "result set
>>>> metadata". That might be thing we still miss. The result code is 
>>>> most important and available instantly, other information is 
>>>> supplemental and can be read additionally.
>>>>>
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>>> future.
>>>>>>
>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>> <ch...@c-ware.de>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>> all my cycles.
>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>
>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>
>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>> default age a value can have in the cache and to make it possible 
>>>>>>> to override this in the connection string options. If a value is 
>>>>>>> too old,
>>>> a return code of:
>>>>>>> STALE_DATA could be returned.
>>>>>>>
>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>> PlcValue
>>>> types.
>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>> could return special versions of PlcValues, that have these
>>> properties?
>>>>>>>
>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>> protocols that don't support this concept (We could even 
>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>>
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>
>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>
>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>> of
>>> wrapping.
>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>> additional method to retrieve all markers. For most of drivers it 
>>>>>>> would return just an empty set. It leaves us also an open door 
>>>>>>> for future drivers (or devices) to ship additional piece of 
>>>>>>> metadata which
>>>> does not fit into present API.
>>>>>>>
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>>
>>>>>>>
>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>
>>>>>>>> uint64_t timestamp;
>>>>>>>> typedef enum {
>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>       SOURCE,
>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>       GENERATED,
>>>>>>>> } TimestampQuality;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>
>>>>>>>>> Timestamps I think are definitely a good idea, especially if we 
>>>>>>>>> eventually DNP3.
>>>>>>>>>
>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>> <lu...@code-house.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Łukasz
>>>>>>>>>>
>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>
>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>
>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Andreas
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>

Re: Source Time of PLC Value?

Posted by Chris Misztur <cm...@mriiot.com>.

I would like to comment on the simplicity of MQTT.  It does indeed simplify
creating the pipe between endpoints, but the problem of data opacity
remains.  I believe that premature implementation of MQTT everywhere and to
everything will do more harm than good, because we will naturally "invent"
our own topic structures and payloads.  Maybe the correct approach is to
continue pushing proprietary protocols over MQTT for distribution
purposes?  I also think that departure from XML to JSON has been premature;
the richness of expression is lost with JSON.

Just my thoughts for the day :)

*Chris*


On Wed, Mar 10, 2021 at 9:22 AM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi Andreas,
>
> Glad I didn't upset you too much ;-)
>
> And I guess I'm also a bit pissed about OPC-UA in general. At first, I had
> such a hard time selling PLC4X as OPC-UA was promising so much, but without
> proof, that almost nobody gave open-source and PLC4X a chance. So much
> money was burnt in 2018 and 2019 with unsuccessful OPC-UA projects by the
> big automation companies.
>
> Then end of 2019 some of the ones, that decided to go the OPC-UA path,
> started contacting me again. But when we wanted to discuss terms, they
> sometimes had the nerve to say something like: We wasted so much money on
> OPC-UA and our budget is almost depleted ... we can only hire your company
> if you greatly reduce the hourly rate. Guess what my answer was? ;-)
>
> Chris
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Andreas Vogler <an...@me.com.INVALID>
> Gesendet: Mittwoch, 10. März 2021 16:06
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
>
> No problem, I think I understand what you mean… and I agree on the device
> level OPC UA is maybe too much / to fat (or not well implemented :-))
>
>
>
> > On 10.03.2021, at 15:50, Christofer Dutz <ch...@c-ware.de>
> wrote:
> >
> > Hi Andreas,
> >
> > I didn't want to upset you. I should have been a bit more clear on this:
> >
> > In 2018, when I was travelling through the country proclaiming PLC4X,
> all I heard was: But OPC-UA can do all of this (Without having tested it)
> ... 2019 most of the people who tried it were already looking for
> alternatives. End of 2019 and beginning of 2020 was definitely the age of
> MQTT (Of course OPC UA now provided a MQTT-like communication form).
> >
> > I think OPC-UA is great for machine-to-machine communication and stuff
> like dashboards. But as soon as you want to do high volume data
> acquisition. Especially for use cases like machine-learning etc. the
> processing power of even modern PLCs can't handle the load.
> >
> > For example, at one of the biggest German car manufacturers, they were
> not able to get more than 200 datapoints every 2 seconds from a new S7 400
> PLC via OPC-UA. These are probably the most powerful ones Siemens has to
> offer. If they requested 300, the PLC shut down the IO process due to
> overload. With PLC4X using the S7 protocol, we were able to get 2600
> datapoints in 200ms from the same machines.
> >
> > On the client side on one of the standard VMs they were using, running
> the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints.
> Then the CPU power was 100% ... in our case we run PLC4X on the same VMs
> using S7 and there we could collect 200 PLCs with 2600 datapoints. So also
> on the client side OPC-UA is a very fat project, that requires a lot of CPU
> time.
> >
> > I hope this explains things a bit more. If you are just reading a few
> datapoints or using OPC-UA for Server-to-Server communication the servers
> will be able to cope with more. And if it doesn't you just add more nodes
> to the compute cluster.
> >
> > Chris
> >
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Andreas Vogler <an...@me.com.INVALID>
> > Gesendet: Mittwoch, 10. März 2021 15:38
> > An: dev@plc4x.apache.org
> > Betreff: Re: Source Time of PLC Value?
> > Priorität: Niedrig
> >
> > Just a side note: it hurts to hear that OPC UA sucks badly
> > performance-wise. I have done tests with Milo OPC UA Client and SCADA
> > OPC UA servers, and they perform pretty fine. 200 MQTT clients writing
> > values via a Gateway with OPC UA Milo (Client) to two OPC UA servers,
> > each OPC UA server handles 50000 values changes per second:
> > https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-v
> > ertx-activity-6769029730797719552-ZkjG
> > <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-
> > vertx-activity-6769029730797719552-ZkjG>
> >
> > But I fully agree with MQTT - it’s simple - and I think that’s the
> > reason why it is so popular…
> >
> >
> >> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de>
> wrote:
> >>
> >> Yes,
> >>
> >> OPC-UA has all of this ... and it's probably one of the reasons so many
> implementations differ, and you have to again interpret results. It's also
> probably a reason why OPC-UA sucks that badly performance-wise.
> >>
> >> Just saying that we should be careful what we choose as reference. In
> contrast typical MQTT doesn't have all of this and it seems to be gaining
> more and more traction, just because it doesn't have all of this overhead.
> >>
> >> Just my thoughts on this.
> >>
> >> Chris
> >>
> >>
> >>
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Łukasz Dywicki <lu...@code-house.org>
> >> Gesendet: Mittwoch, 10. März 2021 12:58
> >> An: dev@plc4x.apache.org
> >> Betreff: Re: AW: Source Time of PLC Value?
> >>
> >> Matthias,
> >> I do believe there are several points which are relevant and still not
> possible to be implemented using available PLC4X API. Metadata might not be
> most fortunate way, but it can enable or ease analytical and
> troubleshooting scenarios.
> >>
> >> Two major points I see:
> >> - Verification of device condition - ie. how does caller know response
> latency? This is especially relevant with frequent polling. Currently
> caller has no measures to know that and can not measure that in reliable
> way due to async nature of API.
> >> There is a way to implement backpressure technics with callback chains
> (we can automatically delay execute call), but first we need to bring
> measures for that.
> >> - Caching of values - when we do cyclic subscriptions or emulation of
> these through passive connections. There is no way to make caller code
> aware of nature of data received. Unless he code everything on his end.
> >> - Per operation timeout settings/automatic retry are another
> functionalities which are relevant in case of querying and writing. We do
> not want every driver to implement it, so making a metadata and/or
> decorator would unify this layer between drivers.
> >>
> >> I speak of above from my own perspective cause I did implement an
> experimental decorator API for PLC4X [1] to cope with CANopen arbitration
> issues I found in devices we interface with. It allows to catch read/write
> failures [2] and re-attempt entire operation. It is proven to work [3]
> independently of driver. Sadly it has no measures to signal its execution
> back to caller. In the end, traceability is pretty tough.
> >>
> >> Obviously OPC UA has everything sorted out, but device specific
> protocols often do not. Even a limited set of metadata can give us a way to
> start bridging the gap and making Ben's Milo/OPC UA server much closer to a
> real thing.
> >>
> >> Best,
> >> Łukasz
> >>
> >> [1]
> >> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
> >> /
> >> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/d
> >> e
> >> corator
> >> [2]
> >> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
> >> /
> >> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/p
> >> l
> >> c4x/decorator/retry/ReadRetry.java
> >> [3]
> >> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
> >> /
> >> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/p
> >> l c4x/decorator/retry/RetryDecoratorReadTest.java
> >>
> >>
> >> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
> >>> Hi all,
> >>>
> >>> i see there a point of how useful this meta information could be in
> >>> some scenarios. But i would rather go the route of chris. Because I
> >>> see there is more of a risk of an overcomplicated inhomogeneous api
> >>> structure. I would not say that all API should have the same java
> >>> structure on each platform but if you have for all the different
> >>> platforms other featuresets you could also just use partially the
> >>> code generation attempt to allow a separate meta info layer.
> >>>
> >>> And there would be the question if that is worth the additional
> feature?
> >>> I mean I know we could add especially on the OPC UA side some meta
> >>> information but is there more than the  actual source timestamp? And
> >>> if there is none, would it be not better to use it as the actual
> >>> timestamp of the value?
> >>>
> >>>
> >>> Einen schönen Abend ;)
> >>> Greetings Matthias
> >>>
> >>>
> >>>
> >>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
> >>> christofer.dutz@c-ware.de>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'm generally more concerned about users expecting us to deliver
> >>>> feature that driver X has for driver Y too.
> >>>>
> >>>> I won't object, if you think it's worth doing.
> >>>>
> >>>> Perhaps if you could whip up an example in a feature branch? I
> >>>> think perhaps I was still not understanding what you propose.
> >>>>
> >>>> Would that be ok?
> >>>>
> >>>>
> >>>> Chris
> >>>>
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Lukas Ott <ot...@gmail.com>
> >>>> Gesendet: Montag, 8. März 2021 14:20
> >>>> An: dev@plc4x.apache.org
> >>>> Betreff: Re: AW: Source Time of PLC Value?
> >>>>
> >>>> +1 to Lukasz
> >>>>
> >>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
> >>>> luke@code-house.org>:
> >>>>
> >>>>> Hold on for a second.
> >>>>>
> >>>>> Making options pushed over connection string is a long term recipe
> >>>>> for disaster. Apache Camel is a prime example of what could happen
> >>>>> if you starting with configuration with URIs and dynamic parameters.
> >>>>> Most of components distinguish producer (writer) and consumer
> >>>>> (receiver) options, most complicated components ended up with 40+
> >>>>> options settable via URI. Having a mix of PlcValues which are
> >>>>> timestamped and
> >>>> 'qualified'
> >>>>> and not will complicate encoding logic (how we push above info to
> >>>>> IEC encoder/handler or decorate it?) and how we keep custom types
> >>>>> compatible with above?
> >>>>>
> >>>>> That is why I rather opt for extending API in a way which does not
> >>>>> force existing drivers to do anything. When any of community
> >>>>> members, regardless if its developer or user, will find it
> >>>>> necessary to be ported in another driver, she or he have right to
> >>>>> do it. It is not only yours duty to keep all drivers and languages
> the same.
> >>>>>
> >>>>> Speaking of which I am also not concerned about making all
> >>>>> languages having the same functionality as languages tend to offer
> >>>>> different options which might be hard to be ported. Lets stick to
> >>>>> basics which are same (connection strings, field definitions,
> >>>>> read/write/subscribe
> >>>>> syntax) and let various runtime lead their own ways toward
> >>>> implementation.
> >>>>>
> >>>>> Key point is simple - we won't be able to maintain all languages
> >>>>> equally. We should not hold moving ie. go or java forward because
> >>>>> we can't port new API functionality to C/Python/C#/whenever. If
> >>>>> there is someone who *does* use above in production and *wish* to
> >>>>> have new API parts then that person/organization duty is to do it.
> >>>>>
> >>>>> Best,
> >>>>> Łukasz
> >>>>>
> >>>>>
> >>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I must admit that I would be in favor of keeping it as simple as
> >>>>> possible (max age in the connection string) and to implement more
> >>>>> of the missing parts in plc4x (like the subscription simulation
> >>>>> layer) and hereby getting the drivers we have a bit more aligned,
> >>>>> than to implement more and more special API features, that only
> >>>>> one or two drivers support. Cause this way we're running away form
> >>>>> the project-promise of running a given program at any type of PLC
> >>>>> with any
> >>>> protocol.
> >>>>>>
> >>>>>> I mean ... at the current velocity the project is underway, I
> >>>>>> wouldn't
> >>>>> really want to add a new MetaData layer that probably in the end
> >>>>> only
> >>>>> 1-2 people will be implementing and which we must not only find
> >>>>> out how to fill with life in the first place, but also port to all
> >>>>> of our supported languages.
> >>>>>>
> >>>>>> If someone's willing to bring in some serious man/womenpower, I'm
> >>>>>> fine
> >>>>> with that ... or we add it to some wish-list for future extensions.
> >>>>> But I wouldn't want to start something like this right now.
> >>>>>>
> >>>>>> Chris
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -----Ursprüngliche Nachricht-----
> >>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>>>>> Gesendet: Montag, 8. März 2021 13:07
> >>>>>> An: dev@plc4x.apache.org
> >>>>>> Betreff: Re: Source Time of PLC Value?
> >>>>>>
> >>>>>> I have not mentioned metadata term in my answer, so credits for
> >>>>>> mining
> >>>>> it API go to you. :-)
> >>>>>>
> >>>>>> Now since you brought it I remember that JDBC has a "result set
> >>>>> metadata". That might be thing we still miss. The result code is
> >>>>> most important and available instantly, other information is
> >>>>> supplemental and can be read additionally.
> >>>>>>
> >>>>>>
> >>>>>> Best,
> >>>>>> Łukasz
> >>>>>>
> >>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
> >>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to
> >>>>>>> +create
> >>>>>>> duplicate PLcValue classes and give us a lot of flexibility in
> >>>>>>> the
> >>>>> future.
> >>>>>>>
> >>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
> >>>>>>> <ch...@c-ware.de>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> sorry for being late to the party ... KNX is currently
> >>>>>>>> consuming all my cycles.
> >>>>>>>> Just wanted to add my thoughts to the discussion.
> >>>>>>>>
> >>>>>>>> Initially I thought about adding this sort of information to
> >>>>>>>> the API, but then I thought that we have so little protocols
> >>>>>>>> supporting this sort of concept that blowing up every PLCValue
> >>>>>>>> with this time information just would waste CPU time and Memory.
> >>>>>>>>
> >>>>>>>> When it comes to simulating subscriptions by polling in the
> >>>>>>>> background, I thought it would be a valid solution to provide a
> >>>>>>>> default age a value can have in the cache and to make it
> >>>>>>>> possible to override this in the connection string options. If
> >>>>>>>> a value is too old,
> >>>>> a return code of:
> >>>>>>>> STALE_DATA could be returned.
> >>>>>>>>
> >>>>>>>> Also, we do have an option of adding an additional set of
> >>>>>>>> PlcValue
> >>>>> types.
> >>>>>>>> If a driver supports this quality-of-service type of data, we
> >>>>>>>> could return special versions of PlcValues, that have these
> >>>> properties?
> >>>>>>>>
> >>>>>>>> This way we don't have to waste the CPU time and Memory for
> >>>>>>>> protocols that don't support this concept (We could even
> >>>>>>>> implement the methods on the default PlvValues to simply return
> >>>>>>>> some constants to the PlcValues all have the same API.
> >>>>>>>>
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>>>>>>> Gesendet: Montag, 8. März 2021 10:43
> >>>>>>>> An: dev@plc4x.apache.org
> >>>>>>>> Betreff: Re: Source Time of PLC Value?
> >>>>>>>>
> >>>>>>>> Looking at issue we could utilize marker interfaces.
> >>>>>>>> At least in Java we could define Qualified and Timestamped types.
> >>>>>>>>
> >>>>>>>> Yet looking at present state of API we are unlikely to make
> >>>>>>>> each and every PlcValue in each and every variant (qualified
> >>>>>>>> and timestamped, timestamped, qualified). It would force quite
> >>>>>>>> a lot of
> >>>> wrapping.
> >>>>>>>> Safest option I see is to extend subscription/response API with
> >>>>>>>> additional method to retrieve all markers. For most of drivers
> >>>>>>>> it would return just an empty set. It leaves us also an open
> >>>>>>>> door for future drivers (or devices) to ship additional piece
> >>>>>>>> of metadata which
> >>>>> does not fit into present API.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Łukasz
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
> >>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
> >>>>>>>>>
> >>>>>>>>> uint64_t timestamp;
> >>>>>>>>> typedef enum {
> >>>>>>>>>     /* timestamp was generated by the source device */
> >>>>>>>>>     SOURCE,
> >>>>>>>>>     /* timestamp was generated by plc4x on receiving */
> >>>>>>>>>     GENERATED,
> >>>>>>>>> } TimestampQuality;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson
> >>>>>>>>>> <be...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> We do have some error codes, but it would just need to be
> >>>>>>>>>> extended a bit. I don't think we have one for BAD (we still
> >>>>>>>>>> receive data but the source has marked it bad).
> >>>>>>>>>>
> >>>>>>>>>> Timestamps I think are definitely a good idea, especially if
> >>>>>>>>>> we eventually DNP3.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
> >>>>>>>>>> <lu...@code-house.org>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> You have response code for each individual field which
> >>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same
> >>>>>>>>>>> information should be available also for subscriptions.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Łukasz
> >>>>>>>>>>>
> >>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
> >>>>>>>>>>>> Ah, forgot: and is there a status available?
> >>>>>>>>>>>> Valid/invalid/good/bad
> >>>>>>>>>>> indicator of a plc value?
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
> >>>>>>>>>>>>> <an...@me.com.INVALID>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How can I get the source time of a PlcValue?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
> >>>>>>>>>>>>> b) from a read request  PlcReadResponse
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>> Andreas
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >
>
>

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

Hi Christof,

Yeah, that does not sound that good…

I typically deal with SCADA systems for that need - collect plc data, add some added value (metadata, structure, object models, ...) - and then publish this enriched data via …hm, maybe via OPC UA?  …  but sure, SCADA systems are not for free … or use PLC4X  .. and maybe mapped.com <http://mapped.com/> to add enrich the data? :-) No glue what is behind the scenes of mapped.com <http://mapped.com/>…

Andreas

> On 10.03.2021, at 16:22, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Andreas,
> 
> Glad I didn't upset you too much ;-)
> 
> And I guess I'm also a bit pissed about OPC-UA in general. At first, I had such a hard time selling PLC4X as OPC-UA was promising so much, but without proof, that almost nobody gave open-source and PLC4X a chance. So much money was burnt in 2018 and 2019 with unsuccessful OPC-UA projects by the big automation companies. 
> 
> Then end of 2019 some of the ones, that decided to go the OPC-UA path, started contacting me again. But when we wanted to discuss terms, they sometimes had the nerve to say something like: We wasted so much money on OPC-UA and our budget is almost depleted ... we can only hire your company if you greatly reduce the hourly rate. Guess what my answer was? ;-)
> 
> Chris
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Andreas Vogler <an...@me.com.INVALID> 
> Gesendet: Mittwoch, 10. März 2021 16:06
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> 
> No problem, I think I understand what you mean… and I agree on the device level OPC UA is maybe too much / to fat (or not well implemented :-))
> 
> 
> 
>> On 10.03.2021, at 15:50, Christofer Dutz <ch...@c-ware.de> wrote:
>> 
>> Hi Andreas,
>> 
>> I didn't want to upset you. I should have been a bit more clear on this:
>> 
>> In 2018, when I was travelling through the country proclaiming PLC4X, all I heard was: But OPC-UA can do all of this (Without having tested it) ... 2019 most of the people who tried it were already looking for alternatives. End of 2019 and beginning of 2020 was definitely the age of MQTT (Of course OPC UA now provided a MQTT-like communication form).
>> 
>> I think OPC-UA is great for machine-to-machine communication and stuff like dashboards. But as soon as you want to do high volume data acquisition. Especially for use cases like machine-learning etc. the processing power of even modern PLCs can't handle the load.
>> 
>> For example, at one of the biggest German car manufacturers, they were not able to get more than 200 datapoints every 2 seconds from a new S7 400 PLC via OPC-UA. These are probably the most powerful ones Siemens has to offer. If they requested 300, the PLC shut down the IO process due to overload. With PLC4X using the S7 protocol, we were able to get 2600 datapoints in 200ms from the same machines. 
>> 
>> On the client side on one of the standard VMs they were using, running the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints. Then the CPU power was 100% ... in our case we run PLC4X on the same VMs using S7 and there we could collect 200 PLCs with 2600 datapoints. So also on the client side OPC-UA is a very fat project, that requires a lot of CPU time.
>> 
>> I hope this explains things a bit more. If you are just reading a few datapoints or using OPC-UA for Server-to-Server communication the servers will be able to cope with more. And if it doesn't you just add more nodes to the compute cluster.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Andreas Vogler <an...@me.com.INVALID>
>> Gesendet: Mittwoch, 10. März 2021 15:38
>> An: dev@plc4x.apache.org
>> Betreff: Re: Source Time of PLC Value?
>> Priorität: Niedrig
>> 
>> Just a side note: it hurts to hear that OPC UA sucks badly 
>> performance-wise. I have done tests with Milo OPC UA Client and SCADA 
>> OPC UA servers, and they perform pretty fine. 200 MQTT clients writing 
>> values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, 
>> each OPC UA server handles 50000 values changes per second: 
>> https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-v
>> ertx-activity-6769029730797719552-ZkjG 
>> <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-
>> vertx-activity-6769029730797719552-ZkjG>
>> 
>> But I fully agree with MQTT - it’s simple - and I think that’s the 
>> reason why it is so popular…
>> 
>> 
>>> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
>>> 
>>> Yes,
>>> 
>>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>>> 
>>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>>> 
>>> Just my thoughts on this.
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>> Gesendet: Mittwoch, 10. März 2021 12:58
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>> 
>>> Matthias,
>>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>>> 
>>> Two major points I see:
>>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>>> 
>>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>>> 
>>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>>> 
>>> Best,
>>> Łukasz
>>> 
>>> [1]
>>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/d
>>> e
>>> corator
>>> [2]
>>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/p
>>> l
>>> c4x/decorator/retry/ReadRetry.java
>>> [3]
>>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/p
>>> l c4x/decorator/retry/RetryDecoratorReadTest.java
>>> 
>>> 
>>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>>> Hi all,
>>>> 
>>>> i see there a point of how useful this meta information could be in 
>>>> some scenarios. But i would rather go the route of chris. Because I 
>>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>>> structure. I would not say that all API should have the same java 
>>>> structure on each platform but if you have for all the different 
>>>> platforms other featuresets you could also just use partially the 
>>>> code generation attempt to allow a separate meta info layer.
>>>> 
>>>> And there would be the question if that is worth the additional feature?
>>>> I mean I know we could add especially on the OPC UA side some meta 
>>>> information but is there more than the  actual source timestamp? And 
>>>> if there is none, would it be not better to use it as the actual 
>>>> timestamp of the value?
>>>> 
>>>> 
>>>> Einen schönen Abend ;)
>>>> Greetings Matthias
>>>> 
>>>> 
>>>> 
>>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>>> christofer.dutz@c-ware.de>:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I'm generally more concerned about users expecting us to deliver 
>>>>> feature that driver X has for driver Y too.
>>>>> 
>>>>> I won't object, if you think it's worth doing.
>>>>> 
>>>>> Perhaps if you could whip up an example in a feature branch? I 
>>>>> think perhaps I was still not understanding what you propose.
>>>>> 
>>>>> Would that be ok?
>>>>> 
>>>>> 
>>>>> Chris
>>>>> 
>>>>> 
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Lukas Ott <ot...@gmail.com>
>>>>> Gesendet: Montag, 8. März 2021 14:20
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>> 
>>>>> +1 to Lukasz
>>>>> 
>>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>>> luke@code-house.org>:
>>>>> 
>>>>>> Hold on for a second.
>>>>>> 
>>>>>> Making options pushed over connection string is a long term recipe 
>>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>>> Most of components distinguish producer (writer) and consumer
>>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>>> timestamped and
>>>>> 'qualified'
>>>>>> and not will complicate encoding logic (how we push above info to 
>>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>>> compatible with above?
>>>>>> 
>>>>>> That is why I rather opt for extending API in a way which does not 
>>>>>> force existing drivers to do anything. When any of community 
>>>>>> members, regardless if its developer or user, will find it 
>>>>>> necessary to be ported in another driver, she or he have right to 
>>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>> 
>>>>>> Speaking of which I am also not concerned about making all 
>>>>>> languages having the same functionality as languages tend to offer 
>>>>>> different options which might be hard to be ported. Lets stick to 
>>>>>> basics which are same (connection strings, field definitions, 
>>>>>> read/write/subscribe
>>>>>> syntax) and let various runtime lead their own ways toward
>>>>> implementation.
>>>>>> 
>>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>>> there is someone who *does* use above in production and *wish* to 
>>>>>> have new API parts then that person/organization duty is to do it.
>>>>>> 
>>>>>> Best,
>>>>>> Łukasz
>>>>>> 
>>>>>> 
>>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>>> possible (max age in the connection string) and to implement more 
>>>>>> of the missing parts in plc4x (like the subscription simulation
>>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>>> than to implement more and more special API features, that only 
>>>>>> one or two drivers support. Cause this way we're running away form 
>>>>>> the project-promise of running a given program at any type of PLC 
>>>>>> with any
>>>>> protocol.
>>>>>>> 
>>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>>> wouldn't
>>>>>> really want to add a new MetaData layer that probably in the end 
>>>>>> only
>>>>>> 1-2 people will be implementing and which we must not only find 
>>>>>> out how to fill with life in the first place, but also port to all 
>>>>>> of our supported languages.
>>>>>>> 
>>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>>> fine
>>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>>> But I wouldn't want to start something like this right now.
>>>>>>> 
>>>>>>> Chris
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>> 
>>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>>> mining
>>>>>> it API go to you. :-)
>>>>>>> 
>>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>>> metadata". That might be thing we still miss. The result code is 
>>>>>> most important and available instantly, other information is 
>>>>>> supplemental and can be read additionally.
>>>>>>> 
>>>>>>> 
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>> 
>>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>>> +create
>>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>>> the
>>>>>> future.
>>>>>>>> 
>>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>>> <ch...@c-ware.de>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> sorry for being late to the party ... KNX is currently 
>>>>>>>>> consuming all my cycles.
>>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>> 
>>>>>>>>> Initially I thought about adding this sort of information to 
>>>>>>>>> the API, but then I thought that we have so little protocols 
>>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>> 
>>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>>> possible to override this in the connection string options. If 
>>>>>>>>> a value is too old,
>>>>>> a return code of:
>>>>>>>>> STALE_DATA could be returned.
>>>>>>>>> 
>>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>>> PlcValue
>>>>>> types.
>>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>>> could return special versions of PlcValues, that have these
>>>>> properties?
>>>>>>>>> 
>>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>> 
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>> 
>>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>> 
>>>>>>>>> Yet looking at present state of API we are unlikely to make 
>>>>>>>>> each and every PlcValue in each and every variant (qualified 
>>>>>>>>> and timestamped, timestamped, qualified). It would force quite 
>>>>>>>>> a lot of
>>>>> wrapping.
>>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>>> door for future drivers (or devices) to ship additional piece 
>>>>>>>>> of metadata which
>>>>>> does not fit into present API.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Łukasz
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>> 
>>>>>>>>>> uint64_t timestamp;
>>>>>>>>>> typedef enum {
>>>>>>>>>>    /* timestamp was generated by the source device */
>>>>>>>>>>    SOURCE,
>>>>>>>>>>    /* timestamp was generated by plc4x on receiving */
>>>>>>>>>>    GENERATED,
>>>>>>>>>> } TimestampQuality;
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson 
>>>>>>>>>>> <be...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>> 
>>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>>> <lu...@code-house.org>
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> You have response code for each individual field which 
>>>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Łukasz
>>>>>>>>>>>> 
>>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>

Re: Source Time of PLC Value?

Posted by Otto Fowler <ot...@gmail.com>.

On Metadata:
I think it is a Good Thing™ to have.  My only ( possibly academic ) concern is conflating data that comes from the device with data that is generated or concerned with PLC4x’s workings.  I think a good separation of concerns there would good to think about.  


On OPC-UA:
Compared to DA, it is ridiculously complex to implement both on the client and server side.  But that is the cost of the capabilities they wanted to  add that were missing from DA ( and the other specs that were all merged together in UA ). Especially if you only want “DA without DCOM”.

> On Mar 10, 2021, at 10:22, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Andreas,
> 
> Glad I didn't upset you too much ;-)
> 
> And I guess I'm also a bit pissed about OPC-UA in general. At first, I had such a hard time selling PLC4X as OPC-UA was promising so much, but without proof, that almost nobody gave open-source and PLC4X a chance. So much money was burnt in 2018 and 2019 with unsuccessful OPC-UA projects by the big automation companies. 
> 
> Then end of 2019 some of the ones, that decided to go the OPC-UA path, started contacting me again. But when we wanted to discuss terms, they sometimes had the nerve to say something like: We wasted so much money on OPC-UA and our budget is almost depleted ... we can only hire your company if you greatly reduce the hourly rate. Guess what my answer was? ;-)
> 
> Chris
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Andreas Vogler <an...@me.com.INVALID> 
> Gesendet: Mittwoch, 10. März 2021 16:06
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> 
> No problem, I think I understand what you mean… and I agree on the device level OPC UA is maybe too much / to fat (or not well implemented :-))
> 
> 
> 
>> On 10.03.2021, at 15:50, Christofer Dutz <ch...@c-ware.de> wrote:
>> 
>> Hi Andreas,
>> 
>> I didn't want to upset you. I should have been a bit more clear on this:
>> 
>> In 2018, when I was travelling through the country proclaiming PLC4X, all I heard was: But OPC-UA can do all of this (Without having tested it) ... 2019 most of the people who tried it were already looking for alternatives. End of 2019 and beginning of 2020 was definitely the age of MQTT (Of course OPC UA now provided a MQTT-like communication form).
>> 
>> I think OPC-UA is great for machine-to-machine communication and stuff like dashboards. But as soon as you want to do high volume data acquisition. Especially for use cases like machine-learning etc. the processing power of even modern PLCs can't handle the load.
>> 
>> For example, at one of the biggest German car manufacturers, they were not able to get more than 200 datapoints every 2 seconds from a new S7 400 PLC via OPC-UA. These are probably the most powerful ones Siemens has to offer. If they requested 300, the PLC shut down the IO process due to overload. With PLC4X using the S7 protocol, we were able to get 2600 datapoints in 200ms from the same machines. 
>> 
>> On the client side on one of the standard VMs they were using, running the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints. Then the CPU power was 100% ... in our case we run PLC4X on the same VMs using S7 and there we could collect 200 PLCs with 2600 datapoints. So also on the client side OPC-UA is a very fat project, that requires a lot of CPU time.
>> 
>> I hope this explains things a bit more. If you are just reading a few datapoints or using OPC-UA for Server-to-Server communication the servers will be able to cope with more. And if it doesn't you just add more nodes to the compute cluster.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Andreas Vogler <an...@me.com.INVALID>
>> Gesendet: Mittwoch, 10. März 2021 15:38
>> An: dev@plc4x.apache.org
>> Betreff: Re: Source Time of PLC Value?
>> Priorität: Niedrig
>> 
>> Just a side note: it hurts to hear that OPC UA sucks badly 
>> performance-wise. I have done tests with Milo OPC UA Client and SCADA 
>> OPC UA servers, and they perform pretty fine. 200 MQTT clients writing 
>> values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, 
>> each OPC UA server handles 50000 values changes per second: 
>> https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-v
>> ertx-activity-6769029730797719552-ZkjG 
>> <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-
>> vertx-activity-6769029730797719552-ZkjG>
>> 
>> But I fully agree with MQTT - it’s simple - and I think that’s the 
>> reason why it is so popular…
>> 
>> 
>>> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
>>> 
>>> Yes,
>>> 
>>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>>> 
>>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>>> 
>>> Just my thoughts on this.
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>> Gesendet: Mittwoch, 10. März 2021 12:58
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>> 
>>> Matthias,
>>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>>> 
>>> Two major points I see:
>>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>>> 
>>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>>> 
>>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>>> 
>>> Best,
>>> Łukasz
>>> 
>>> [1]
>>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/d
>>> e
>>> corator
>>> [2]
>>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/p
>>> l
>>> c4x/decorator/retry/ReadRetry.java
>>> [3]
>>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>>> / 
>>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/p
>>> l c4x/decorator/retry/RetryDecoratorReadTest.java
>>> 
>>> 
>>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>>> Hi all,
>>>> 
>>>> i see there a point of how useful this meta information could be in 
>>>> some scenarios. But i would rather go the route of chris. Because I 
>>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>>> structure. I would not say that all API should have the same java 
>>>> structure on each platform but if you have for all the different 
>>>> platforms other featuresets you could also just use partially the 
>>>> code generation attempt to allow a separate meta info layer.
>>>> 
>>>> And there would be the question if that is worth the additional feature?
>>>> I mean I know we could add especially on the OPC UA side some meta 
>>>> information but is there more than the  actual source timestamp? And 
>>>> if there is none, would it be not better to use it as the actual 
>>>> timestamp of the value?
>>>> 
>>>> 
>>>> Einen schönen Abend ;)
>>>> Greetings Matthias
>>>> 
>>>> 
>>>> 
>>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>>> christofer.dutz@c-ware.de>:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I'm generally more concerned about users expecting us to deliver 
>>>>> feature that driver X has for driver Y too.
>>>>> 
>>>>> I won't object, if you think it's worth doing.
>>>>> 
>>>>> Perhaps if you could whip up an example in a feature branch? I 
>>>>> think perhaps I was still not understanding what you propose.
>>>>> 
>>>>> Would that be ok?
>>>>> 
>>>>> 
>>>>> Chris
>>>>> 
>>>>> 
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Lukas Ott <ot...@gmail.com>
>>>>> Gesendet: Montag, 8. März 2021 14:20
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>> 
>>>>> +1 to Lukasz
>>>>> 
>>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>>> luke@code-house.org>:
>>>>> 
>>>>>> Hold on for a second.
>>>>>> 
>>>>>> Making options pushed over connection string is a long term recipe 
>>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>>> Most of components distinguish producer (writer) and consumer
>>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>>> timestamped and
>>>>> 'qualified'
>>>>>> and not will complicate encoding logic (how we push above info to 
>>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>>> compatible with above?
>>>>>> 
>>>>>> That is why I rather opt for extending API in a way which does not 
>>>>>> force existing drivers to do anything. When any of community 
>>>>>> members, regardless if its developer or user, will find it 
>>>>>> necessary to be ported in another driver, she or he have right to 
>>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>> 
>>>>>> Speaking of which I am also not concerned about making all 
>>>>>> languages having the same functionality as languages tend to offer 
>>>>>> different options which might be hard to be ported. Lets stick to 
>>>>>> basics which are same (connection strings, field definitions, 
>>>>>> read/write/subscribe
>>>>>> syntax) and let various runtime lead their own ways toward
>>>>> implementation.
>>>>>> 
>>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>>> there is someone who *does* use above in production and *wish* to 
>>>>>> have new API parts then that person/organization duty is to do it.
>>>>>> 
>>>>>> Best,
>>>>>> Łukasz
>>>>>> 
>>>>>> 
>>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>>> possible (max age in the connection string) and to implement more 
>>>>>> of the missing parts in plc4x (like the subscription simulation
>>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>>> than to implement more and more special API features, that only 
>>>>>> one or two drivers support. Cause this way we're running away form 
>>>>>> the project-promise of running a given program at any type of PLC 
>>>>>> with any
>>>>> protocol.
>>>>>>> 
>>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>>> wouldn't
>>>>>> really want to add a new MetaData layer that probably in the end 
>>>>>> only
>>>>>> 1-2 people will be implementing and which we must not only find 
>>>>>> out how to fill with life in the first place, but also port to all 
>>>>>> of our supported languages.
>>>>>>> 
>>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>>> fine
>>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>>> But I wouldn't want to start something like this right now.
>>>>>>> 
>>>>>>> Chris
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>> 
>>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>>> mining
>>>>>> it API go to you. :-)
>>>>>>> 
>>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>>> metadata". That might be thing we still miss. The result code is 
>>>>>> most important and available instantly, other information is 
>>>>>> supplemental and can be read additionally.
>>>>>>> 
>>>>>>> 
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>> 
>>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>>> +create
>>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>>> the
>>>>>> future.
>>>>>>>> 
>>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>>> <ch...@c-ware.de>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> sorry for being late to the party ... KNX is currently 
>>>>>>>>> consuming all my cycles.
>>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>> 
>>>>>>>>> Initially I thought about adding this sort of information to 
>>>>>>>>> the API, but then I thought that we have so little protocols 
>>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>> 
>>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>>> possible to override this in the connection string options. If 
>>>>>>>>> a value is too old,
>>>>>> a return code of:
>>>>>>>>> STALE_DATA could be returned.
>>>>>>>>> 
>>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>>> PlcValue
>>>>>> types.
>>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>>> could return special versions of PlcValues, that have these
>>>>> properties?
>>>>>>>>> 
>>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>> 
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>> 
>>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>> 
>>>>>>>>> Yet looking at present state of API we are unlikely to make 
>>>>>>>>> each and every PlcValue in each and every variant (qualified 
>>>>>>>>> and timestamped, timestamped, qualified). It would force quite 
>>>>>>>>> a lot of
>>>>> wrapping.
>>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>>> door for future drivers (or devices) to ship additional piece 
>>>>>>>>> of metadata which
>>>>>> does not fit into present API.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Łukasz
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>> 
>>>>>>>>>> uint64_t timestamp;
>>>>>>>>>> typedef enum {
>>>>>>>>>>    /* timestamp was generated by the source device */
>>>>>>>>>>    SOURCE,
>>>>>>>>>>    /* timestamp was generated by plc4x on receiving */
>>>>>>>>>>    GENERATED,
>>>>>>>>>> } TimestampQuality;
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson 
>>>>>>>>>>> <be...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>> 
>>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>>> <lu...@code-house.org>
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> You have response code for each individual field which 
>>>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Łukasz
>>>>>>>>>>>> 
>>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>

AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi Andreas,

Glad I didn't upset you too much ;-)

And I guess I'm also a bit pissed about OPC-UA in general. At first, I had such a hard time selling PLC4X as OPC-UA was promising so much, but without proof, that almost nobody gave open-source and PLC4X a chance. So much money was burnt in 2018 and 2019 with unsuccessful OPC-UA projects by the big automation companies. 

Then end of 2019 some of the ones, that decided to go the OPC-UA path, started contacting me again. But when we wanted to discuss terms, they sometimes had the nerve to say something like: We wasted so much money on OPC-UA and our budget is almost depleted ... we can only hire your company if you greatly reduce the hourly rate. Guess what my answer was? ;-)

Chris

 

-----Ursprüngliche Nachricht-----
Von: Andreas Vogler <an...@me.com.INVALID> 
Gesendet: Mittwoch, 10. März 2021 16:06
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?

No problem, I think I understand what you mean… and I agree on the device level OPC UA is maybe too much / to fat (or not well implemented :-))



> On 10.03.2021, at 15:50, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Andreas,
> 
> I didn't want to upset you. I should have been a bit more clear on this:
> 
> In 2018, when I was travelling through the country proclaiming PLC4X, all I heard was: But OPC-UA can do all of this (Without having tested it) ... 2019 most of the people who tried it were already looking for alternatives. End of 2019 and beginning of 2020 was definitely the age of MQTT (Of course OPC UA now provided a MQTT-like communication form).
> 
> I think OPC-UA is great for machine-to-machine communication and stuff like dashboards. But as soon as you want to do high volume data acquisition. Especially for use cases like machine-learning etc. the processing power of even modern PLCs can't handle the load.
> 
> For example, at one of the biggest German car manufacturers, they were not able to get more than 200 datapoints every 2 seconds from a new S7 400 PLC via OPC-UA. These are probably the most powerful ones Siemens has to offer. If they requested 300, the PLC shut down the IO process due to overload. With PLC4X using the S7 protocol, we were able to get 2600 datapoints in 200ms from the same machines. 
> 
> On the client side on one of the standard VMs they were using, running the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints. Then the CPU power was 100% ... in our case we run PLC4X on the same VMs using S7 and there we could collect 200 PLCs with 2600 datapoints. So also on the client side OPC-UA is a very fat project, that requires a lot of CPU time.
> 
> I hope this explains things a bit more. If you are just reading a few datapoints or using OPC-UA for Server-to-Server communication the servers will be able to cope with more. And if it doesn't you just add more nodes to the compute cluster.
> 
> Chris
> 
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Andreas Vogler <an...@me.com.INVALID>
> Gesendet: Mittwoch, 10. März 2021 15:38
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> Priorität: Niedrig
> 
> Just a side note: it hurts to hear that OPC UA sucks badly 
> performance-wise. I have done tests with Milo OPC UA Client and SCADA 
> OPC UA servers, and they perform pretty fine. 200 MQTT clients writing 
> values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, 
> each OPC UA server handles 50000 values changes per second: 
> https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-v
> ertx-activity-6769029730797719552-ZkjG 
> <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-
> vertx-activity-6769029730797719552-ZkjG>
> 
> But I fully agree with MQTT - it’s simple - and I think that’s the 
> reason why it is so popular…
> 
> 
>> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
>> 
>> Yes,
>> 
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>> 
>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>> 
>> Just my thoughts on this.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>> 
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>> 
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>> 
>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>> 
>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>> 
>> Best,
>> Łukasz
>> 
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
>> / 
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/d
>> e
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> / 
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/p
>> l
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> / 
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/p
>> l c4x/decorator/retry/RetryDecoratorReadTest.java
>> 
>> 
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>> 
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>> 
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>> 
>>> 
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>> 
>>> 
>>> 
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>> 
>>>> Hi,
>>>> 
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>> 
>>>> I won't object, if you think it's worth doing.
>>>> 
>>>> Perhaps if you could whip up an example in a feature branch? I 
>>>> think perhaps I was still not understanding what you propose.
>>>> 
>>>> Would that be ok?
>>>> 
>>>> 
>>>> Chris
>>>> 
>>>> 
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>> 
>>>> +1 to Lukasz
>>>> 
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>> 
>>>>> Hold on for a second.
>>>>> 
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>> 
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>> 
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>> 
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>> 
>>>>> Best,
>>>>> Łukasz
>>>>> 
>>>>> 
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only 
>>>>> one or two drivers support. Cause this way we're running away form 
>>>>> the project-promise of running a given program at any type of PLC 
>>>>> with any
>>>> protocol.
>>>>>> 
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find 
>>>>> out how to fill with life in the first place, but also port to all 
>>>>> of our supported languages.
>>>>>> 
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>> 
>>>>>> Chris
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>> 
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>> 
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> Łukasz
>>>>>> 
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>> 
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> sorry for being late to the party ... KNX is currently 
>>>>>>>> consuming all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>> 
>>>>>>>> Initially I thought about adding this sort of information to 
>>>>>>>> the API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>> 
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If 
>>>>>>>> a value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>> 
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>> 
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>> 
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>> 
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>> 
>>>>>>>> Yet looking at present state of API we are unlikely to make 
>>>>>>>> each and every PlcValue in each and every variant (qualified 
>>>>>>>> and timestamped, timestamped, qualified). It would force quite 
>>>>>>>> a lot of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece 
>>>>>>>> of metadata which
>>>>> does not fit into present API.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>> 
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>     /* timestamp was generated by the source device */
>>>>>>>>>     SOURCE,
>>>>>>>>>     /* timestamp was generated by plc4x on receiving */
>>>>>>>>>     GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson 
>>>>>>>>>> <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>> 
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> You have response code for each individual field which 
>>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>> 
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

No problem, I think I understand what you mean… and I agree on the device level OPC UA is maybe too much / to fat (or not well implemented :-))



> On 10.03.2021, at 15:50, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Andreas,
> 
> I didn't want to upset you. I should have been a bit more clear on this:
> 
> In 2018, when I was travelling through the country proclaiming PLC4X, all I heard was: But OPC-UA can do all of this (Without having tested it) ... 2019 most of the people who tried it were already looking for alternatives. End of 2019 and beginning of 2020 was definitely the age of MQTT (Of course OPC UA now provided a MQTT-like communication form).
> 
> I think OPC-UA is great for machine-to-machine communication and stuff like dashboards. But as soon as you want to do high volume data acquisition. Especially for use cases like machine-learning etc. the processing power of even modern PLCs can't handle the load.
> 
> For example, at one of the biggest German car manufacturers, they were not able to get more than 200 datapoints every 2 seconds from a new S7 400 PLC via OPC-UA. These are probably the most powerful ones Siemens has to offer. If they requested 300, the PLC shut down the IO process due to overload. With PLC4X using the S7 protocol, we were able to get 2600 datapoints in 200ms from the same machines. 
> 
> On the client side on one of the standard VMs they were using, running the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints. Then the CPU power was 100% ... in our case we run PLC4X on the same VMs using S7 and there we could collect 200 PLCs with 2600 datapoints. So also on the client side OPC-UA is a very fat project, that requires a lot of CPU time.
> 
> I hope this explains things a bit more. If you are just reading a few datapoints or using OPC-UA for Server-to-Server communication the servers will be able to cope with more. And if it doesn't you just add more nodes to the compute cluster.
> 
> Chris
> 
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Andreas Vogler <an...@me.com.INVALID> 
> Gesendet: Mittwoch, 10. März 2021 15:38
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> Priorität: Niedrig
> 
> Just a side note: it hurts to hear that OPC UA sucks badly performance-wise. I have done tests with Milo OPC UA Client and SCADA OPC UA servers, and they perform pretty fine. 200 MQTT clients writing values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, each OPC UA server handles 50000 values changes per second: https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG> 
> 
> But I fully agree with MQTT - it’s simple - and I think that’s the reason why it is so popular… 
> 
> 
>> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
>> 
>> Yes,
>> 
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
>> 
>> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
>> 
>> Just my thoughts on this.
>> 
>> Chris
>> 
>> 
>> 
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>> 
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
>> 
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
>> 
>> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
>> 
>> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
>> 
>> Best,
>> Łukasz
>> 
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
>> c4x/decorator/retry/RetryDecoratorReadTest.java
>> 
>> 
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>> 
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>> 
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>> 
>>> 
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>> 
>>> 
>>> 
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>> 
>>>> Hi,
>>>> 
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>> 
>>>> I won't object, if you think it's worth doing.
>>>> 
>>>> Perhaps if you could whip up an example in a feature branch? I think 
>>>> perhaps I was still not understanding what you propose.
>>>> 
>>>> Would that be ok?
>>>> 
>>>> 
>>>> Chris
>>>> 
>>>> 
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>> 
>>>> +1 to Lukasz
>>>> 
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>> 
>>>>> Hold on for a second.
>>>>> 
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>> 
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>> 
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>> 
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>> 
>>>>> Best,
>>>>> Łukasz
>>>>> 
>>>>> 
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation 
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only one 
>>>>> or two drivers support. Cause this way we're running away form the 
>>>>> project-promise of running a given program at any type of PLC with 
>>>>> any
>>>> protocol.
>>>>>> 
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find out 
>>>>> how to fill with life in the first place, but also port to all of 
>>>>> our supported languages.
>>>>>> 
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>> 
>>>>>> Chris
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>> 
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>> 
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> Łukasz
>>>>>> 
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>> 
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>>> all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>> 
>>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>> 
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If a 
>>>>>>>> value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>> 
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>> 
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>> 
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>> 
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>> 
>>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>>> of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece of 
>>>>>>>> metadata which
>>>>> does not fit into present API.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>> 
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>     /* timestamp was generated by the source device */
>>>>>>>>>     SOURCE,
>>>>>>>>>     /* timestamp was generated by plc4x on receiving */
>>>>>>>>>     GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>> 
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>> 
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>

AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi Andreas,

I didn't want to upset you. I should have been a bit more clear on this:

In 2018, when I was travelling through the country proclaiming PLC4X, all I heard was: But OPC-UA can do all of this (Without having tested it) ... 2019 most of the people who tried it were already looking for alternatives. End of 2019 and beginning of 2020 was definitely the age of MQTT (Of course OPC UA now provided a MQTT-like communication form).

I think OPC-UA is great for machine-to-machine communication and stuff like dashboards. But as soon as you want to do high volume data acquisition. Especially for use cases like machine-learning etc. the processing power of even modern PLCs can't handle the load.

For example, at one of the biggest German car manufacturers, they were not able to get more than 200 datapoints every 2 seconds from a new S7 400 PLC via OPC-UA. These are probably the most powerful ones Siemens has to offer. If they requested 300, the PLC shut down the IO process due to overload. With PLC4X using the S7 protocol, we were able to get 2600 datapoints in 200ms from the same machines. 

On the client side on one of the standard VMs they were using, running the OPC-UA stack was able to collect data on 10 PLCs with 200 datapoints. Then the CPU power was 100% ... in our case we run PLC4X on the same VMs using S7 and there we could collect 200 PLCs with 2600 datapoints. So also on the client side OPC-UA is a very fat project, that requires a lot of CPU time.

I hope this explains things a bit more. If you are just reading a few datapoints or using OPC-UA for Server-to-Server communication the servers will be able to cope with more. And if it doesn't you just add more nodes to the compute cluster.

Chris

 



-----Ursprüngliche Nachricht-----
Von: Andreas Vogler <an...@me.com.INVALID> 
Gesendet: Mittwoch, 10. März 2021 15:38
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?
Priorität: Niedrig

Just a side note: it hurts to hear that OPC UA sucks badly performance-wise. I have done tests with Milo OPC UA Client and SCADA OPC UA servers, and they perform pretty fine. 200 MQTT clients writing values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, each OPC UA server handles 50000 values changes per second: https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG> 

But I fully agree with MQTT - it’s simple - and I think that’s the reason why it is so popular… 


> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Yes,
> 
> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
> 
> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
> 
> Just my thoughts on this.
> 
> Chris
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org>
> Gesendet: Mittwoch, 10. März 2021 12:58
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Source Time of PLC Value?
> 
> Matthias,
> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
> 
> Two major points I see:
> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
> 
> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
> 
> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
> 
> Best,
> Łukasz
> 
> [1]
> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
> corator
> [2]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
> c4x/decorator/retry/ReadRetry.java
> [3]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
> c4x/decorator/retry/RetryDecoratorReadTest.java
> 
> 
> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>> Hi all,
>> 
>> i see there a point of how useful this meta information could be in 
>> some scenarios. But i would rather go the route of chris. Because I 
>> see there is more of a risk of an overcomplicated inhomogeneous api 
>> structure. I would not say that all API should have the same java 
>> structure on each platform but if you have for all the different 
>> platforms other featuresets you could also just use partially the 
>> code generation attempt to allow a separate meta info layer.
>> 
>> And there would be the question if that is worth the additional feature?
>> I mean I know we could add especially on the OPC UA side some meta 
>> information but is there more than the  actual source timestamp? And 
>> if there is none, would it be not better to use it as the actual 
>> timestamp of the value?
>> 
>> 
>> Einen schönen Abend ;)
>> Greetings Matthias
>> 
>> 
>> 
>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>> christofer.dutz@c-ware.de>:
>> 
>>> Hi,
>>> 
>>> I'm generally more concerned about users expecting us to deliver 
>>> feature that driver X has for driver Y too.
>>> 
>>> I won't object, if you think it's worth doing.
>>> 
>>> Perhaps if you could whip up an example in a feature branch? I think 
>>> perhaps I was still not understanding what you propose.
>>> 
>>> Would that be ok?
>>> 
>>> 
>>> Chris
>>> 
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Lukas Ott <ot...@gmail.com>
>>> Gesendet: Montag, 8. März 2021 14:20
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>> 
>>> +1 to Lukasz
>>> 
>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>> luke@code-house.org>:
>>> 
>>>> Hold on for a second.
>>>> 
>>>> Making options pushed over connection string is a long term recipe 
>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>> if you starting with configuration with URIs and dynamic parameters.
>>>> Most of components distinguish producer (writer) and consumer
>>>> (receiver) options, most complicated components ended up with 40+ 
>>>> options settable via URI. Having a mix of PlcValues which are 
>>>> timestamped and
>>> 'qualified'
>>>> and not will complicate encoding logic (how we push above info to 
>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>> compatible with above?
>>>> 
>>>> That is why I rather opt for extending API in a way which does not 
>>>> force existing drivers to do anything. When any of community 
>>>> members, regardless if its developer or user, will find it 
>>>> necessary to be ported in another driver, she or he have right to 
>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>> 
>>>> Speaking of which I am also not concerned about making all 
>>>> languages having the same functionality as languages tend to offer 
>>>> different options which might be hard to be ported. Lets stick to 
>>>> basics which are same (connection strings, field definitions, 
>>>> read/write/subscribe
>>>> syntax) and let various runtime lead their own ways toward
>>> implementation.
>>>> 
>>>> Key point is simple - we won't be able to maintain all languages 
>>>> equally. We should not hold moving ie. go or java forward because 
>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>> there is someone who *does* use above in production and *wish* to 
>>>> have new API parts then that person/organization duty is to do it.
>>>> 
>>>> Best,
>>>> Łukasz
>>>> 
>>>> 
>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>> Hi,
>>>>> 
>>>>> I must admit that I would be in favor of keeping it as simple as
>>>> possible (max age in the connection string) and to implement more 
>>>> of the missing parts in plc4x (like the subscription simulation 
>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>> than to implement more and more special API features, that only one 
>>>> or two drivers support. Cause this way we're running away form the 
>>>> project-promise of running a given program at any type of PLC with 
>>>> any
>>> protocol.
>>>>> 
>>>>> I mean ... at the current velocity the project is underway, I 
>>>>> wouldn't
>>>> really want to add a new MetaData layer that probably in the end 
>>>> only
>>>> 1-2 people will be implementing and which we must not only find out 
>>>> how to fill with life in the first place, but also port to all of 
>>>> our supported languages.
>>>>> 
>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>> fine
>>>> with that ... or we add it to some wish-list for future extensions.
>>>> But I wouldn't want to start something like this right now.
>>>>> 
>>>>> Chris
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Source Time of PLC Value?
>>>>> 
>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>> mining
>>>> it API go to you. :-)
>>>>> 
>>>>> Now since you brought it I remember that JDBC has a "result set
>>>> metadata". That might be thing we still miss. The result code is 
>>>> most important and available instantly, other information is 
>>>> supplemental and can be read additionally.
>>>>> 
>>>>> 
>>>>> Best,
>>>>> Łukasz
>>>>> 
>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>> +create
>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>> the
>>>> future.
>>>>>> 
>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>> <ch...@c-ware.de>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>> all my cycles.
>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>> 
>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>> 
>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>> default age a value can have in the cache and to make it 
>>>>>>> possible to override this in the connection string options. If a 
>>>>>>> value is too old,
>>>> a return code of:
>>>>>>> STALE_DATA could be returned.
>>>>>>> 
>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>> PlcValue
>>>> types.
>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>> could return special versions of PlcValues, that have these
>>> properties?
>>>>>>> 
>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>> protocols that don't support this concept (We could even 
>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>> 
>>>>>>> Chris
>>>>>>> 
>>>>>>> 
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>> 
>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>> 
>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>> of
>>> wrapping.
>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>> door for future drivers (or devices) to ship additional piece of 
>>>>>>> metadata which
>>>> does not fit into present API.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>> 
>>>>>>> 
>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>> 
>>>>>>>> uint64_t timestamp;
>>>>>>>> typedef enum {
>>>>>>>>      /* timestamp was generated by the source device */
>>>>>>>>      SOURCE,
>>>>>>>>      /* timestamp was generated by plc4x on receiving */
>>>>>>>>      GENERATED,
>>>>>>>> } TimestampQuality;
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>> 
>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>> we eventually DNP3.
>>>>>>>>> 
>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>> <lu...@code-house.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Łukasz
>>>>>>>>>> 
>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>> 
>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>> 
>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Andreas
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>>

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

Just a side note: it hurts to hear that OPC UA sucks badly performance-wise. I have done tests with Milo OPC UA Client and SCADA OPC UA servers, and they perform pretty fine. 200 MQTT clients writing values via a Gateway with OPC UA Milo (Client) to two OPC UA servers, each OPC UA server handles 50000 values changes per second: https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG <https://www.linkedin.com/posts/andreas-vogler-361115122_graphql-mqtt-vertx-activity-6769029730797719552-ZkjG> 

But I fully agree with MQTT - it’s simple - and I think that’s the reason why it is so popular… 


> On 10.03.2021, at 14:34, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Yes,
> 
> OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.
> 
> Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.
> 
> Just my thoughts on this.
> 
> Chris
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org> 
> Gesendet: Mittwoch, 10. März 2021 12:58
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Source Time of PLC Value?
> 
> Matthias,
> I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.
> 
> Two major points I see:
> - Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
> There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
> - Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
> - Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.
> 
> I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.
> 
> Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.
> 
> Best,
> Łukasz
> 
> [1]
> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
> [2]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
> [3]
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java
> 
> 
> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>> Hi all,
>> 
>> i see there a point of how useful this meta information could be in 
>> some scenarios. But i would rather go the route of chris. Because I 
>> see there is more of a risk of an overcomplicated inhomogeneous api 
>> structure. I would not say that all API should have the same java 
>> structure on each platform but if you have for all the different 
>> platforms other featuresets you could also just use partially the code 
>> generation attempt to allow a separate meta info layer.
>> 
>> And there would be the question if that is worth the additional feature?
>> I mean I know we could add especially on the OPC UA side some meta 
>> information but is there more than the  actual source timestamp? And 
>> if there is none, would it be not better to use it as the actual 
>> timestamp of the value?
>> 
>> 
>> Einen schönen Abend ;)
>> Greetings Matthias
>> 
>> 
>> 
>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>> christofer.dutz@c-ware.de>:
>> 
>>> Hi,
>>> 
>>> I'm generally more concerned about users expecting us to deliver 
>>> feature that driver X has for driver Y too.
>>> 
>>> I won't object, if you think it's worth doing.
>>> 
>>> Perhaps if you could whip up an example in a feature branch? I think 
>>> perhaps I was still not understanding what you propose.
>>> 
>>> Would that be ok?
>>> 
>>> 
>>> Chris
>>> 
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Lukas Ott <ot...@gmail.com>
>>> Gesendet: Montag, 8. März 2021 14:20
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Source Time of PLC Value?
>>> 
>>> +1 to Lukasz
>>> 
>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>> luke@code-house.org>:
>>> 
>>>> Hold on for a second.
>>>> 
>>>> Making options pushed over connection string is a long term recipe 
>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>> if you starting with configuration with URIs and dynamic parameters. 
>>>> Most of components distinguish producer (writer) and consumer 
>>>> (receiver) options, most complicated components ended up with 40+ 
>>>> options settable via URI. Having a mix of PlcValues which are 
>>>> timestamped and
>>> 'qualified'
>>>> and not will complicate encoding logic (how we push above info to 
>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>> compatible with above?
>>>> 
>>>> That is why I rather opt for extending API in a way which does not 
>>>> force existing drivers to do anything. When any of community 
>>>> members, regardless if its developer or user, will find it necessary 
>>>> to be ported in another driver, she or he have right to do it. It is 
>>>> not only yours duty to keep all drivers and languages the same.
>>>> 
>>>> Speaking of which I am also not concerned about making all languages 
>>>> having the same functionality as languages tend to offer different 
>>>> options which might be hard to be ported. Lets stick to basics which 
>>>> are same (connection strings, field definitions, 
>>>> read/write/subscribe
>>>> syntax) and let various runtime lead their own ways toward
>>> implementation.
>>>> 
>>>> Key point is simple - we won't be able to maintain all languages 
>>>> equally. We should not hold moving ie. go or java forward because we 
>>>> can't port new API functionality to C/Python/C#/whenever. If there 
>>>> is someone who *does* use above in production and *wish* to have new 
>>>> API parts then that person/organization duty is to do it.
>>>> 
>>>> Best,
>>>> Łukasz
>>>> 
>>>> 
>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>> Hi,
>>>>> 
>>>>> I must admit that I would be in favor of keeping it as simple as
>>>> possible (max age in the connection string) and to implement more of 
>>>> the missing parts in plc4x (like the subscription simulation layer) 
>>>> and hereby getting the drivers we have a bit more aligned, than to 
>>>> implement more and more special API features, that only one or two 
>>>> drivers support. Cause this way we're running away form the 
>>>> project-promise of running a given program at any type of PLC with 
>>>> any
>>> protocol.
>>>>> 
>>>>> I mean ... at the current velocity the project is underway, I 
>>>>> wouldn't
>>>> really want to add a new MetaData layer that probably in the end 
>>>> only
>>>> 1-2 people will be implementing and which we must not only find out 
>>>> how to fill with life in the first place, but also port to all of 
>>>> our supported languages.
>>>>> 
>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>> fine
>>>> with that ... or we add it to some wish-list for future extensions.
>>>> But I wouldn't want to start something like this right now.
>>>>> 
>>>>> Chris
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Source Time of PLC Value?
>>>>> 
>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>> mining
>>>> it API go to you. :-)
>>>>> 
>>>>> Now since you brought it I remember that JDBC has a "result set
>>>> metadata". That might be thing we still miss. The result code is 
>>>> most important and available instantly, other information is 
>>>> supplemental and can be read additionally.
>>>>> 
>>>>> 
>>>>> Best,
>>>>> Łukasz
>>>>> 
>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>>> future.
>>>>>> 
>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>> <ch...@c-ware.de>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>>> all my cycles.
>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>> 
>>>>>>> Initially I thought about adding this sort of information to the 
>>>>>>> API, but then I thought that we have so little protocols 
>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>> 
>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>> default age a value can have in the cache and to make it possible 
>>>>>>> to override this in the connection string options. If a value is 
>>>>>>> too old,
>>>> a return code of:
>>>>>>> STALE_DATA could be returned.
>>>>>>> 
>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>> PlcValue
>>>> types.
>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>> could return special versions of PlcValues, that have these
>>> properties?
>>>>>>> 
>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>> protocols that don't support this concept (We could even 
>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>> 
>>>>>>> Chris
>>>>>>> 
>>>>>>> 
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>> 
>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>> 
>>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>>> of
>>> wrapping.
>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>> additional method to retrieve all markers. For most of drivers it 
>>>>>>> would return just an empty set. It leaves us also an open door 
>>>>>>> for future drivers (or devices) to ship additional piece of 
>>>>>>> metadata which
>>>> does not fit into present API.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Łukasz
>>>>>>> 
>>>>>>> 
>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>> 
>>>>>>>> uint64_t timestamp;
>>>>>>>> typedef enum {
>>>>>>>>      /* timestamp was generated by the source device */
>>>>>>>>      SOURCE,
>>>>>>>>      /* timestamp was generated by plc4x on receiving */
>>>>>>>>      GENERATED,
>>>>>>>> } TimestampQuality;
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>> 
>>>>>>>>> Timestamps I think are definitely a good idea, especially if we 
>>>>>>>>> eventually DNP3.
>>>>>>>>> 
>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>> <lu...@code-house.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Łukasz
>>>>>>>>>> 
>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>> 
>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>> 
>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Andreas
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>>

AW: AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Yes,

OPC-UA has all of this ... and it's probably one of the reasons so many implementations differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks that badly performance-wise.

Just saying that we should be careful what we choose as reference. In contrast typical MQTT doesn't have all of this and it seems to be gaining more and more traction, just because it doesn't have all of this overhead.

Just my thoughts on this.

Chris




-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Mittwoch, 10. März 2021 12:58
An: dev@plc4x.apache.org
Betreff: Re: AW: Source Time of PLC Value?

Matthias,
I do believe there are several points which are relevant and still not possible to be implemented using available PLC4X API. Metadata might not be most fortunate way, but it can enable or ease analytical and troubleshooting scenarios.

Two major points I see:
- Verification of device condition - ie. how does caller know response latency? This is especially relevant with frequent polling. Currently caller has no measures to know that and can not measure that in reliable way due to async nature of API.
There is a way to implement backpressure technics with callback chains (we can automatically delay execute call), but first we need to bring measures for that.
- Caching of values - when we do cyclic subscriptions or emulation of these through passive connections. There is no way to make caller code aware of nature of data received. Unless he code everything on his end.
- Per operation timeout settings/automatic retry are another functionalities which are relevant in case of querying and writing. We do not want every driver to implement it, so making a metadata and/or decorator would unify this layer between drivers.

I speak of above from my own perspective cause I did implement an experimental decorator API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven to work [3] independently of driver. Sadly it has no measures to signal its execution back to caller. In the end, traceability is pretty tough.

Obviously OPC UA has everything sorted out, but device specific protocols often do not. Even a limited set of metadata can give us a way to start bridging the gap and making Ben's Milo/OPC UA server much closer to a real thing.

Best,
Łukasz

[1]
https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
[2]
https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
[3]
https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java


On 08.03.2021 23:15, Matthias Milan Strljic wrote:
> Hi all,
> 
> i see there a point of how useful this meta information could be in 
> some scenarios. But i would rather go the route of chris. Because I 
> see there is more of a risk of an overcomplicated inhomogeneous api 
> structure. I would not say that all API should have the same java 
> structure on each platform but if you have for all the different 
> platforms other featuresets you could also just use partially the code 
> generation attempt to allow a separate meta info layer.
> 
> And there would be the question if that is worth the additional feature?
> I mean I know we could add especially on the OPC UA side some meta 
> information but is there more than the  actual source timestamp? And 
> if there is none, would it be not better to use it as the actual 
> timestamp of the value?
> 
> 
> Einen schönen Abend ;)
> Greetings Matthias
> 
> 
> 
> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
> christofer.dutz@c-ware.de>:
> 
>> Hi,
>>
>> I'm generally more concerned about users expecting us to deliver 
>> feature that driver X has for driver Y too.
>>
>> I won't object, if you think it's worth doing.
>>
>> Perhaps if you could whip up an example in a feature branch? I think 
>> perhaps I was still not understanding what you propose.
>>
>> Would that be ok?
>>
>>
>> Chris
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Lukas Ott <ot...@gmail.com>
>> Gesendet: Montag, 8. März 2021 14:20
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> +1 to Lukasz
>>
>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>> luke@code-house.org>:
>>
>>> Hold on for a second.
>>>
>>> Making options pushed over connection string is a long term recipe 
>>> for disaster. Apache Camel is a prime example of what could happen 
>>> if you starting with configuration with URIs and dynamic parameters. 
>>> Most of components distinguish producer (writer) and consumer 
>>> (receiver) options, most complicated components ended up with 40+ 
>>> options settable via URI. Having a mix of PlcValues which are 
>>> timestamped and
>> 'qualified'
>>> and not will complicate encoding logic (how we push above info to 
>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>> compatible with above?
>>>
>>> That is why I rather opt for extending API in a way which does not 
>>> force existing drivers to do anything. When any of community 
>>> members, regardless if its developer or user, will find it necessary 
>>> to be ported in another driver, she or he have right to do it. It is 
>>> not only yours duty to keep all drivers and languages the same.
>>>
>>> Speaking of which I am also not concerned about making all languages 
>>> having the same functionality as languages tend to offer different 
>>> options which might be hard to be ported. Lets stick to basics which 
>>> are same (connection strings, field definitions, 
>>> read/write/subscribe
>>> syntax) and let various runtime lead their own ways toward
>> implementation.
>>>
>>> Key point is simple - we won't be able to maintain all languages 
>>> equally. We should not hold moving ie. go or java forward because we 
>>> can't port new API functionality to C/Python/C#/whenever. If there 
>>> is someone who *does* use above in production and *wish* to have new 
>>> API parts then that person/organization duty is to do it.
>>>
>>> Best,
>>> Łukasz
>>>
>>>
>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>> Hi,
>>>>
>>>> I must admit that I would be in favor of keeping it as simple as
>>> possible (max age in the connection string) and to implement more of 
>>> the missing parts in plc4x (like the subscription simulation layer) 
>>> and hereby getting the drivers we have a bit more aligned, than to 
>>> implement more and more special API features, that only one or two 
>>> drivers support. Cause this way we're running away form the 
>>> project-promise of running a given program at any type of PLC with 
>>> any
>> protocol.
>>>>
>>>> I mean ... at the current velocity the project is underway, I 
>>>> wouldn't
>>> really want to add a new MetaData layer that probably in the end 
>>> only
>>> 1-2 people will be implementing and which we must not only find out 
>>> how to fill with life in the first place, but also port to all of 
>>> our supported languages.
>>>>
>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>> fine
>>> with that ... or we add it to some wish-list for future extensions.
>>> But I wouldn't want to start something like this right now.
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>> Gesendet: Montag, 8. März 2021 13:07
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Source Time of PLC Value?
>>>>
>>>> I have not mentioned metadata term in my answer, so credits for 
>>>> mining
>>> it API go to you. :-)
>>>>
>>>> Now since you brought it I remember that JDBC has a "result set
>>> metadata". That might be thing we still miss. The result code is 
>>> most important and available instantly, other information is 
>>> supplemental and can be read additionally.
>>>>
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>> future.
>>>>>
>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>> <ch...@c-ware.de>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> sorry for being late to the party ... KNX is currently consuming 
>>>>>> all my cycles.
>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>
>>>>>> Initially I thought about adding this sort of information to the 
>>>>>> API, but then I thought that we have so little protocols 
>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>
>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>> default age a value can have in the cache and to make it possible 
>>>>>> to override this in the connection string options. If a value is 
>>>>>> too old,
>>> a return code of:
>>>>>> STALE_DATA could be returned.
>>>>>>
>>>>>> Also, we do have an option of adding an additional set of 
>>>>>> PlcValue
>>> types.
>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>> could return special versions of PlcValues, that have these
>> properties?
>>>>>>
>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>> protocols that don't support this concept (We could even 
>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>> some constants to the PlcValues all have the same API.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>
>>>>>> Yet looking at present state of API we are unlikely to make each 
>>>>>> and every PlcValue in each and every variant (qualified and 
>>>>>> timestamped, timestamped, qualified). It would force quite a lot 
>>>>>> of
>> wrapping.
>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>> additional method to retrieve all markers. For most of drivers it 
>>>>>> would return just an empty set. It leaves us also an open door 
>>>>>> for future drivers (or devices) to ship additional piece of 
>>>>>> metadata which
>>> does not fit into present API.
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>>
>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>
>>>>>>> uint64_t timestamp;
>>>>>>> typedef enum {
>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>       SOURCE,
>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>       GENERATED,
>>>>>>> } TimestampQuality;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>
>>>>>>>> Timestamps I think are definitely a good idea, especially if we 
>>>>>>>> eventually DNP3.
>>>>>>>>
>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>> <lu...@code-house.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You have response code for each individual field which allows 
>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Łukasz
>>>>>>>>>
>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>> indicator of a plc value?
>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>
>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andreas
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>

AW: AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Lukas,

for me the situation is more like this: 

I have been working on PLC4X for 3,5 years full time now. If I sum up the paid gigs I got for the Java stuff in that time, I would probably be a beggar if codecentric hadn't been paying my bills. I am working on Go, because I am paid for it because the company, I work for, uses it (and not the Java version, because Java is too fat for most embedded edge gateways). I implemented the C version because it was a paid, by the EU funded project. I even started on PLC4Net because it looked like we would be needing that, but now we're using Go in the could too, but there are people just getting started working on this. 

I know I put a lot of additional work into PLC4X, way more than I'm paid for, but generally I implement what people want and prioritize on those willing to spend a few bucks. Right now, I know about way more companies that told me: Contact me when you have C, C#, ... than for Java. 

I never said that you shouldn't extend the API with Metadata as you propose, I just said that I personally would like the API a bit more streamlined and us to think of ways to implement it, so the functionalities and APIs aren't completely different - There were ways to implement things before Generics. You never came up with an alternative proposal, so my opinion didn't change. I'm definitely not going to try to come up with an alternative, as I don't see the point of the extension to start with. You also tend to say: This path leads to disaster, have a look at Camel. I have to admit, I like Camel :-)

Also, there's a huge difference between changing an API and breaking an API. If later on we decide to add new features to the API, this doesn't necessarily break anything. 

And I never said I would block your proposal. I just pointed out some things that came to mind on it and I said that I'm probably not willing to invest my free time in something I don't see as important as you do. 

You say Java is most crowded ... well definitely there are more people involved in the project that could work on Java, but let's face it ... most aren't doing very much recently. The last few days I was having lots of fun with Sebastian who's really interested in the Go stuff.

Chris



-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Donnerstag, 11. März 2021 19:14
An: dev@plc4x.apache.org
Betreff: Re: AW: Source Time of PLC Value?

Hey Matthias and Chris,
I agree with you about complexity and alignment. I am far from trying to cover each and every case. At the same time I am aware that making thing fine in first run is crucial to avoid major changes afterwards. That's also why I did throw pseudo-code upfront to brainstorm. Leaving it undiscussed could lead to complicated implementation and also usage.

While I do appreciate work made to bring all (C/Python/C#?) languages under Apache PLC4X umbrella I also do see that sometimes they are being a baggage we carry on for no real use. When I say no use I mean that these are not officially adopted by anyone except Chris who made most of the work. Chris can't keep up all these alone. If theese are adopted then obviously users can and should take care of them given very limited resources we have.
Above leads us to situation where Java which is simplest (or most
crowded) part of our project can't move much forward. Java API can not extend its functionality because there are 2 or 3 other languages to carry on, with very few people who can do it. Any bigger step involving MSpec is a trouble forcing update of generators for few languages, holding us hostage of undisclosed use.

Isn't it a time to make sure that C/C#/Python can still compile with 0.8 release?  I feel it is a time to move forward with features we could ship in 0.9.
And yes, effectively what I do propose is making unused languages/runtimes a lower priority. We never received external inquiry of these (at least here), never had external contribution and barely have docs for these. Despite of huge investment made they are not picking up to speed.
If we focus on languages which been adopted and do not divide ourselves too much we can still attract people. When time comes we will help adopters who use different programming languages to close the gap.

These are my 0.02€.

Best,
Łukasz

On 11.03.2021 14:44, Matthias Milan Strljic wrote:
> Hi Lukasz,
> 
> I see your point there and I am not against "fatter" data structure either.
> My problem is just that we should not make it complicated to use, but 
> mainly to implement. And if we add to go the route with additional 
> metadata, I would consider the need for a policy to homogenize certain 
> properties. I hope I wasn't a bit rude in my reply earlier and I'm not 
> against the idea of metadata either. Because when we start to make 
> deeper features optional on the one hand and different between 
> protocols or programming platforms on the other hand, the charm of 
> using PLC4X instead of a native driver gets increasingly lost. So I'm 
> actually +1 for the idea, but critical about the implementation. I 
> just love simplicity :D
> 
> Greetings the simple boy Matthias
> 
> 
> Am Mi., 10. März 2021 um 12:58 Uhr schrieb Łukasz Dywicki <
> luke@code-house.org>:
> 
>> Matthias,
>> I do believe there are several points which are relevant and still 
>> not possible to be implemented using available PLC4X API. Metadata 
>> might not be most fortunate way, but it can enable or ease analytical 
>> and troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know 
>> response latency? This is especially relevant with frequent polling. 
>> Currently caller has no measures to know that and can not measure 
>> that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback 
>> chains (we can automatically delay execute call), but first we need 
>> to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of 
>> these through passive connections. There is no way to make caller 
>> code aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another 
>> functionalities which are relevant in case of querying and writing. 
>> We do not want every driver to implement it, so making a metadata 
>> and/or decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an 
>> experimental decorator API for PLC4X [1] to cope with CANopen 
>> arbitration issues I found in devices we interface with. It allows to 
>> catch read/write failures [2] and re-attempt entire operation. It is 
>> proven to work [3] independently of driver. Sadly it has no measures 
>> to signal its execution back to caller. In the end, traceability is 
>> pretty tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific 
>> protocols often do not. Even a limited set of metadata can give us a 
>> way to start bridging the gap and making Ben's Milo/OPC UA server 
>> much closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>>
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles
>> /org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/
>> decorator
>> [2]
>>
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> /org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/
>> plc4x/decorator/retry/ReadRetry.java
>> [3]
>>
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles
>> /org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/
>> plc4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there
>> is
>>> more of a risk of an overcomplicated inhomogeneous api structure. I 
>>> would not say that all API should have the same java structure on 
>>> each platform but if you have for all the different platforms other 
>>> featuresets you
>> could
>>> also just use partially the code generation attempt to allow a 
>>> separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp
>> of
>>> the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I 
>>>> think perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic 
>>>>> parameters. Most of components distinguish producer (writer) and 
>>>>> consumer (receiver) options, most complicated components ended up 
>>>>> with 40+ options settable via URI. Having a mix of PlcValues which 
>>>>> are timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation 
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only 
>>>>> one or two drivers support. Cause this way we're running away form 
>>>>> the project-promise of running a given program at any type of PLC 
>>>>> with any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find 
>>>>> out how to fill with life in the first place, but also port to all 
>>>>> of our supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm 
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to 
>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in 
>>>>>>> the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently 
>>>>>>>> consuming all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information to 
>>>>>>>> the API, but then I thought that we have so little protocols 
>>>>>>>> supporting this sort of concept that blowing up every PLCValue 
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the 
>>>>>>>> background, I thought it would be a valid solution to provide a 
>>>>>>>> default age a value can have in the cache and to make it 
>>>>>>>> possible to override this in the connection string options. If 
>>>>>>>> a value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of 
>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for 
>>>>>>>> protocols that don't support this concept (We could even 
>>>>>>>> implement the methods on the default PlvValues to simply return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make 
>>>>>>>> each and every PlcValue in each and every variant (qualified 
>>>>>>>> and timestamped, timestamped, qualified). It would force quite 
>>>>>>>> a lot of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with 
>>>>>>>> additional method to retrieve all markers. For most of drivers 
>>>>>>>> it would return just an empty set. It leaves us also an open 
>>>>>>>> door for future drivers (or devices) to ship additional piece 
>>>>>>>> of metadata which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson 
>>>>>>>>>> <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field which 
>>>>>>>>>>> allows you to determine state (OK, REMOTE_ERROR etc.). Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: AW: Source Time of PLC Value?

Posted by Stephen Snow <s4...@gmail.com>.

On Thu, 2021-03-11 at 19:13 +0100, Łukasz Dywicki wrote:
> [snip] ..

> While I do appreciate work made to bring all (C/Python/C#?) languages
> under Apache PLC4X umbrella I also do see that sometimes they are
> being
> {snip]

> Isn't it a time to make sure that C/C#/Python can still compile with
> 0.8
> release?  
Is no one else supporting these beside Chris? That is seemingly unfair.
I would be willing to try some work with Python, what version is in use
in the PLC4X?

[great big snip]

I am more familiar with Java and prefer it, but since I am a Fedora guy
I use Python on occasion, currently Python 3, but 2 is still doable if
necessary. I would be willing to try to help in this area if there is
lacking support.

Regards,
Stephen

Re: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Hey Matthias and Chris,
I agree with you about complexity and alignment. I am far from trying to
cover each and every case. At the same time I am aware that making thing
fine in first run is crucial to avoid major changes afterwards. That's
also why I did throw pseudo-code upfront to brainstorm. Leaving it
undiscussed could lead to complicated implementation and also usage.

While I do appreciate work made to bring all (C/Python/C#?) languages
under Apache PLC4X umbrella I also do see that sometimes they are being
a baggage we carry on for no real use. When I say no use I mean that
these are not officially adopted by anyone except Chris who made most of
the work. Chris can't keep up all these alone. If theese are adopted
then obviously users can and should take care of them given very limited
resources we have.
Above leads us to situation where Java which is simplest (or most
crowded) part of our project can't move much forward. Java API can not
extend its functionality because there are 2 or 3 other languages to
carry on, with very few people who can do it. Any bigger step involving
MSpec is a trouble forcing update of generators for few languages,
holding us hostage of undisclosed use.

Isn't it a time to make sure that C/C#/Python can still compile with 0.8
release?  I feel it is a time to move forward with features we could
ship in 0.9.
And yes, effectively what I do propose is making unused
languages/runtimes a lower priority. We never received external inquiry
of these (at least here), never had external contribution and barely
have docs for these. Despite of huge investment made they are not
picking up to speed.
If we focus on languages which been adopted and do not divide ourselves
too much we can still attract people. When time comes we will help
adopters who use different programming languages to close the gap.

These are my 0.02€.

Best,
Łukasz

On 11.03.2021 14:44, Matthias Milan Strljic wrote:
> Hi Lukasz,
> 
> I see your point there and I am not against "fatter" data structure either.
> My problem is just that we should not make it complicated to use, but
> mainly to implement. And if we add to go the route with additional
> metadata, I would consider the need for a policy to homogenize certain
> properties. I hope I wasn't a bit rude in my reply earlier and I'm not
> against the idea of metadata either. Because when we start to make deeper
> features optional on the one hand and different between protocols or
> programming platforms on the other hand, the charm of using PLC4X instead
> of a native driver gets increasingly lost. So I'm actually +1 for the idea,
> but critical about the implementation. I just love simplicity :D
> 
> Greetings the simple boy Matthias
> 
> 
> Am Mi., 10. März 2021 um 12:58 Uhr schrieb Łukasz Dywicki <
> luke@code-house.org>:
> 
>> Matthias,
>> I do believe there are several points which are relevant and still not
>> possible to be implemented using available PLC4X API. Metadata might not
>> be most fortunate way, but it can enable or ease analytical and
>> troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response
>> latency? This is especially relevant with frequent polling. Currently
>> caller has no measures to know that and can not measure that in reliable
>> way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains
>> (we can automatically delay execute call), but first we need to bring
>> measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of
>> these through passive connections. There is no way to make caller code
>> aware of nature of data received. Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another
>> functionalities which are relevant in case of querying and writing. We
>> do not want every driver to implement it, so making a metadata and/or
>> decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an
>> experimental decorator API for PLC4X [1] to cope with CANopen
>> arbitration issues I found in devices we interface with. It allows to
>> catch read/write failures [2] and re-attempt entire operation. It is
>> proven to work [3] independently of driver. Sadly it has no measures to
>> signal its execution back to caller. In the end, traceability is pretty
>> tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific
>> protocols often do not. Even a limited set of metadata can give us a way
>> to start bridging the gap and making Ben's Milo/OPC UA server much
>> closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>>
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
>> [2]
>>
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
>> [3]
>>
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in some
>>> scenarios. But i would rather go the route of chris. Because I see there
>> is
>>> more of a risk of an overcomplicated inhomogeneous api structure. I would
>>> not say that all API should have the same java structure on each platform
>>> but if you have for all the different platforms other featuresets you
>> could
>>> also just use partially the code generation attempt to allow a
>>> separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta
>>> information but is there more than the  actual source timestamp? And if
>>> there is none, would it be not better to use it as the actual timestamp
>> of
>>> the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver feature
>>>> that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I think
>>>> perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ot...@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe for
>>>>> disaster. Apache Camel is a prime example of what could happen if you
>>>>> starting with configuration with URIs and dynamic parameters. Most of
>>>>> components distinguish producer (writer) and consumer (receiver)
>>>>> options, most complicated components ended up with 40+ options
>>>>> settable via URI. Having a mix of PlcValues which are timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to IEC
>>>>> encoder/handler or decorate it?) and how we keep custom types
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not
>>>>> force existing drivers to do anything. When any of community members,
>>>>> regardless if its developer or user, will find it necessary to be
>>>>> ported in another driver, she or he have right to do it. It is not
>>>>> only yours duty to keep all drivers and languages the same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all languages
>>>>> having the same functionality as languages tend to offer different
>>>>> options which might be hard to be ported. Lets stick to basics which
>>>>> are same (connection strings, field definitions, read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages
>>>>> equally. We should not hold moving ie. go or java forward because we
>>>>> can't port new API functionality to C/Python/C#/whenever. If there is
>>>>> someone who *does* use above in production and *wish* to have new API
>>>>> parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more of
>>>>> the missing parts in plc4x (like the subscription simulation layer)
>>>>> and hereby getting the drivers we have a bit more aligned, than to
>>>>> implement more and more special API features, that only one or two
>>>>> drivers support. Cause this way we're running away form the
>>>>> project-promise of running a given program at any type of PLC with any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end only
>>>>> 1-2 people will be implementing and which we must not only find out
>>>>> how to fill with life in the first place, but also port to all of our
>>>>> supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm
>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is most
>>>>> important and available instantly, other information is supplemental
>>>>> and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
>>>>>>> <ch...@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently consuming
>>>>>>>> all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information to the
>>>>>>>> API, but then I thought that we have so little protocols
>>>>>>>> supporting this sort of concept that blowing up every PLCValue
>>>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the
>>>>>>>> background, I thought it would be a valid solution to provide a
>>>>>>>> default age a value can have in the cache and to make it possible
>>>>>>>> to override this in the connection string options. If a value is
>>>>>>>> too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data, we
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for
>>>>>>>> protocols that don't support this concept (We could even implement
>>>>>>>> the methods on the default PlvValues to simply return some
>>>>>>>> constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make each
>>>>>>>> and every PlcValue in each and every variant (qualified and
>>>>>>>> timestamped, timestamped, qualified). It would force quite a lot of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API with
>>>>>>>> additional method to retrieve all markers. For most of drivers it
>>>>>>>> would return just an empty set. It leaves us also an open door for
>>>>>>>> future drivers (or devices) to ship additional piece of metadata
>>>>>>>> which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need to be
>>>>>>>>>> extended a bit. I don't think we have one for BAD (we still
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially if we
>>>>>>>>>> eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
>>>>>>>>>> <lu...@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field which allows
>>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
>>>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: AW: Source Time of PLC Value?

Posted by Matthias Milan Strljic <ma...@gmail.com>.

Hi Lukasz,

I see your point there and I am not against "fatter" data structure either.
My problem is just that we should not make it complicated to use, but
mainly to implement. And if we add to go the route with additional
metadata, I would consider the need for a policy to homogenize certain
properties. I hope I wasn't a bit rude in my reply earlier and I'm not
against the idea of metadata either. Because when we start to make deeper
features optional on the one hand and different between protocols or
programming platforms on the other hand, the charm of using PLC4X instead
of a native driver gets increasingly lost. So I'm actually +1 for the idea,
but critical about the implementation. I just love simplicity :D

Greetings the simple boy Matthias


Am Mi., 10. März 2021 um 12:58 Uhr schrieb Łukasz Dywicki <
luke@code-house.org>:

> Matthias,
> I do believe there are several points which are relevant and still not
> possible to be implemented using available PLC4X API. Metadata might not
> be most fortunate way, but it can enable or ease analytical and
> troubleshooting scenarios.
>
> Two major points I see:
> - Verification of device condition - ie. how does caller know response
> latency? This is especially relevant with frequent polling. Currently
> caller has no measures to know that and can not measure that in reliable
> way due to async nature of API.
> There is a way to implement backpressure technics with callback chains
> (we can automatically delay execute call), but first we need to bring
> measures for that.
> - Caching of values - when we do cyclic subscriptions or emulation of
> these through passive connections. There is no way to make caller code
> aware of nature of data received. Unless he code everything on his end.
> - Per operation timeout settings/automatic retry are another
> functionalities which are relevant in case of querying and writing. We
> do not want every driver to implement it, so making a metadata and/or
> decorator would unify this layer between drivers.
>
> I speak of above from my own perspective cause I did implement an
> experimental decorator API for PLC4X [1] to cope with CANopen
> arbitration issues I found in devices we interface with. It allows to
> catch read/write failures [2] and re-attempt entire operation. It is
> proven to work [3] independently of driver. Sadly it has no measures to
> signal its execution back to caller. In the end, traceability is pretty
> tough.
>
> Obviously OPC UA has everything sorted out, but device specific
> protocols often do not. Even a limited set of metadata can give us a way
> to start bridging the gap and making Ben's Milo/OPC UA server much
> closer to a real thing.
>
> Best,
> Łukasz
>
> [1]
>
> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
> [2]
>
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
> [3]
>
> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java
>
>
> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
> > Hi all,
> >
> > i see there a point of how useful this meta information could be in some
> > scenarios. But i would rather go the route of chris. Because I see there
> is
> > more of a risk of an overcomplicated inhomogeneous api structure. I would
> > not say that all API should have the same java structure on each platform
> > but if you have for all the different platforms other featuresets you
> could
> > also just use partially the code generation attempt to allow a
> > separate meta info layer.
> >
> > And there would be the question if that is worth the additional feature?
> > I mean I know we could add especially on the OPC UA side some meta
> > information but is there more than the  actual source timestamp? And if
> > there is none, would it be not better to use it as the actual timestamp
> of
> > the value?
> >
> >
> > Einen schönen Abend ;)
> > Greetings Matthias
> >
> >
> >
> > Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
> > christofer.dutz@c-ware.de>:
> >
> >> Hi,
> >>
> >> I'm generally more concerned about users expecting us to deliver feature
> >> that driver X has for driver Y too.
> >>
> >> I won't object, if you think it's worth doing.
> >>
> >> Perhaps if you could whip up an example in a feature branch? I think
> >> perhaps I was still not understanding what you propose.
> >>
> >> Would that be ok?
> >>
> >>
> >> Chris
> >>
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Lukas Ott <ot...@gmail.com>
> >> Gesendet: Montag, 8. März 2021 14:20
> >> An: dev@plc4x.apache.org
> >> Betreff: Re: AW: Source Time of PLC Value?
> >>
> >> +1 to Lukasz
> >>
> >> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
> >> luke@code-house.org>:
> >>
> >>> Hold on for a second.
> >>>
> >>> Making options pushed over connection string is a long term recipe for
> >>> disaster. Apache Camel is a prime example of what could happen if you
> >>> starting with configuration with URIs and dynamic parameters. Most of
> >>> components distinguish producer (writer) and consumer (receiver)
> >>> options, most complicated components ended up with 40+ options
> >>> settable via URI. Having a mix of PlcValues which are timestamped and
> >> 'qualified'
> >>> and not will complicate encoding logic (how we push above info to IEC
> >>> encoder/handler or decorate it?) and how we keep custom types
> >>> compatible with above?
> >>>
> >>> That is why I rather opt for extending API in a way which does not
> >>> force existing drivers to do anything. When any of community members,
> >>> regardless if its developer or user, will find it necessary to be
> >>> ported in another driver, she or he have right to do it. It is not
> >>> only yours duty to keep all drivers and languages the same.
> >>>
> >>> Speaking of which I am also not concerned about making all languages
> >>> having the same functionality as languages tend to offer different
> >>> options which might be hard to be ported. Lets stick to basics which
> >>> are same (connection strings, field definitions, read/write/subscribe
> >>> syntax) and let various runtime lead their own ways toward
> >> implementation.
> >>>
> >>> Key point is simple - we won't be able to maintain all languages
> >>> equally. We should not hold moving ie. go or java forward because we
> >>> can't port new API functionality to C/Python/C#/whenever. If there is
> >>> someone who *does* use above in production and *wish* to have new API
> >>> parts then that person/organization duty is to do it.
> >>>
> >>> Best,
> >>> Łukasz
> >>>
> >>>
> >>> On 08.03.2021 13:17, Christofer Dutz wrote:
> >>>> Hi,
> >>>>
> >>>> I must admit that I would be in favor of keeping it as simple as
> >>> possible (max age in the connection string) and to implement more of
> >>> the missing parts in plc4x (like the subscription simulation layer)
> >>> and hereby getting the drivers we have a bit more aligned, than to
> >>> implement more and more special API features, that only one or two
> >>> drivers support. Cause this way we're running away form the
> >>> project-promise of running a given program at any type of PLC with any
> >> protocol.
> >>>>
> >>>> I mean ... at the current velocity the project is underway, I
> >>>> wouldn't
> >>> really want to add a new MetaData layer that probably in the end only
> >>> 1-2 people will be implementing and which we must not only find out
> >>> how to fill with life in the first place, but also port to all of our
> >>> supported languages.
> >>>>
> >>>> If someone's willing to bring in some serious man/womenpower, I'm
> >>>> fine
> >>> with that ... or we add it to some wish-list for future extensions.
> >>> But I wouldn't want to start something like this right now.
> >>>>
> >>>> Chris
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>>> Gesendet: Montag, 8. März 2021 13:07
> >>>> An: dev@plc4x.apache.org
> >>>> Betreff: Re: Source Time of PLC Value?
> >>>>
> >>>> I have not mentioned metadata term in my answer, so credits for
> >>>> mining
> >>> it API go to you. :-)
> >>>>
> >>>> Now since you brought it I remember that JDBC has a "result set
> >>> metadata". That might be thing we still miss. The result code is most
> >>> important and available instantly, other information is supplemental
> >>> and can be read additionally.
> >>>>
> >>>>
> >>>> Best,
> >>>> Łukasz
> >>>>
> >>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
> >>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
> >>>>> duplicate PLcValue classes and give us a lot of flexibility in the
> >>> future.
> >>>>>
> >>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
> >>>>> <ch...@c-ware.de>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> sorry for being late to the party ... KNX is currently consuming
> >>>>>> all my cycles.
> >>>>>> Just wanted to add my thoughts to the discussion.
> >>>>>>
> >>>>>> Initially I thought about adding this sort of information to the
> >>>>>> API, but then I thought that we have so little protocols
> >>>>>> supporting this sort of concept that blowing up every PLCValue
> >>>>>> with this time information just would waste CPU time and Memory.
> >>>>>>
> >>>>>> When it comes to simulating subscriptions by polling in the
> >>>>>> background, I thought it would be a valid solution to provide a
> >>>>>> default age a value can have in the cache and to make it possible
> >>>>>> to override this in the connection string options. If a value is
> >>>>>> too old,
> >>> a return code of:
> >>>>>> STALE_DATA could be returned.
> >>>>>>
> >>>>>> Also, we do have an option of adding an additional set of PlcValue
> >>> types.
> >>>>>> If a driver supports this quality-of-service type of data, we
> >>>>>> could return special versions of PlcValues, that have these
> >> properties?
> >>>>>>
> >>>>>> This way we don't have to waste the CPU time and Memory for
> >>>>>> protocols that don't support this concept (We could even implement
> >>>>>> the methods on the default PlvValues to simply return some
> >>>>>> constants to the PlcValues all have the same API.
> >>>>>>
> >>>>>> Chris
> >>>>>>
> >>>>>>
> >>>>>> -----Ursprüngliche Nachricht-----
> >>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>>>>> Gesendet: Montag, 8. März 2021 10:43
> >>>>>> An: dev@plc4x.apache.org
> >>>>>> Betreff: Re: Source Time of PLC Value?
> >>>>>>
> >>>>>> Looking at issue we could utilize marker interfaces.
> >>>>>> At least in Java we could define Qualified and Timestamped types.
> >>>>>>
> >>>>>> Yet looking at present state of API we are unlikely to make each
> >>>>>> and every PlcValue in each and every variant (qualified and
> >>>>>> timestamped, timestamped, qualified). It would force quite a lot of
> >> wrapping.
> >>>>>> Safest option I see is to extend subscription/response API with
> >>>>>> additional method to retrieve all markers. For most of drivers it
> >>>>>> would return just an empty set. It leaves us also an open door for
> >>>>>> future drivers (or devices) to ship additional piece of metadata
> >>>>>> which
> >>> does not fit into present API.
> >>>>>>
> >>>>>> Best,
> >>>>>> Łukasz
> >>>>>>
> >>>>>>
> >>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
> >>>>>>> So, maybe the extension is timestamp *and* timestamp source.
> >>>>>>>
> >>>>>>> uint64_t timestamp;
> >>>>>>> typedef enum {
> >>>>>>>       /* timestamp was generated by the source device */
> >>>>>>>       SOURCE,
> >>>>>>>       /* timestamp was generated by plc4x on receiving */
> >>>>>>>       GENERATED,
> >>>>>>> } TimestampQuality;
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> We do have some error codes, but it would just need to be
> >>>>>>>> extended a bit. I don't think we have one for BAD (we still
> >>>>>>>> receive data but the source has marked it bad).
> >>>>>>>>
> >>>>>>>> Timestamps I think are definitely a good idea, especially if we
> >>>>>>>> eventually DNP3.
> >>>>>>>>
> >>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
> >>>>>>>> <lu...@code-house.org>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> You have response code for each individual field which allows
> >>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same
> >>>>>>>>> information should be available also for subscriptions.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Łukasz
> >>>>>>>>>
> >>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
> >>>>>>>>>> Ah, forgot: and is there a status available?
> >>>>>>>>>> Valid/invalid/good/bad
> >>>>>>>>> indicator of a plc value?
> >>>>>>>>>>
> >>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
> >>>>>>>>>>> <an...@me.com.INVALID>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> How can I get the source time of a PlcValue?
> >>>>>>>>>>>
> >>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
> >>>>>>>>>>> b) from a read request  PlcReadResponse
> >>>>>>>>>>>
> >>>>>>>>>>> Best regards,
> >>>>>>>>>>> Andreas
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> >
>

Re: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Matthias,
I do believe there are several points which are relevant and still not
possible to be implemented using available PLC4X API. Metadata might not
be most fortunate way, but it can enable or ease analytical and
troubleshooting scenarios.

Two major points I see:
- Verification of device condition - ie. how does caller know response
latency? This is especially relevant with frequent polling. Currently
caller has no measures to know that and can not measure that in reliable
way due to async nature of API.
There is a way to implement backpressure technics with callback chains
(we can automatically delay execute call), but first we need to bring
measures for that.
- Caching of values - when we do cyclic subscriptions or emulation of
these through passive connections. There is no way to make caller code
aware of nature of data received. Unless he code everything on his end.
- Per operation timeout settings/automatic retry are another
functionalities which are relevant in case of querying and writing. We
do not want every driver to implement it, so making a metadata and/or
decorator would unify this layer between drivers.

I speak of above from my own perspective cause I did implement an
experimental decorator API for PLC4X [1] to cope with CANopen
arbitration issues I found in devices we interface with. It allows to
catch read/write failures [2] and re-attempt entire operation. It is
proven to work [3] independently of driver. Sadly it has no measures to
signal its execution back to caller. In the end, traceability is pretty
tough.

Obviously OPC UA has everything sorted out, but device specific
protocols often do not. Even a limited set of metadata can give us a way
to start bridging the gap and making Ben's Milo/OPC UA server much
closer to a real thing.

Best,
Łukasz

[1]
https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/decorator
[2]
https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/plc4x/decorator/retry/ReadRetry.java
[3]
https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/plc4x/decorator/retry/RetryDecoratorReadTest.java


On 08.03.2021 23:15, Matthias Milan Strljic wrote:
> Hi all,
> 
> i see there a point of how useful this meta information could be in some
> scenarios. But i would rather go the route of chris. Because I see there is
> more of a risk of an overcomplicated inhomogeneous api structure. I would
> not say that all API should have the same java structure on each platform
> but if you have for all the different platforms other featuresets you could
> also just use partially the code generation attempt to allow a
> separate meta info layer.
> 
> And there would be the question if that is worth the additional feature?
> I mean I know we could add especially on the OPC UA side some meta
> information but is there more than the  actual source timestamp? And if
> there is none, would it be not better to use it as the actual timestamp of
> the value?
> 
> 
> Einen schönen Abend ;)
> Greetings Matthias
> 
> 
> 
> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
> christofer.dutz@c-ware.de>:
> 
>> Hi,
>>
>> I'm generally more concerned about users expecting us to deliver feature
>> that driver X has for driver Y too.
>>
>> I won't object, if you think it's worth doing.
>>
>> Perhaps if you could whip up an example in a feature branch? I think
>> perhaps I was still not understanding what you propose.
>>
>> Would that be ok?
>>
>>
>> Chris
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Lukas Ott <ot...@gmail.com>
>> Gesendet: Montag, 8. März 2021 14:20
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> +1 to Lukasz
>>
>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>> luke@code-house.org>:
>>
>>> Hold on for a second.
>>>
>>> Making options pushed over connection string is a long term recipe for
>>> disaster. Apache Camel is a prime example of what could happen if you
>>> starting with configuration with URIs and dynamic parameters. Most of
>>> components distinguish producer (writer) and consumer (receiver)
>>> options, most complicated components ended up with 40+ options
>>> settable via URI. Having a mix of PlcValues which are timestamped and
>> 'qualified'
>>> and not will complicate encoding logic (how we push above info to IEC
>>> encoder/handler or decorate it?) and how we keep custom types
>>> compatible with above?
>>>
>>> That is why I rather opt for extending API in a way which does not
>>> force existing drivers to do anything. When any of community members,
>>> regardless if its developer or user, will find it necessary to be
>>> ported in another driver, she or he have right to do it. It is not
>>> only yours duty to keep all drivers and languages the same.
>>>
>>> Speaking of which I am also not concerned about making all languages
>>> having the same functionality as languages tend to offer different
>>> options which might be hard to be ported. Lets stick to basics which
>>> are same (connection strings, field definitions, read/write/subscribe
>>> syntax) and let various runtime lead their own ways toward
>> implementation.
>>>
>>> Key point is simple - we won't be able to maintain all languages
>>> equally. We should not hold moving ie. go or java forward because we
>>> can't port new API functionality to C/Python/C#/whenever. If there is
>>> someone who *does* use above in production and *wish* to have new API
>>> parts then that person/organization duty is to do it.
>>>
>>> Best,
>>> Łukasz
>>>
>>>
>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>> Hi,
>>>>
>>>> I must admit that I would be in favor of keeping it as simple as
>>> possible (max age in the connection string) and to implement more of
>>> the missing parts in plc4x (like the subscription simulation layer)
>>> and hereby getting the drivers we have a bit more aligned, than to
>>> implement more and more special API features, that only one or two
>>> drivers support. Cause this way we're running away form the
>>> project-promise of running a given program at any type of PLC with any
>> protocol.
>>>>
>>>> I mean ... at the current velocity the project is underway, I
>>>> wouldn't
>>> really want to add a new MetaData layer that probably in the end only
>>> 1-2 people will be implementing and which we must not only find out
>>> how to fill with life in the first place, but also port to all of our
>>> supported languages.
>>>>
>>>> If someone's willing to bring in some serious man/womenpower, I'm
>>>> fine
>>> with that ... or we add it to some wish-list for future extensions.
>>> But I wouldn't want to start something like this right now.
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>> Gesendet: Montag, 8. März 2021 13:07
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Source Time of PLC Value?
>>>>
>>>> I have not mentioned metadata term in my answer, so credits for
>>>> mining
>>> it API go to you. :-)
>>>>
>>>> Now since you brought it I remember that JDBC has a "result set
>>> metadata". That might be thing we still miss. The result code is most
>>> important and available instantly, other information is supplemental
>>> and can be read additionally.
>>>>
>>>>
>>>> Best,
>>>> Łukasz
>>>>
>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>>>>> duplicate PLcValue classes and give us a lot of flexibility in the
>>> future.
>>>>>
>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
>>>>> <ch...@c-ware.de>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> sorry for being late to the party ... KNX is currently consuming
>>>>>> all my cycles.
>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>
>>>>>> Initially I thought about adding this sort of information to the
>>>>>> API, but then I thought that we have so little protocols
>>>>>> supporting this sort of concept that blowing up every PLCValue
>>>>>> with this time information just would waste CPU time and Memory.
>>>>>>
>>>>>> When it comes to simulating subscriptions by polling in the
>>>>>> background, I thought it would be a valid solution to provide a
>>>>>> default age a value can have in the cache and to make it possible
>>>>>> to override this in the connection string options. If a value is
>>>>>> too old,
>>> a return code of:
>>>>>> STALE_DATA could be returned.
>>>>>>
>>>>>> Also, we do have an option of adding an additional set of PlcValue
>>> types.
>>>>>> If a driver supports this quality-of-service type of data, we
>>>>>> could return special versions of PlcValues, that have these
>> properties?
>>>>>>
>>>>>> This way we don't have to waste the CPU time and Memory for
>>>>>> protocols that don't support this concept (We could even implement
>>>>>> the methods on the default PlvValues to simply return some
>>>>>> constants to the PlcValues all have the same API.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>> At least in Java we could define Qualified and Timestamped types.
>>>>>>
>>>>>> Yet looking at present state of API we are unlikely to make each
>>>>>> and every PlcValue in each and every variant (qualified and
>>>>>> timestamped, timestamped, qualified). It would force quite a lot of
>> wrapping.
>>>>>> Safest option I see is to extend subscription/response API with
>>>>>> additional method to retrieve all markers. For most of drivers it
>>>>>> would return just an empty set. It leaves us also an open door for
>>>>>> future drivers (or devices) to ship additional piece of metadata
>>>>>> which
>>> does not fit into present API.
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>>
>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>>>>
>>>>>>> uint64_t timestamp;
>>>>>>> typedef enum {
>>>>>>>       /* timestamp was generated by the source device */
>>>>>>>       SOURCE,
>>>>>>>       /* timestamp was generated by plc4x on receiving */
>>>>>>>       GENERATED,
>>>>>>> } TimestampQuality;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> We do have some error codes, but it would just need to be
>>>>>>>> extended a bit. I don't think we have one for BAD (we still
>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>
>>>>>>>> Timestamps I think are definitely a good idea, especially if we
>>>>>>>> eventually DNP3.
>>>>>>>>
>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
>>>>>>>> <lu...@code-house.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You have response code for each individual field which allows
>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same
>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Łukasz
>>>>>>>>>
>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>> indicator of a plc value?
>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
>>>>>>>>>>> <an...@me.com.INVALID>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>
>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andreas
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>

Re: AW: Source Time of PLC Value?

Posted by Matthias Milan Strljic <ma...@gmail.com>.

Hi all,

i see there a point of how useful this meta information could be in some
scenarios. But i would rather go the route of chris. Because I see there is
more of a risk of an overcomplicated inhomogeneous api structure. I would
not say that all API should have the same java structure on each platform
but if you have for all the different platforms other featuresets you could
also just use partially the code generation attempt to allow a
separate meta info layer.

And there would be the question if that is worth the additional feature?
I mean I know we could add especially on the OPC UA side some meta
information but is there more than the  actual source timestamp? And if
there is none, would it be not better to use it as the actual timestamp of
the value?


Einen schönen Abend ;)
Greetings Matthias



Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
christofer.dutz@c-ware.de>:

> Hi,
>
> I'm generally more concerned about users expecting us to deliver feature
> that driver X has for driver Y too.
>
> I won't object, if you think it's worth doing.
>
> Perhaps if you could whip up an example in a feature branch? I think
> perhaps I was still not understanding what you propose.
>
> Would that be ok?
>
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Lukas Ott <ot...@gmail.com>
> Gesendet: Montag, 8. März 2021 14:20
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Source Time of PLC Value?
>
> +1 to Lukasz
>
> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
> luke@code-house.org>:
>
> > Hold on for a second.
> >
> > Making options pushed over connection string is a long term recipe for
> > disaster. Apache Camel is a prime example of what could happen if you
> > starting with configuration with URIs and dynamic parameters. Most of
> > components distinguish producer (writer) and consumer (receiver)
> > options, most complicated components ended up with 40+ options
> > settable via URI. Having a mix of PlcValues which are timestamped and
> 'qualified'
> > and not will complicate encoding logic (how we push above info to IEC
> > encoder/handler or decorate it?) and how we keep custom types
> > compatible with above?
> >
> > That is why I rather opt for extending API in a way which does not
> > force existing drivers to do anything. When any of community members,
> > regardless if its developer or user, will find it necessary to be
> > ported in another driver, she or he have right to do it. It is not
> > only yours duty to keep all drivers and languages the same.
> >
> > Speaking of which I am also not concerned about making all languages
> > having the same functionality as languages tend to offer different
> > options which might be hard to be ported. Lets stick to basics which
> > are same (connection strings, field definitions, read/write/subscribe
> > syntax) and let various runtime lead their own ways toward
> implementation.
> >
> > Key point is simple - we won't be able to maintain all languages
> > equally. We should not hold moving ie. go or java forward because we
> > can't port new API functionality to C/Python/C#/whenever. If there is
> > someone who *does* use above in production and *wish* to have new API
> > parts then that person/organization duty is to do it.
> >
> > Best,
> > Łukasz
> >
> >
> > On 08.03.2021 13:17, Christofer Dutz wrote:
> > > Hi,
> > >
> > > I must admit that I would be in favor of keeping it as simple as
> > possible (max age in the connection string) and to implement more of
> > the missing parts in plc4x (like the subscription simulation layer)
> > and hereby getting the drivers we have a bit more aligned, than to
> > implement more and more special API features, that only one or two
> > drivers support. Cause this way we're running away form the
> > project-promise of running a given program at any type of PLC with any
> protocol.
> > >
> > > I mean ... at the current velocity the project is underway, I
> > > wouldn't
> > really want to add a new MetaData layer that probably in the end only
> > 1-2 people will be implementing and which we must not only find out
> > how to fill with life in the first place, but also port to all of our
> > supported languages.
> > >
> > > If someone's willing to bring in some serious man/womenpower, I'm
> > > fine
> > with that ... or we add it to some wish-list for future extensions.
> > But I wouldn't want to start something like this right now.
> > >
> > > Chris
> > >
> > >
> > >
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Łukasz Dywicki <lu...@code-house.org>
> > > Gesendet: Montag, 8. März 2021 13:07
> > > An: dev@plc4x.apache.org
> > > Betreff: Re: Source Time of PLC Value?
> > >
> > > I have not mentioned metadata term in my answer, so credits for
> > > mining
> > it API go to you. :-)
> > >
> > > Now since you brought it I remember that JDBC has a "result set
> > metadata". That might be thing we still miss. The result code is most
> > important and available instantly, other information is supplemental
> > and can be read additionally.
> > >
> > >
> > > Best,
> > > Łukasz
> > >
> > > On 08.03.2021 13:01, Ben Hutcheson wrote:
> > >> +1 for Łukasz‘s metadata approach, it would avoid having to create
> > >> duplicate PLcValue classes and give us a lot of flexibility in the
> > future.
> > >>
> > >> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
> > >> <ch...@c-ware.de>
> > >> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> sorry for being late to the party ... KNX is currently consuming
> > >>> all my cycles.
> > >>> Just wanted to add my thoughts to the discussion.
> > >>>
> > >>> Initially I thought about adding this sort of information to the
> > >>> API, but then I thought that we have so little protocols
> > >>> supporting this sort of concept that blowing up every PLCValue
> > >>> with this time information just would waste CPU time and Memory.
> > >>>
> > >>> When it comes to simulating subscriptions by polling in the
> > >>> background, I thought it would be a valid solution to provide a
> > >>> default age a value can have in the cache and to make it possible
> > >>> to override this in the connection string options. If a value is
> > >>> too old,
> > a return code of:
> > >>> STALE_DATA could be returned.
> > >>>
> > >>> Also, we do have an option of adding an additional set of PlcValue
> > types.
> > >>> If a driver supports this quality-of-service type of data, we
> > >>> could return special versions of PlcValues, that have these
> properties?
> > >>>
> > >>> This way we don't have to waste the CPU time and Memory for
> > >>> protocols that don't support this concept (We could even implement
> > >>> the methods on the default PlvValues to simply return some
> > >>> constants to the PlcValues all have the same API.
> > >>>
> > >>> Chris
> > >>>
> > >>>
> > >>> -----Ursprüngliche Nachricht-----
> > >>> Von: Łukasz Dywicki <lu...@code-house.org>
> > >>> Gesendet: Montag, 8. März 2021 10:43
> > >>> An: dev@plc4x.apache.org
> > >>> Betreff: Re: Source Time of PLC Value?
> > >>>
> > >>> Looking at issue we could utilize marker interfaces.
> > >>> At least in Java we could define Qualified and Timestamped types.
> > >>>
> > >>> Yet looking at present state of API we are unlikely to make each
> > >>> and every PlcValue in each and every variant (qualified and
> > >>> timestamped, timestamped, qualified). It would force quite a lot of
> wrapping.
> > >>> Safest option I see is to extend subscription/response API with
> > >>> additional method to retrieve all markers. For most of drivers it
> > >>> would return just an empty set. It leaves us also an open door for
> > >>> future drivers (or devices) to ship additional piece of metadata
> > >>> which
> > does not fit into present API.
> > >>>
> > >>> Best,
> > >>> Łukasz
> > >>>
> > >>>
> > >>> On 08.03.2021 04:30, Otto Fowler wrote:
> > >>>> So, maybe the extension is timestamp *and* timestamp source.
> > >>>>
> > >>>> uint64_t timestamp;
> > >>>> typedef enum {
> > >>>>       /* timestamp was generated by the source device */
> > >>>>       SOURCE,
> > >>>>       /* timestamp was generated by plc4x on receiving */
> > >>>>       GENERATED,
> > >>>> } TimestampQuality;
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>> We do have some error codes, but it would just need to be
> > >>>>> extended a bit. I don't think we have one for BAD (we still
> > >>>>> receive data but the source has marked it bad).
> > >>>>>
> > >>>>> Timestamps I think are definitely a good idea, especially if we
> > >>>>> eventually DNP3.
> > >>>>>
> > >>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki
> > >>>>> <lu...@code-house.org>
> > >>> wrote:
> > >>>>>
> > >>>>>> You have response code for each individual field which allows
> > >>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same
> > >>>>>> information should be available also for subscriptions.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Łukasz
> > >>>>>>
> > >>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
> > >>>>>>> Ah, forgot: and is there a status available?
> > >>>>>>> Valid/invalid/good/bad
> > >>>>>> indicator of a plc value?
> > >>>>>>>
> > >>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
> > >>>>>>>> <an...@me.com.INVALID>
> > >>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> How can I get the source time of a PlcValue?
> > >>>>>>>>
> > >>>>>>>> a) from a subscription PlcSubscriptionEvent
> > >>>>>>>> b) from a read request  PlcReadResponse
> > >>>>>>>>
> > >>>>>>>> Best regards,
> > >>>>>>>> Andreas
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
>

AW: AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi,

I'm generally more concerned about users expecting us to deliver feature that driver X has for driver Y too.

I won't object, if you think it's worth doing. 

Perhaps if you could whip up an example in a feature branch? I think perhaps I was still not understanding what you propose.

Would that be ok?


Chris
 

-----Ursprüngliche Nachricht-----
Von: Lukas Ott <ot...@gmail.com> 
Gesendet: Montag, 8. März 2021 14:20
An: dev@plc4x.apache.org
Betreff: Re: AW: Source Time of PLC Value?

+1 to Lukasz

Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
luke@code-house.org>:

> Hold on for a second.
>
> Making options pushed over connection string is a long term recipe for 
> disaster. Apache Camel is a prime example of what could happen if you 
> starting with configuration with URIs and dynamic parameters. Most of 
> components distinguish producer (writer) and consumer (receiver) 
> options, most complicated components ended up with 40+ options 
> settable via URI. Having a mix of PlcValues which are timestamped and 'qualified'
> and not will complicate encoding logic (how we push above info to IEC 
> encoder/handler or decorate it?) and how we keep custom types 
> compatible with above?
>
> That is why I rather opt for extending API in a way which does not 
> force existing drivers to do anything. When any of community members, 
> regardless if its developer or user, will find it necessary to be 
> ported in another driver, she or he have right to do it. It is not 
> only yours duty to keep all drivers and languages the same.
>
> Speaking of which I am also not concerned about making all languages 
> having the same functionality as languages tend to offer different 
> options which might be hard to be ported. Lets stick to basics which 
> are same (connection strings, field definitions, read/write/subscribe
> syntax) and let various runtime lead their own ways toward implementation.
>
> Key point is simple - we won't be able to maintain all languages 
> equally. We should not hold moving ie. go or java forward because we 
> can't port new API functionality to C/Python/C#/whenever. If there is 
> someone who *does* use above in production and *wish* to have new API 
> parts then that person/organization duty is to do it.
>
> Best,
> Łukasz
>
>
> On 08.03.2021 13:17, Christofer Dutz wrote:
> > Hi,
> >
> > I must admit that I would be in favor of keeping it as simple as
> possible (max age in the connection string) and to implement more of 
> the missing parts in plc4x (like the subscription simulation layer) 
> and hereby getting the drivers we have a bit more aligned, than to 
> implement more and more special API features, that only one or two 
> drivers support. Cause this way we're running away form the 
> project-promise of running a given program at any type of PLC with any protocol.
> >
> > I mean ... at the current velocity the project is underway, I 
> > wouldn't
> really want to add a new MetaData layer that probably in the end only 
> 1-2 people will be implementing and which we must not only find out 
> how to fill with life in the first place, but also port to all of our 
> supported languages.
> >
> > If someone's willing to bring in some serious man/womenpower, I'm 
> > fine
> with that ... or we add it to some wish-list for future extensions. 
> But I wouldn't want to start something like this right now.
> >
> > Chris
> >
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Łukasz Dywicki <lu...@code-house.org>
> > Gesendet: Montag, 8. März 2021 13:07
> > An: dev@plc4x.apache.org
> > Betreff: Re: Source Time of PLC Value?
> >
> > I have not mentioned metadata term in my answer, so credits for 
> > mining
> it API go to you. :-)
> >
> > Now since you brought it I remember that JDBC has a "result set
> metadata". That might be thing we still miss. The result code is most 
> important and available instantly, other information is supplemental 
> and can be read additionally.
> >
> >
> > Best,
> > Łukasz
> >
> > On 08.03.2021 13:01, Ben Hutcheson wrote:
> >> +1 for Łukasz‘s metadata approach, it would avoid having to create
> >> duplicate PLcValue classes and give us a lot of flexibility in the
> future.
> >>
> >> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
> >> <ch...@c-ware.de>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> sorry for being late to the party ... KNX is currently consuming 
> >>> all my cycles.
> >>> Just wanted to add my thoughts to the discussion.
> >>>
> >>> Initially I thought about adding this sort of information to the 
> >>> API, but then I thought that we have so little protocols 
> >>> supporting this sort of concept that blowing up every PLCValue 
> >>> with this time information just would waste CPU time and Memory.
> >>>
> >>> When it comes to simulating subscriptions by polling in the 
> >>> background, I thought it would be a valid solution to provide a 
> >>> default age a value can have in the cache and to make it possible 
> >>> to override this in the connection string options. If a value is 
> >>> too old,
> a return code of:
> >>> STALE_DATA could be returned.
> >>>
> >>> Also, we do have an option of adding an additional set of PlcValue
> types.
> >>> If a driver supports this quality-of-service type of data, we 
> >>> could return special versions of PlcValues, that have these properties?
> >>>
> >>> This way we don't have to waste the CPU time and Memory for 
> >>> protocols that don't support this concept (We could even implement 
> >>> the methods on the default PlvValues to simply return some 
> >>> constants to the PlcValues all have the same API.
> >>>
> >>> Chris
> >>>
> >>>
> >>> -----Ursprüngliche Nachricht-----
> >>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>> Gesendet: Montag, 8. März 2021 10:43
> >>> An: dev@plc4x.apache.org
> >>> Betreff: Re: Source Time of PLC Value?
> >>>
> >>> Looking at issue we could utilize marker interfaces.
> >>> At least in Java we could define Qualified and Timestamped types.
> >>>
> >>> Yet looking at present state of API we are unlikely to make each 
> >>> and every PlcValue in each and every variant (qualified and 
> >>> timestamped, timestamped, qualified). It would force quite a lot of wrapping.
> >>> Safest option I see is to extend subscription/response API with 
> >>> additional method to retrieve all markers. For most of drivers it 
> >>> would return just an empty set. It leaves us also an open door for 
> >>> future drivers (or devices) to ship additional piece of metadata 
> >>> which
> does not fit into present API.
> >>>
> >>> Best,
> >>> Łukasz
> >>>
> >>>
> >>> On 08.03.2021 04:30, Otto Fowler wrote:
> >>>> So, maybe the extension is timestamp *and* timestamp source.
> >>>>
> >>>> uint64_t timestamp;
> >>>> typedef enum {
> >>>>       /* timestamp was generated by the source device */
> >>>>       SOURCE,
> >>>>       /* timestamp was generated by plc4x on receiving */
> >>>>       GENERATED,
> >>>> } TimestampQuality;
> >>>>
> >>>>
> >>>>
> >>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
> wrote:
> >>>>>
> >>>>> We do have some error codes, but it would just need to be 
> >>>>> extended a bit. I don't think we have one for BAD (we still 
> >>>>> receive data but the source has marked it bad).
> >>>>>
> >>>>> Timestamps I think are definitely a good idea, especially if we 
> >>>>> eventually DNP3.
> >>>>>
> >>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
> >>>>> <lu...@code-house.org>
> >>> wrote:
> >>>>>
> >>>>>> You have response code for each individual field which allows 
> >>>>>> you to determine state (OK, REMOTE_ERROR etc.). Same 
> >>>>>> information should be available also for subscriptions.
> >>>>>>
> >>>>>> Best,
> >>>>>> Łukasz
> >>>>>>
> >>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
> >>>>>>> Ah, forgot: and is there a status available?
> >>>>>>> Valid/invalid/good/bad
> >>>>>> indicator of a plc value?
> >>>>>>>
> >>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
> >>>>>>>> <an...@me.com.INVALID>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> How can I get the source time of a PlcValue?
> >>>>>>>>
> >>>>>>>> a) from a subscription PlcSubscriptionEvent
> >>>>>>>> b) from a read request  PlcReadResponse
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Andreas
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>

Re: AW: Source Time of PLC Value?

Posted by Lukas Ott <ot...@gmail.com>.

+1 to Lukasz

Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
luke@code-house.org>:

> Hold on for a second.
>
> Making options pushed over connection string is a long term recipe for
> disaster. Apache Camel is a prime example of what could happen if you
> starting with configuration with URIs and dynamic parameters. Most of
> components distinguish producer (writer) and consumer (receiver)
> options, most complicated components ended up with 40+ options settable
> via URI. Having a mix of PlcValues which are timestamped and 'qualified'
> and not will complicate encoding logic (how we push above info to IEC
> encoder/handler or decorate it?) and how we keep custom types compatible
> with above?
>
> That is why I rather opt for extending API in a way which does not force
> existing drivers to do anything. When any of community members,
> regardless if its developer or user, will find it necessary to be ported
> in another driver, she or he have right to do it. It is not only yours
> duty to keep all drivers and languages the same.
>
> Speaking of which I am also not concerned about making all languages
> having the same functionality as languages tend to offer different
> options which might be hard to be ported. Lets stick to basics which are
> same (connection strings, field definitions, read/write/subscribe
> syntax) and let various runtime lead their own ways toward implementation.
>
> Key point is simple - we won't be able to maintain all languages
> equally. We should not hold moving ie. go or java forward because we
> can't port new API functionality to C/Python/C#/whenever. If there is
> someone who *does* use above in production and *wish* to have new API
> parts then that person/organization duty is to do it.
>
> Best,
> Łukasz
>
>
> On 08.03.2021 13:17, Christofer Dutz wrote:
> > Hi,
> >
> > I must admit that I would be in favor of keeping it as simple as
> possible (max age in the connection string) and to implement more of the
> missing parts in plc4x (like the subscription simulation layer) and hereby
> getting the drivers we have a bit more aligned, than to implement more and
> more special API features, that only one or two drivers support. Cause this
> way we're running away form the project-promise of running a given program
> at any type of PLC with any protocol.
> >
> > I mean ... at the current velocity the project is underway, I wouldn't
> really want to add a new MetaData layer that probably in the end only 1-2
> people will be implementing and which we must not only find out how to fill
> with life in the first place, but also port to all of our supported
> languages.
> >
> > If someone's willing to bring in some serious man/womenpower, I'm fine
> with that ... or we add it to some wish-list for future extensions. But I
> wouldn't want to start something like this right now.
> >
> > Chris
> >
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Łukasz Dywicki <lu...@code-house.org>
> > Gesendet: Montag, 8. März 2021 13:07
> > An: dev@plc4x.apache.org
> > Betreff: Re: Source Time of PLC Value?
> >
> > I have not mentioned metadata term in my answer, so credits for mining
> it API go to you. :-)
> >
> > Now since you brought it I remember that JDBC has a "result set
> metadata". That might be thing we still miss. The result code is most
> important and available instantly, other information is supplemental and
> can be read additionally.
> >
> >
> > Best,
> > Łukasz
> >
> > On 08.03.2021 13:01, Ben Hutcheson wrote:
> >> +1 for Łukasz‘s metadata approach, it would avoid having to create
> >> duplicate PLcValue classes and give us a lot of flexibility in the
> future.
> >>
> >> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz
> >> <ch...@c-ware.de>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> sorry for being late to the party ... KNX is currently consuming all
> >>> my cycles.
> >>> Just wanted to add my thoughts to the discussion.
> >>>
> >>> Initially I thought about adding this sort of information to the API,
> >>> but then I thought that we have so little protocols supporting this
> >>> sort of concept that blowing up every PLCValue with this time
> >>> information just would waste CPU time and Memory.
> >>>
> >>> When it comes to simulating subscriptions by polling in the
> >>> background, I thought it would be a valid solution to provide a
> >>> default age a value can have in the cache and to make it possible to
> >>> override this in the connection string options. If a value is too old,
> a return code of:
> >>> STALE_DATA could be returned.
> >>>
> >>> Also, we do have an option of adding an additional set of PlcValue
> types.
> >>> If a driver supports this quality-of-service type of data, we could
> >>> return special versions of PlcValues, that have these properties?
> >>>
> >>> This way we don't have to waste the CPU time and Memory for protocols
> >>> that don't support this concept (We could even implement the methods
> >>> on the default PlvValues to simply return some constants to the
> >>> PlcValues all have the same API.
> >>>
> >>> Chris
> >>>
> >>>
> >>> -----Ursprüngliche Nachricht-----
> >>> Von: Łukasz Dywicki <lu...@code-house.org>
> >>> Gesendet: Montag, 8. März 2021 10:43
> >>> An: dev@plc4x.apache.org
> >>> Betreff: Re: Source Time of PLC Value?
> >>>
> >>> Looking at issue we could utilize marker interfaces.
> >>> At least in Java we could define Qualified and Timestamped types.
> >>>
> >>> Yet looking at present state of API we are unlikely to make each and
> >>> every PlcValue in each and every variant (qualified and timestamped,
> >>> timestamped, qualified). It would force quite a lot of wrapping.
> >>> Safest option I see is to extend subscription/response API with
> >>> additional method to retrieve all markers. For most of drivers it
> >>> would return just an empty set. It leaves us also an open door for
> >>> future drivers (or devices) to ship additional piece of metadata which
> does not fit into present API.
> >>>
> >>> Best,
> >>> Łukasz
> >>>
> >>>
> >>> On 08.03.2021 04:30, Otto Fowler wrote:
> >>>> So, maybe the extension is timestamp *and* timestamp source.
> >>>>
> >>>> uint64_t timestamp;
> >>>> typedef enum {
> >>>>       /* timestamp was generated by the source device */
> >>>>       SOURCE,
> >>>>       /* timestamp was generated by plc4x on receiving */
> >>>>       GENERATED,
> >>>> } TimestampQuality;
> >>>>
> >>>>
> >>>>
> >>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com>
> wrote:
> >>>>>
> >>>>> We do have some error codes, but it would just need to be extended
> >>>>> a bit. I don't think we have one for BAD (we still receive data but
> >>>>> the source has marked it bad).
> >>>>>
> >>>>> Timestamps I think are definitely a good idea, especially if we
> >>>>> eventually DNP3.
> >>>>>
> >>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org>
> >>> wrote:
> >>>>>
> >>>>>> You have response code for each individual field which allows you
> >>>>>> to determine state (OK, REMOTE_ERROR etc.). Same information
> >>>>>> should be available also for subscriptions.
> >>>>>>
> >>>>>> Best,
> >>>>>> Łukasz
> >>>>>>
> >>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
> >>>>>>> Ah, forgot: and is there a status available?
> >>>>>>> Valid/invalid/good/bad
> >>>>>> indicator of a plc value?
> >>>>>>>
> >>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
> >>>>>>>> <an...@me.com.INVALID>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> How can I get the source time of a PlcValue?
> >>>>>>>>
> >>>>>>>> a) from a subscription PlcSubscriptionEvent
> >>>>>>>> b) from a read request  PlcReadResponse
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Andreas
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>

Re: AW: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Hold on for a second.

Making options pushed over connection string is a long term recipe for
disaster. Apache Camel is a prime example of what could happen if you
starting with configuration with URIs and dynamic parameters. Most of
components distinguish producer (writer) and consumer (receiver)
options, most complicated components ended up with 40+ options settable
via URI. Having a mix of PlcValues which are timestamped and 'qualified'
and not will complicate encoding logic (how we push above info to IEC
encoder/handler or decorate it?) and how we keep custom types compatible
with above?

That is why I rather opt for extending API in a way which does not force
existing drivers to do anything. When any of community members,
regardless if its developer or user, will find it necessary to be ported
in another driver, she or he have right to do it. It is not only yours
duty to keep all drivers and languages the same.

Speaking of which I am also not concerned about making all languages
having the same functionality as languages tend to offer different
options which might be hard to be ported. Lets stick to basics which are
same (connection strings, field definitions, read/write/subscribe
syntax) and let various runtime lead their own ways toward implementation.

Key point is simple - we won't be able to maintain all languages
equally. We should not hold moving ie. go or java forward because we
can't port new API functionality to C/Python/C#/whenever. If there is
someone who *does* use above in production and *wish* to have new API
parts then that person/organization duty is to do it.

Best,
Łukasz


On 08.03.2021 13:17, Christofer Dutz wrote:
> Hi,
> 
> I must admit that I would be in favor of keeping it as simple as possible (max age in the connection string) and to implement more of the missing parts in plc4x (like the subscription simulation layer) and hereby getting the drivers we have a bit more aligned, than to implement more and more special API features, that only one or two drivers support. Cause this way we're running away form the project-promise of running a given program at any type of PLC with any protocol.
> 
> I mean ... at the current velocity the project is underway, I wouldn't really want to add a new MetaData layer that probably in the end only 1-2 people will be implementing and which we must not only find out how to fill with life in the first place, but also port to all of our supported languages.
> 
> If someone's willing to bring in some serious man/womenpower, I'm fine with that ... or we add it to some wish-list for future extensions. But I wouldn't want to start something like this right now. 
> 
> Chris
> 
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org> 
> Gesendet: Montag, 8. März 2021 13:07
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> 
> I have not mentioned metadata term in my answer, so credits for mining it API go to you. :-)
> 
> Now since you brought it I remember that JDBC has a "result set metadata". That might be thing we still miss. The result code is most important and available instantly, other information is supplemental and can be read additionally.
> 
> 
> Best,
> Łukasz
> 
> On 08.03.2021 13:01, Ben Hutcheson wrote:
>> +1 for Łukasz‘s metadata approach, it would avoid having to create
>> duplicate PLcValue classes and give us a lot of flexibility in the future.
>>
>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>> <ch...@c-ware.de>
>> wrote:
>>
>>> Hi all,
>>>
>>> sorry for being late to the party ... KNX is currently consuming all 
>>> my cycles.
>>> Just wanted to add my thoughts to the discussion.
>>>
>>> Initially I thought about adding this sort of information to the API, 
>>> but then I thought that we have so little protocols supporting this 
>>> sort of concept that blowing up every PLCValue with this time 
>>> information just would waste CPU time and Memory.
>>>
>>> When it comes to simulating subscriptions by polling in the 
>>> background, I thought it would be a valid solution to provide a 
>>> default age a value can have in the cache and to make it possible to 
>>> override this in the connection string options. If a value is too old, a return code of:
>>> STALE_DATA could be returned.
>>>
>>> Also, we do have an option of adding an additional set of PlcValue types.
>>> If a driver supports this quality-of-service type of data, we could 
>>> return special versions of PlcValues, that have these properties?
>>>
>>> This way we don't have to waste the CPU time and Memory for protocols 
>>> that don't support this concept (We could even implement the methods 
>>> on the default PlvValues to simply return some constants to the 
>>> PlcValues all have the same API.
>>>
>>> Chris
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Łukasz Dywicki <lu...@code-house.org>
>>> Gesendet: Montag, 8. März 2021 10:43
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: Source Time of PLC Value?
>>>
>>> Looking at issue we could utilize marker interfaces.
>>> At least in Java we could define Qualified and Timestamped types.
>>>
>>> Yet looking at present state of API we are unlikely to make each and 
>>> every PlcValue in each and every variant (qualified and timestamped, 
>>> timestamped, qualified). It would force quite a lot of wrapping.
>>> Safest option I see is to extend subscription/response API with 
>>> additional method to retrieve all markers. For most of drivers it 
>>> would return just an empty set. It leaves us also an open door for 
>>> future drivers (or devices) to ship additional piece of metadata which does not fit into present API.
>>>
>>> Best,
>>> Łukasz
>>>
>>>
>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>> So, maybe the extension is timestamp *and* timestamp source.
>>>>
>>>> uint64_t timestamp;
>>>> typedef enum {
>>>>       /* timestamp was generated by the source device */
>>>>       SOURCE,
>>>>       /* timestamp was generated by plc4x on receiving */
>>>>       GENERATED,
>>>> } TimestampQuality;
>>>>
>>>>
>>>>
>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
>>>>>
>>>>> We do have some error codes, but it would just need to be extended 
>>>>> a bit. I don't think we have one for BAD (we still receive data but 
>>>>> the source has marked it bad).
>>>>>
>>>>> Timestamps I think are definitely a good idea, especially if we 
>>>>> eventually DNP3.
>>>>>
>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org>
>>> wrote:
>>>>>
>>>>>> You have response code for each individual field which allows you 
>>>>>> to determine state (OK, REMOTE_ERROR etc.). Same information 
>>>>>> should be available also for subscriptions.
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>> Ah, forgot: and is there a status available? 
>>>>>>> Valid/invalid/good/bad
>>>>>> indicator of a plc value?
>>>>>>>
>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>>> <an...@me.com.INVALID>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>
>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Andreas
>>>>>>>
>>>>>>
>>>>
>>>
>>

AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi,

I must admit that I would be in favor of keeping it as simple as possible (max age in the connection string) and to implement more of the missing parts in plc4x (like the subscription simulation layer) and hereby getting the drivers we have a bit more aligned, than to implement more and more special API features, that only one or two drivers support. Cause this way we're running away form the project-promise of running a given program at any type of PLC with any protocol.

I mean ... at the current velocity the project is underway, I wouldn't really want to add a new MetaData layer that probably in the end only 1-2 people will be implementing and which we must not only find out how to fill with life in the first place, but also port to all of our supported languages.

If someone's willing to bring in some serious man/womenpower, I'm fine with that ... or we add it to some wish-list for future extensions. But I wouldn't want to start something like this right now. 

Chris





-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Montag, 8. März 2021 13:07
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?

I have not mentioned metadata term in my answer, so credits for mining it API go to you. :-)

Now since you brought it I remember that JDBC has a "result set metadata". That might be thing we still miss. The result code is most important and available instantly, other information is supplemental and can be read additionally.


Best,
Łukasz

On 08.03.2021 13:01, Ben Hutcheson wrote:
> +1 for Łukasz‘s metadata approach, it would avoid having to create
> duplicate PLcValue classes and give us a lot of flexibility in the future.
> 
> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
> <ch...@c-ware.de>
> wrote:
> 
>> Hi all,
>>
>> sorry for being late to the party ... KNX is currently consuming all 
>> my cycles.
>> Just wanted to add my thoughts to the discussion.
>>
>> Initially I thought about adding this sort of information to the API, 
>> but then I thought that we have so little protocols supporting this 
>> sort of concept that blowing up every PLCValue with this time 
>> information just would waste CPU time and Memory.
>>
>> When it comes to simulating subscriptions by polling in the 
>> background, I thought it would be a valid solution to provide a 
>> default age a value can have in the cache and to make it possible to 
>> override this in the connection string options. If a value is too old, a return code of:
>> STALE_DATA could be returned.
>>
>> Also, we do have an option of adding an additional set of PlcValue types.
>> If a driver supports this quality-of-service type of data, we could 
>> return special versions of PlcValues, that have these properties?
>>
>> This way we don't have to waste the CPU time and Memory for protocols 
>> that don't support this concept (We could even implement the methods 
>> on the default PlvValues to simply return some constants to the 
>> PlcValues all have the same API.
>>
>> Chris
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Montag, 8. März 2021 10:43
>> An: dev@plc4x.apache.org
>> Betreff: Re: Source Time of PLC Value?
>>
>> Looking at issue we could utilize marker interfaces.
>> At least in Java we could define Qualified and Timestamped types.
>>
>> Yet looking at present state of API we are unlikely to make each and 
>> every PlcValue in each and every variant (qualified and timestamped, 
>> timestamped, qualified). It would force quite a lot of wrapping.
>> Safest option I see is to extend subscription/response API with 
>> additional method to retrieve all markers. For most of drivers it 
>> would return just an empty set. It leaves us also an open door for 
>> future drivers (or devices) to ship additional piece of metadata which does not fit into present API.
>>
>> Best,
>> Łukasz
>>
>>
>> On 08.03.2021 04:30, Otto Fowler wrote:
>>> So, maybe the extension is timestamp *and* timestamp source.
>>>
>>> uint64_t timestamp;
>>> typedef enum {
>>>       /* timestamp was generated by the source device */
>>>       SOURCE,
>>>       /* timestamp was generated by plc4x on receiving */
>>>       GENERATED,
>>> } TimestampQuality;
>>>
>>>
>>>
>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
>>>>
>>>> We do have some error codes, but it would just need to be extended 
>>>> a bit. I don't think we have one for BAD (we still receive data but 
>>>> the source has marked it bad).
>>>>
>>>> Timestamps I think are definitely a good idea, especially if we 
>>>> eventually DNP3.
>>>>
>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org>
>> wrote:
>>>>
>>>>> You have response code for each individual field which allows you 
>>>>> to determine state (OK, REMOTE_ERROR etc.). Same information 
>>>>> should be available also for subscriptions.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>> Ah, forgot: and is there a status available? 
>>>>>> Valid/invalid/good/bad
>>>>> indicator of a plc value?
>>>>>>
>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>>>> <an...@me.com.INVALID>
>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>
>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Andreas
>>>>>>
>>>>>
>>>
>>
>

Re: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

I have not mentioned metadata term in my answer, so credits for mining
it API go to you. :-)

Now since you brought it I remember that JDBC has a "result set
metadata". That might be thing we still miss. The result code is most
important and available instantly, other information is supplemental and
can be read additionally.


Best,
Łukasz

On 08.03.2021 13:01, Ben Hutcheson wrote:
> +1 for Łukasz‘s metadata approach, it would avoid having to create
> duplicate PLcValue classes and give us a lot of flexibility in the future.
> 
> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz <ch...@c-ware.de>
> wrote:
> 
>> Hi all,
>>
>> sorry for being late to the party ... KNX is currently consuming all my
>> cycles.
>> Just wanted to add my thoughts to the discussion.
>>
>> Initially I thought about adding this sort of information to the API, but
>> then I thought that we have so little protocols supporting this sort of
>> concept that blowing up every PLCValue with this time information just
>> would waste CPU time and Memory.
>>
>> When it comes to simulating subscriptions by polling in the background, I
>> thought it would be a valid solution to provide a default age a value can
>> have in the cache and to make it possible to override this in the
>> connection string options. If a value is too old, a return code of:
>> STALE_DATA could be returned.
>>
>> Also, we do have an option of adding an additional set of PlcValue types.
>> If a driver supports this quality-of-service type of data, we could return
>> special versions of PlcValues, that have these properties?
>>
>> This way we don't have to waste the CPU time and Memory for protocols that
>> don't support this concept (We could even implement the methods on the
>> default PlvValues to simply return some constants to the PlcValues all have
>> the same API.
>>
>> Chris
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <lu...@code-house.org>
>> Gesendet: Montag, 8. März 2021 10:43
>> An: dev@plc4x.apache.org
>> Betreff: Re: Source Time of PLC Value?
>>
>> Looking at issue we could utilize marker interfaces.
>> At least in Java we could define Qualified and Timestamped types.
>>
>> Yet looking at present state of API we are unlikely to make each and every
>> PlcValue in each and every variant (qualified and timestamped, timestamped,
>> qualified). It would force quite a lot of wrapping.
>> Safest option I see is to extend subscription/response API with additional
>> method to retrieve all markers. For most of drivers it would return just an
>> empty set. It leaves us also an open door for future drivers (or devices)
>> to ship additional piece of metadata which does not fit into present API.
>>
>> Best,
>> Łukasz
>>
>>
>> On 08.03.2021 04:30, Otto Fowler wrote:
>>> So, maybe the extension is timestamp *and* timestamp source.
>>>
>>> uint64_t timestamp;
>>> typedef enum {
>>>       /* timestamp was generated by the source device */
>>>       SOURCE,
>>>       /* timestamp was generated by plc4x on receiving */
>>>       GENERATED,
>>> } TimestampQuality;
>>>
>>>
>>>
>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
>>>>
>>>> We do have some error codes, but it would just need to be extended a
>>>> bit. I don't think we have one for BAD (we still receive data but the
>>>> source has marked it bad).
>>>>
>>>> Timestamps I think are definitely a good idea, especially if we
>>>> eventually DNP3.
>>>>
>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org>
>> wrote:
>>>>
>>>>> You have response code for each individual field which allows you to
>>>>> determine state (OK, REMOTE_ERROR etc.). Same information should be
>>>>> available also for subscriptions.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>> Ah, forgot: and is there a status available? Valid/invalid/good/bad
>>>>> indicator of a plc value?
>>>>>>
>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler
>>>>>>> <an...@me.com.INVALID>
>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>
>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Andreas
>>>>>>
>>>>>
>>>
>>
>

Re: Source Time of PLC Value?

Posted by Ben Hutcheson <be...@gmail.com>.

+1 for Łukasz‘s metadata approach, it would avoid having to create
duplicate PLcValue classes and give us a lot of flexibility in the future.

On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi all,
>
> sorry for being late to the party ... KNX is currently consuming all my
> cycles.
> Just wanted to add my thoughts to the discussion.
>
> Initially I thought about adding this sort of information to the API, but
> then I thought that we have so little protocols supporting this sort of
> concept that blowing up every PLCValue with this time information just
> would waste CPU time and Memory.
>
> When it comes to simulating subscriptions by polling in the background, I
> thought it would be a valid solution to provide a default age a value can
> have in the cache and to make it possible to override this in the
> connection string options. If a value is too old, a return code of:
> STALE_DATA could be returned.
>
> Also, we do have an option of adding an additional set of PlcValue types.
> If a driver supports this quality-of-service type of data, we could return
> special versions of PlcValues, that have these properties?
>
> This way we don't have to waste the CPU time and Memory for protocols that
> don't support this concept (We could even implement the methods on the
> default PlvValues to simply return some constants to the PlcValues all have
> the same API.
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <lu...@code-house.org>
> Gesendet: Montag, 8. März 2021 10:43
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
>
> Looking at issue we could utilize marker interfaces.
> At least in Java we could define Qualified and Timestamped types.
>
> Yet looking at present state of API we are unlikely to make each and every
> PlcValue in each and every variant (qualified and timestamped, timestamped,
> qualified). It would force quite a lot of wrapping.
> Safest option I see is to extend subscription/response API with additional
> method to retrieve all markers. For most of drivers it would return just an
> empty set. It leaves us also an open door for future drivers (or devices)
> to ship additional piece of metadata which does not fit into present API.
>
> Best,
> Łukasz
>
>
> On 08.03.2021 04:30, Otto Fowler wrote:
> > So, maybe the extension is timestamp *and* timestamp source.
> >
> > uint64_t timestamp;
> > typedef enum {
> >       /* timestamp was generated by the source device */
> >       SOURCE,
> >       /* timestamp was generated by plc4x on receiving */
> >       GENERATED,
> > } TimestampQuality;
> >
> >
> >
> >> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
> >>
> >> We do have some error codes, but it would just need to be extended a
> >> bit. I don't think we have one for BAD (we still receive data but the
> >> source has marked it bad).
> >>
> >> Timestamps I think are definitely a good idea, especially if we
> >> eventually DNP3.
> >>
> >> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org>
> wrote:
> >>
> >>> You have response code for each individual field which allows you to
> >>> determine state (OK, REMOTE_ERROR etc.). Same information should be
> >>> available also for subscriptions.
> >>>
> >>> Best,
> >>> Łukasz
> >>>
> >>> On 05.03.2021 21:04, Andreas Vogler wrote:
> >>>> Ah, forgot: and is there a status available? Valid/invalid/good/bad
> >>> indicator of a plc value?
> >>>>
> >>>>> On 05.03.2021, at 21:02, Andreas Vogler
> >>>>> <an...@me.com.INVALID>
> >>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> How can I get the source time of a PlcValue?
> >>>>>
> >>>>> a) from a subscription PlcSubscriptionEvent
> >>>>> b) from a read request  PlcReadResponse
> >>>>>
> >>>>> Best regards,
> >>>>> Andreas
> >>>>
> >>>
> >
>

AW: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Hi all,

sorry for being late to the party ... KNX is currently consuming all my cycles.
Just wanted to add my thoughts to the discussion.

Initially I thought about adding this sort of information to the API, but then I thought that we have so little protocols supporting this sort of concept that blowing up every PLCValue with this time information just would waste CPU time and Memory.

When it comes to simulating subscriptions by polling in the background, I thought it would be a valid solution to provide a default age a value can have in the cache and to make it possible to override this in the connection string options. If a value is too old, a return code of: STALE_DATA could be returned.

Also, we do have an option of adding an additional set of PlcValue types. If a driver supports this quality-of-service type of data, we could return special versions of PlcValues, that have these properties? 

This way we don't have to waste the CPU time and Memory for protocols that don't support this concept (We could even implement the methods on the default PlvValues to simply return some constants to the PlcValues all have the same API.

Chris

-----Ursprüngliche Nachricht-----
Von: Łukasz Dywicki <lu...@code-house.org> 
Gesendet: Montag, 8. März 2021 10:43
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?

Looking at issue we could utilize marker interfaces.
At least in Java we could define Qualified and Timestamped types.

Yet looking at present state of API we are unlikely to make each and every PlcValue in each and every variant (qualified and timestamped, timestamped, qualified). It would force quite a lot of wrapping.
Safest option I see is to extend subscription/response API with additional method to retrieve all markers. For most of drivers it would return just an empty set. It leaves us also an open door for future drivers (or devices) to ship additional piece of metadata which does not fit into present API.

Best,
Łukasz

On 08.03.2021 04:30, Otto Fowler wrote:
> So, maybe the extension is timestamp *and* timestamp source.
> 
> uint64_t timestamp;
> typedef enum {
> 	/* timestamp was generated by the source device */
> 	SOURCE,
> 	/* timestamp was generated by plc4x on receiving */
> 	GENERATED,
> } TimestampQuality;
> 
> 
> 
>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
>>
>> We do have some error codes, but it would just need to be extended a 
>> bit. I don't think we have one for BAD (we still receive data but the 
>> source has marked it bad).
>>
>> Timestamps I think are definitely a good idea, especially if we 
>> eventually DNP3.
>>
>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org> wrote:
>>
>>> You have response code for each individual field which allows you to 
>>> determine state (OK, REMOTE_ERROR etc.). Same information should be 
>>> available also for subscriptions.
>>>
>>> Best,
>>> Łukasz
>>>
>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>> Ah, forgot: and is there a status available? Valid/invalid/good/bad
>>> indicator of a plc value?
>>>>
>>>>> On 05.03.2021, at 21:02, Andreas Vogler 
>>>>> <an...@me.com.INVALID>
>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> How can I get the source time of a PlcValue?
>>>>>
>>>>> a) from a subscription PlcSubscriptionEvent
>>>>> b) from a read request  PlcReadResponse
>>>>>
>>>>> Best regards,
>>>>> Andreas
>>>>
>>>
>

Re: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

Looking at issue we could utilize marker interfaces.
At least in Java we could define Qualified and Timestamped types.

Yet looking at present state of API we are unlikely to make each and
every PlcValue in each and every variant (qualified and timestamped,
timestamped, qualified). It would force quite a lot of wrapping.
Safest option I see is to extend subscription/response API with
additional method to retrieve all markers. For most of drivers it would
return just an empty set. It leaves us also an open door for future
drivers (or devices) to ship additional piece of metadata which does not
fit into present API.

Best,
Łukasz

On 08.03.2021 04:30, Otto Fowler wrote:
> So, maybe the extension is timestamp *and* timestamp source.
> 
> uint64_t timestamp;
> typedef enum {
> 	/* timestamp was generated by the source device */
> 	SOURCE,
> 	/* timestamp was generated by plc4x on receiving */
> 	GENERATED,
> } TimestampQuality;
> 
> 
> 
>> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
>>
>> We do have some error codes, but it would just need to be extended a bit. I
>> don't think we have one for BAD (we still receive data but the source has
>> marked it bad).
>>
>> Timestamps I think are definitely a good idea, especially if we eventually
>> DNP3.
>>
>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org> wrote:
>>
>>> You have response code for each individual field which allows you to
>>> determine state (OK, REMOTE_ERROR etc.). Same information should be
>>> available also for subscriptions.
>>>
>>> Best,
>>> Łukasz
>>>
>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>> Ah, forgot: and is there a status available? Valid/invalid/good/bad
>>> indicator of a plc value?
>>>>
>>>>> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID>
>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> How can I get the source time of a PlcValue?
>>>>>
>>>>> a) from a subscription PlcSubscriptionEvent
>>>>> b) from a read request  PlcReadResponse
>>>>>
>>>>> Best regards,
>>>>> Andreas
>>>>
>>>
>

Re: Source Time of PLC Value?

Posted by Otto Fowler <ot...@gmail.com>.

So, maybe the extension is timestamp *and* timestamp source.

uint64_t timestamp;
typedef enum {
	/* timestamp was generated by the source device */
	SOURCE,
	/* timestamp was generated by plc4x on receiving */
	GENERATED,
} TimestampQuality;



> On Mar 5, 2021, at 18:04, Ben Hutcheson <be...@gmail.com> wrote:
> 
> We do have some error codes, but it would just need to be extended a bit. I
> don't think we have one for BAD (we still receive data but the source has
> marked it bad).
> 
> Timestamps I think are definitely a good idea, especially if we eventually
> DNP3.
> 
> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org> wrote:
> 
>> You have response code for each individual field which allows you to
>> determine state (OK, REMOTE_ERROR etc.). Same information should be
>> available also for subscriptions.
>> 
>> Best,
>> Łukasz
>> 
>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>> Ah, forgot: and is there a status available? Valid/invalid/good/bad
>> indicator of a plc value?
>>> 
>>>> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID>
>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> How can I get the source time of a PlcValue?
>>>> 
>>>> a) from a subscription PlcSubscriptionEvent
>>>> b) from a read request  PlcReadResponse
>>>> 
>>>> Best regards,
>>>> Andreas
>>> 
>>

Re: Source Time of PLC Value?

Posted by Ben Hutcheson <be...@gmail.com>.

We do have some error codes, but it would just need to be extended a bit. I
don't think we have one for BAD (we still receive data but the source has
marked it bad).

Timestamps I think are definitely a good idea, especially if we eventually
DNP3.

On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki <lu...@code-house.org> wrote:

> You have response code for each individual field which allows you to
> determine state (OK, REMOTE_ERROR etc.). Same information should be
> available also for subscriptions.
>
> Best,
> Łukasz
>
> On 05.03.2021 21:04, Andreas Vogler wrote:
> > Ah, forgot: and is there a status available? Valid/invalid/good/bad
> indicator of a plc value?
> >
> >> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID>
> wrote:
> >>
> >> Hi,
> >>
> >> How can I get the source time of a PlcValue?
> >>
> >> a) from a subscription PlcSubscriptionEvent
> >> b) from a read request  PlcReadResponse
> >>
> >> Best regards,
> >> Andreas
> >
>

Re: Source Time of PLC Value?

Posted by Łukasz Dywicki <lu...@code-house.org>.

You have response code for each individual field which allows you to
determine state (OK, REMOTE_ERROR etc.). Same information should be
available also for subscriptions.

Best,
Łukasz

On 05.03.2021 21:04, Andreas Vogler wrote:
> Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?
> 
>> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
>>
>> Hi,
>>
>> How can I get the source time of a PlcValue? 
>>
>> a) from a subscription PlcSubscriptionEvent
>> b) from a read request  PlcReadResponse
>>
>> Best regards,
>> Andreas
>

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

Currently I use ModBus, but the goal is to support any kind of PLC4X driver …

My hope was that every PlcValue in PLC4X has a timestamp and a status … and if the protocol does not support it, then the timestamp/status should be “undefined” ...


> On 05.03.2021, at 21:54, Julian Feinauer <j....@pragmaticminds.de> wrote:
> 
> This is not a simple questing and depends on the protocol and device you are using. What are you using?
> 
> J
> 
> Von meinem Mobiltelefon gesendet
> 
> 
> -------- Ursprüngliche Nachricht --------
> Von: Andreas Vogler <an...@me.com.INVALID>
> Datum: Fr., 5. März 2021, 21:04
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?
> 
>> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
>> 
>> Hi,
>> 
>> How can I get the source time of a PlcValue?
>> 
>> a) from a subscription PlcSubscriptionEvent
>> b) from a read request  PlcReadResponse
>> 
>> Best regards,
>> Andreas
>

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

and OPC UA … but yes, maybe most protocols do not have this kind of information… 
Andreas

> On 05.03.2021, at 22:00, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Yeah,
> 
> So far the only protocol I worked on, that supports this "quality of data" feature, was the delta-v protocol. For all otbers we would have to provide the constant max value.
> 
> Unfortunately their was the only driver I wasn't able to finish. As I have no hardware available.
> 
> So this feature was discussed, but never implemented.
> 
> Chris
> 
> 
> Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
> ________________________________
> From: Julian Feinauer <j....@pragmaticminds.de>
> Sent: Friday, March 5, 2021 9:54:11 PM
> To: Andreas Vogler <an...@me.com.INVALID>; dev@plc4x.apache.org <de...@plc4x.apache.org>
> Subject: AW: Source Time of PLC Value?
> 
> This is not a simple questing and depends on the protocol and device you are using. What are you using?
> 
> J
> 
> Von meinem Mobiltelefon gesendet
> 
> 
> -------- Ursprüngliche Nachricht --------
> Von: Andreas Vogler <an...@me.com.INVALID>
> Datum: Fr., 5. März 2021, 21:04
> An: dev@plc4x.apache.org
> Betreff: Re: Source Time of PLC Value?
> Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?
> 
>> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
>> 
>> Hi,
>> 
>> How can I get the source time of a PlcValue?
>> 
>> a) from a subscription PlcSubscriptionEvent
>> b) from a read request  PlcReadResponse
>> 
>> Best regards,
>> Andreas
>

Re: Source Time of PLC Value?

Posted by Christofer Dutz <ch...@c-ware.de>.

Yeah,

So far the only protocol I worked on, that supports this "quality of data" feature, was the delta-v protocol. For all otbers we would have to provide the constant max value.

Unfortunately their was the only driver I wasn't able to finish. As I have no hardware available.

So this feature was discussed, but never implemented.

Chris

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
________________________________
From: Julian Feinauer <j....@pragmaticminds.de>
Sent: Friday, March 5, 2021 9:54:11 PM
To: Andreas Vogler <an...@me.com.INVALID>; dev@plc4x.apache.org <de...@plc4x.apache.org>
Subject: AW: Source Time of PLC Value?

This is not a simple questing and depends on the protocol and device you are using. What are you using?

J

Von meinem Mobiltelefon gesendet

-------- Ursprüngliche Nachricht --------
Von: Andreas Vogler <an...@me.com.INVALID>
Datum: Fr., 5. März 2021, 21:04
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?
Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?

> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
>
> Hi,
>
> How can I get the source time of a PlcValue?
>
> a) from a subscription PlcSubscriptionEvent
> b) from a read request  PlcReadResponse
>
> Best regards,
> Andreas

AW: Source Time of PLC Value?

Posted by Julian Feinauer <j....@pragmaticminds.de>.

This is not a simple questing and depends on the protocol and device you are using. What are you using?

J

Von meinem Mobiltelefon gesendet


-------- Ursprüngliche Nachricht --------
Von: Andreas Vogler <an...@me.com.INVALID>
Datum: Fr., 5. März 2021, 21:04
An: dev@plc4x.apache.org
Betreff: Re: Source Time of PLC Value?
Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?

> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
>
> Hi,
>
> How can I get the source time of a PlcValue?
>
> a) from a subscription PlcSubscriptionEvent
> b) from a read request  PlcReadResponse
>
> Best regards,
> Andreas

Re: Source Time of PLC Value?

Posted by Andreas Vogler <an...@me.com.INVALID>.

Ah, forgot: and is there a status available? Valid/invalid/good/bad indicator of a plc value?

> On 05.03.2021, at 21:02, Andreas Vogler <an...@me.com.INVALID> wrote:
> 
> Hi,
> 
> How can I get the source time of a PlcValue? 
> 
> a) from a subscription PlcSubscriptionEvent
> b) from a read request  PlcReadResponse
> 
> Best regards,
> Andreas