You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Iñigo Angulo <ia...@zylk.net.INVALID> on 2021/07/05 10:38:14 UTC
Re: Nifi integration record oriented processor for reading
Hi Otto,
thank you for the review of the PR and your comments.
We have answered them, and will review the sections you suggested. I would like to know if there are any things you want us to priorize. Maybe you have some deadlines for the next release or something, and we could try to get things done by this time. If you miss some task we could do here before doing the merge, please let us know.
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 24 de Junio 2021 14:40:12
Asunto: Re: Nifi integration record oriented processor for reading
Great,
I’m doing the review, so you should mark it ready :)
From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: June 24, 2021 at 04:52:19
To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: Nifi integration record oriented processor for reading
Hi Otto,
I have opened the new pull request, from a dedicated branch in our fork,
and a commit per contributor, the team members that have been working on
this.
Please take a look, and let me know if you want me to mark it as ready to
review.
iñigo
----------------------------------------- br/>Iñigo Anngulo br/> <
ZYLK.net :: consultoría.openSource br/>telf.: 7474123337 br/>Ribera de
Axpe, 11 br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
br/>----------------------------------------- <
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 17 de Junio 2021 16:38:13
Asunto: Re: Nifi integration record oriented processor for reading
That sounds great.
> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
> br/>> Hi Otto, <
> br/>> So we did this workaround, we re-forked the reepo and added the
changes of the nifi-record feature on a dedicated branch. I closed the
previous PR, will upload the new one on the following days (we are doing
some last test just in case we forgot to copy something...)
> br/>> I will keep you informed. <
> br/>> regards, <
> iñigo
> br/>> ----------------------------------------- br/>> Iñigo Angulo br/>>
br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
Ribera de Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950 Erandio
(Bizkaia) br/>> ----------------------------------------- <
> br/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 10 de Junio 2021 16:14:54
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> Yes, that sounds great. <
> Thank you
> br/>>> On Jun 10, 2021, at 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
wrote:
>> br/>>> Hi Otto, <
>> br/>>> I have tried to do the rebase of our ccommits, but I am having
difficulties at it... br/>>> br/>>> The issue is: we forked the repo on
september 2020, and started making tests and commits to our fork (on
'develop' branch). Now I am trying to do a rebase (using git rebase -i ID)
specifying the ID of our first commit. But when the file is open for
interactive mode, it gets ~900 commits on the develop branch (belonging to
members of the PLC4X community). I think this happens because before
opening the PullRequest, I did a 'merge upstream' with the actual PLC4X
repo, to get the updates of the code. So, I understand that in the
interactive mode file, I have to leave all commits to 'pick' (by default),
and change our commits of the nifi feature to 'squash' except from the
first one (which also remains as 'pick'). However, when I tried this, many
conflicts appear, almost one per each commit (comunity members' commits)...
>> I may be doing something wrong (never did a rebase before...) and I
prefered just to ask, as I dont want to break or cause any conflict to the
repo code.. If you see anything Im missing please let me know.
>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do the
fork of the PLC4X repo, and add the changes to the code on a dedicated
'feature-nifi-record' branch. Maybe this could make things clearer...
>> What do you think?
>> br/>>> thank you, <
>> iñigo
>> br/>>> ------------------------------------------ br/>>> Iñigo Angulo
br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: 747412337
br/>>> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203 br/>>> 48950
Erandio (Bizkaia) br/>>> ----------------------------------------- <
>> br/>>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>>> Awesome. <
>> br/>>> If you can, can I ask you to: <
>> br/>>> 1. Mark the PR as ready to review in github
>> 2. rebase or squash it to a single commit and force push to your branch
to clean it up
>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
<ii...@zylk.net.INVALID> wrote:
>>> br/>>>> Hi Otto, Chris <
>>> br/>>>> we have finally commited the uupdates on the Nifi Processor to
the Pull Request. The changes we have done are the following:
>>> - deducing avro datatypes from PlcResponse. Here we may check the
method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in order to
see if it is the best way to do it.
>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
org.apache.nifi:nifi-standard-nar, as it is used on runtime and was rising
errors during compilation.
>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
(comparing to BaseProcessor), as we have included the posibility to have an
input connection into the processor, and indicate the target addressMap
through flowfile attributes. The addressMap is now created in the onTrigger
method.
>>> - we have tested the performance with S7 and Modbus protocols (using a
Siemens S7-1200 and Schneider M221). We will upload an updated nifi
template for both protocols, but regarding this, do you have any testing
environment to simulate PLCs? If that is the case, we could prepare the
Processors configuration to match these ones (connection Strings and
addressMaps).
>>> br/>>>> Please take a look at the code,, any suggestion will be very
welcome.
>>> br/>>>> iñigo <
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Hi Otto, Chris, <
>>> br/>>>> we have been working on the prrocessor to include the logic of
getting the values for variables from the PLC response. We tested it with
the S7-1200 and seems to work fine, however we would like to make some
further tests before commiting it.
>>> br/>>>> Regarding the actual method whiich takes the datatype from the
response object, we did it in the following way:
>>> br/>>>> //PlcReadResponse readResponse
>>> Map<String, ? extends PlcValue> responseDataStructure =
readResponse.getAsPlcValue().getStruct();
>>> for (Map.Entry<String, ? extends PlcValue> entry :
responseDataStructure.entrySet()) {
>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().and().intType().endUnion().noDefault();
>>> }else if (entry.getValue() instanceof PlcREAL) {
>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();
>>> } ... and so on for the rest of the classes on the package
(org.apache.plc4x.java.spi.values.*)
>>> br/>>>> //the builder object is used tto build avro schema, with
desired datatypes (for example intType())
>>> }
>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
there is a better way to access PlcValues, please let us know and we will
update it.
>>> br/>>>> We will upload the code soon soo that you can take a deeper
look.
>>> br/>>>> thank you!!
>>> iñigo
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Sounds like a good plan!!
>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> we have been reviewingg the comments on the pull request, and
started to think about the approach of extracting values from response
directly. During next week, we will work on this and make some updates to
the code with the suggestions you made. We keep you informed
>>>> br/>>>>> thank you, <
>>>> br/>>>>> iñigo <
>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo 201-203
br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
----------------------------------------- <
>>>> br/>>>>> ----- Mensaje originall -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>> Asunto: AW: AW: Nifi integration record oriented processor for reading
>>>> br/>>>>> Hi Inigo, <
>>>> br/>>>>> especially if you havee a look at the KNX protocol. This
doesn't define the usual IEC datatypes we tried to use for all normal PLC
drivers.
>>>> Here we have hundreds of datatypes that don't match any other
protocol. I think the PLCValue approach would be the simplest.
>>>> The one thing you have to keep in mind, is that you should check a
PLCValue, if it's a list (Array type) or a Structure (Which sort of relates
to komplex types with a more sophisticated structure).
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
FFreitag, 23. April 2021 15:34
>>>> An: dev <de...@plc4x.apache.org>
>>>> Betreff: Re: AW: Nifi integration record oriented processor for
reading
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> Yes, I think the approoach you propose will be best. By now,
we are generating the schema ourselves. We have a record writer who is in
charge of reading PLC values. Schema is defined previously to reading the
values. We build this schema getting the protocol from the
'connectionString' (S7, Modbus) and the specified variable type from the
'PLC resource address String' containing the list of variable to read. From
this we deduce the expected Avro datatype when reading, for instance, a
word in S7 or a coil in Modbus. br/>&ggt;>>> br/>>>>> However, as you
mentioned, the oother approach will be much clearer and useful. Ideally,
getting the actual datatype from PLCValue when getting the response.
Regarding this, we tried to keep the previously described 'mapping'
separated from the rest of the code, so that hopefully it can be easily
replaced..
>>>> br/>>>>> We have done the pull rrequest, hope you can take a look at
the code and let us know what you think. We will fill the ICLA document
too.
>>>> br/>>>>> thank you <
>>>> iñigo
>>>> br/>>>>> br/>>>>> br/>>>>> ----------------------------------------- <
>>>> Iñigo Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> br/>>>>> ----- Mensaje original -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>> br/>>>>> Hi all, <
>>>> br/>>>>> Well, you get PlcValuees from the response that wrap the
different datatypes. So generally you shouldn't care about the detail type.
>>>> br/>>>>> However, you can call ggetObject() which returns the core
value the plc-value has ... so if it's the PLCValue for a Short, getObject
will return a short value.
>>>> br/>>>>> Does that help??
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Otto Fowler <ot...@gmail.com>
>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>> br/>>>>> So, you are generating the schema yourself, such that
downstream if they inherit schema they will just get what you generate? And
you are trying to do that by the connection string? If so, a different way
I could imagine doing would be to get the ’types’ of the data from the
responses themselves. This would be more generic. The flow I could imagine
( in OnTrigger ):
>>>> br/>>>>> DO READ <
>>>> IF NOT HAS SCHEMA
>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH SCHEMA
>>>> br/>>>>> Maybe Chris can speak tto how to get the types from the
responses.
>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
<ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to give
an overview. In Nifi, information packages are called Flowfiles, and these
are the actual units of information that are exchanged between Procesors,
all along the dataflow. Flowfiles have two sections where we can manage
data: Attributes and Content. In the "traditional" Nifi approach, you work
with both sections, extracting information from the Content to the
Attributes and viceversa to perform operations. This approach could have
one limitation when you are processing batch data (lines from a CSV file
for instance), where you need to split each of the lines into different
Flowfiles. Thus, a 1000 line CSV file leads to 1000 Flowfiles to process,
each of them containing a single record.
>>>>> br/>>>>>> On later versioons of the product, they introduced the
Record oriented approach. This approach allows you to manage multiple
records on a single Flowfile's Content, as long as these records have all
the same schema. This means that the operations defined by the Processors
are applied simultaneously to the whole content at once. Following with the
previous example, a 1000 line CSV file could produce a single Flowfile with
a content of 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro,
to serialize thee Flowfile's Content. Then, the Record Oriented Processors
use Writers and Readers to present this information in the desired format
(such as Avro, Json, CSV, etc). Basically, with the record oriented
approach, Nifi introduced multiple new Processors, and also included the
Record version of many of the "old" ones. Using this Record approach, Nifi
perfomance enhances notably, specially when working with large structured
information.
>>>>> br/>>>>>> The work we didd was creating a Record Oriented Processor,
based on the previously existing one Plc4xSourceProcessor, to read values
from the devices. We have also included a README on the
plc4x/plc4j/integrations/apache-nifi module explaining the Processor
configuration and giving an example. Moreover, we put a nifi template with
a dataflow for testing these processors, if useful.
>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor, that
is right. We added the writer capability to the existing
PLC4XSourceProcessor, so that it formats the output to the desired
configuration in a record manner. At the actual implementation, we did this
"protocol adaptation" from the sintax of the particular properties on
Processor's configuration. For example, from connection string
's7://IP:PORT', we extract the S7 idenifier and map variable datatypes to
the actual Avro datatypes for build the record output schema. However, here
we dont have vast experience with PLC4X libraries, and for sure there will
be better ways for doing this.
>>>>> Also about the Base Processor, we were thinking that maybe the best
approach could be to have this Base Processor, and then implement readers
for particular protocols as Controller Services. But here also, it could be
very helpful to have your opinion.
>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
defined some naming br/>>>>>> conventions, or expected structure to
faciliitate later work. At the br/>>>>>> present, we have a fork of the
project where we have been working on br/>>>>&ggt;> these Nifi changes. We
updated the content of our fork (fetch/merge
>>>>> upstream) about 2 weeks ago, and commited our changes to the
'develop' br/>>>>>> branch. Do we bettter create a new branch with our
commits? how do you br/>>>&>>> prefer to receive the code? (we are not very
experts on git, just in br/>>>>>> case we could cause some problemss...)
>>>>> br/>>>>>> thank you in addvance
>>>>> br/>>>>>> iñigo <
>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>> Iñigo Angulo
>>>>> br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>>> The more I thinnk of it,
>>>>> br/>>>>>> Perhaps we shouuld also think of potentially providing some
information on supported configuration options.
>>>>> Wouldn't it be cool if the driver could say: "I generally have these
options and they have these datatypes and mean this"
>>>>> Additionally, the transports could too say: "I generally have these
options and they have these datatypes and mean this"
>>>>> br/>>>>>> I would be our StreamPipes friends would love something
like that? Right?
>>>>> br/>>>>>> Chris <
>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>>> br/>>>>>> Hi Inigo, <
>>>>> br/>>>>>> I’m a coommitter on Apache Nifi as well as PLC4X, I would
be happy to review your processor.
>>>>> If I understand what you are saying correctly, you have a single
processor which supports record writing output?
>>>>> br/>>>>>> plc4x -> reccords
>>>>> br/>>>>>> And that you haave, for configuration purposes for that
processor created support on a per protocol basis for configuration and
validation?
>>>>> br/>>>>>> If there is perr protocol configuration / validation etc,
it may be better to have a base processor, and derived processors per
protocol to handle those differences.
>>>>> br/>>>>>> I look forward to seeing the code.
>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>>> Hi all,,
>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache Nifi
integration part of the project. We have created a Record oriented
processor for reading PLC data. It is based on the previous existing
SourceProcessor, but works with records, using a Nifi Writer (such as Avro,
Json, and so on) to write data on flowfiles content. br/>>>>>>> br/>>>>>>>
We updated the code on our fork with thee actual PLC4X git repo about 2
weeks ago, and tested it reading values with S7 from a S7-1200 CPU from
Nifi. Also, one of our customers has recently started to use it for
validation. br/>>>>>>> br/>>>>>>> Currently, it works with S7 and Modbus
oover TCP. This is because we had to write some classes to map
connectionString and variableList properties (sintax) of the processor to
the actual protocol, to be able to build then avro schema for output
flowfile, taking into account variable datatypes, etc. We only did this for
S7 and Modbus. I am sure that there is a better way to do this, so at this
point you maybe could take a look to find the best solution and avoid
needing to do this mapping. br/>>>>>>> br/>>>>>>> If you find this useful,
we could do a ppull request to the main PLC4x repo. Let us know what you
think. br/>>>>>>>> br/>>>>>>> best regards,
>>>>>> iñigo
>>>>>> br/>>>>>>> ------------------------------------------
>>>>>> Iñigo Angulo
>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
AW: AW: AW: AW: Nifi integration record oriented processor for
reading
Posted by Christofer Dutz <ch...@c-ware.de>.
And please keep in mind ... nothing has to be "perfect" ... if it's better than the current status I'm all for replacing it and improving it as we go.
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
Gesendet: Dienstag, 2. November 2021 16:20
An: dev <de...@plc4x.apache.org>
Betreff: Re: AW: AW: AW: Nifi integration record oriented processor for reading
Hi Otto
Thanks for the new review, we try to fix everything in two weeks. We think that the most importat thing is to confirm if the way that we are inferring the avro-schema and the way that we are casting the data from PLCValue are corret way.
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Otto Fowler" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Sábado, 30 de Octubre 2021 2:15:41
> Asunto: Re: AW: AW: AW: Nifi integration record oriented processor for
> reading
> I just reviewed the latest change set and resolved the conversations
> that could be resolved, there are a couple of showstoppers that are left however.
>
> The top two:
>
> - you have supported expression language for the plc address property,
> but you do not evaluate the expression language when accessing the
> property
>
> - the caching scheme you have implemented is incorrect I think
>
> - need to change to asyncCommit
>
> I have commented on the existing items as I reviewed.
>
> Still great work
>
> From: Gustavo Fernandez <gu...@zylk.net.invalid> <gu...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <de...@plc4x.apache.org>, Gustavo Fernandez <gu...@zylk.net>
> <gu...@zylk.net>
> Date: October 25, 2021 at 11:15:00
> To: Christofer Dutz <ch...@c-ware.de>
> <ch...@c-ware.de>
> Cc: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: AW: AW: AW: Nifi integration record oriented processor
> for reading
>
> Hi Chris,
>
> We are making a review of pending open topics on the PR, we think most
> important issues are resolved by now.. However, we still have some
> doubts regarding the actual mapping between PlcValues (from
> readResponse), AvroSchema (to build the record schema) and the Java
> types. br/>At the present there is a mapping that is working, but we
> would like you to take a look to give your opinion, as we think there
> could be a better way to make the casting there. The code to be
> reviewed is org.apache.plc4x.nifi.util.Plc4xCommon (
> https://github.com/zylklab/plc4x/blob/c535aa3698515375792fde3bc4248004
> 66b35cab/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main
> /java/org/apache/plc4x/nifi/util/Plc4xCommon.java)
>
>
> thank you in advance!
> --gustavo
>
> -----------------------------------------
> Gustavo Fernández
>
> ZYLK.net :: consultoría.openSource
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
>
> movil: +34 637546184
> ofic.: +34 944272119
> email: gus at zylk.net
>
> http://www.zylk.net/web/guest/web-2-0/blog
> -----------------------------------------
>
> ----- Mensaje original -----
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernández"
>> <gu...@zylk.net>
>> Enviados: Viernes, 22 de Octubre 2021 9:17:38
>> Asunto: AW: AW: AW: Nifi integration record oriented processor for
> reading
>
>> Ping ...
>> br/>> Just wanted to check-in on you foks here ... iss there anything
>> I
> can do to help
>> finish this?
>> br/>> Chris <
>> br/>> -----Ursprüngliche Nachricht----- <
>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
>> Gesendet: Mittwoch, 25. August 2021 12:10
>> An: dev <de...@plc4x.apache.org>
>> Betreff: Re: AW: AW: Nifi integration record oriented processor for
> reading
>> br/>> Hi, <
>> br/>> thanks Otto <
>> br/>> --gustavo <
>> br/>> br/>> ----- Mensaje original ----- <
>>> De: "Otto Fowler" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Martes, 24 de Agosto 2021 19:22:49
>>> Asunto: Re: AW: AW: Nifi integration record oriented processor for
>>> reading
>> br/>>> ok, new review up <
>>> br/>>> FFrom: Otto Fowler <ot...@gmail.com> <
> ottobackwards@gmail.com>
>>> Reply: Otto Fowler <ot...@gmail.com>
>>> <ot...@gmail.com>
>>> Date: August 24, 2021 at 12:30:23
>>> To: dev@plc4x.apache.org <de...@plc4x.apache.org>
>>> <de...@plc4x.apache.org>
>>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>>> reading br/>>> Sorry, I’ve been crushed by IRL wwork, I’ll review
>>> this as soon
> as
>>> possible
>>> br/>>> FFrom: Iñigo Angulo <ia...@zylk.net.invalid>
>>> <ia...@zylk.net.invalid>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
>>> <de...@plc4x.apache.org>
>>> Date: August 19, 2021 at 02:42:02
>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
>>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>>> reading br/>>> Hi Chris, < br/>>> I also had the confirmation from
>>> Apachhe that my ICLA is filed.
> My
>>> username is inigoao.
>>> br/>>> Our collegue Alvaro has also sent it, II think in few days he
> will
>>> receive the confirmation. His username is asellart.
>>> br/>>> regards, <
>>> iñigo
>>> br/>>> ------------------------------------------ r/>Iñigo Annngulo
>>> r/>
> <
>>> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera
>>> de Axpe,
>>> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
>>> r/>----------------------------------------- < br/>>> ----- Mensaje
>>> original ----- <
>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez"
>>> <gu...@zylk.net>
>>> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>> reading br/>>> Hi Gustavo, < br/>>> great ... I had something in the
>>> back of my mind, but I wasn't
> sure
>>> anymore ... then we're safe on that side ;-) br/>>> Chris < br/>>>
>>> -----Ursprüngliche Nachricht------
>>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet:
>>> Mittwoch,
> 18.
>>> August 2021 12:32
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>> reading br/>>> Hi Chris < br/>>> I sent my ICLA a month ago. The
>>> responnse from ASF was:
>>> br/>>> [[...]
>>> This message acknowledges receipt of your ICLA, which has been filed
>>> in the Apache Software Foundation records.
>>> With this message, the PLC4X PMC has been notified that your ICLA
>>> has been filed.
>>> [...]
>>> br/>>> My apache user id is zgus <
>>> br/>>> un saludo <
>>> --gustavo
>>> br/>>> ------------------------------------------
>>> Gustavo Fernández
>>> br/>>> ZYLK.net :: consultoría.openSourrce Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> br/>>> movil: +34 637546184 <
>>> ofic.: +34 944272119
>>> email: gus at zylk.net
>>> skype: gus.zylk
>>> br/>>> http://twitter.com/zylknet <
>>> http://www.zylk.net/web/guest/web-2-0/blog
>>> -----------------------------------------
>>> br/>>> ----- Mensaje original ----- <
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: dev@plc4x.apache.org
>>>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>> br/>>>> HI Inigo, <
>>>> r/>> great news ... I hope Otto can spare some cycleees to review
>>>> the PR
>>> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>>>> r/>> As it's been quite some time ... I am pretty suuure we have an
>>>> ICLA
>>> on r/>> file from your side, correct???
>>>> r/>> Chris <
>>>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>>>> Gesendet: Mittwoch, 18. August 2021 10:46
>>>> An: dev <de...@plc4x.apache.org>
>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the
>>>> Nifi PR and made a commit r/>>
>>> including revisions for all (or most) topppics.
>>>> r/>> the last changes we did were related to reforciiing the
>>>> methods for
>>> r/>> datatype conversion (for the record schheema construction), and
>>> r/>> extending the tests. About this lasst one, we changed the r/>>
>>> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This
>>> led r/>> to renaming the test claassess package to match
>>> BasePlc4xProcessor's one r/>>
>>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
>>> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a
>>> conflict
> on the
>>> PR, but we think it can be easily corrected when doing the merge.
>>>> r/>> At the moment, we think this nifi feature is prrretty much
>>> completed. Of r/>> course there are things that can bbee extended
>>> yet, which we will continue working on.
>>>> So please, if you could take a look and check if you miss something
>>>> r/>>
>>> would be great. Hopefully this could be soon tested by users in the
>>> r/>> community too and get some feedback :) <
>>>> r/>> thank you <
>>>> iñigo
>>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> r/>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, < r/>> thank you for the review of the PR and your
>>>> cooomments.
>>>> r/>> We have answered them, and will review the sectttions you
> suggested.
>>> I r/>> would like to know if there are any tthings you want us to
> priorize.
>>> r/>> Maybe you have some deadliinees for the next release or
>>> something, and r/>> we could try too geet things done by this time.
>>> If you miss some task we r/>> coould do here before doing the merge,
>>> please
> let us
>>> know.
>>>> r/>> iñigo <
>>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> r/>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> Great, < I’m doing the review, so you should mark it ready :)
>>>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
>>> <ia...@zylk.net.invalid>
>>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
>>> plc4x.apache.org>
>>>> Date: June 24, 2021 at 04:52:19
>>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>>> Subject: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, < r/>> I have opened the new pull request, from a
>>>> dedddicated branch in our
>>> r/>> fork, and a commit per contributor, thhhe team members that
>>> have been working on this.
>>>> r/>> Please take a look, and let me know if you wanttt me to mark
>>>> it as
>>> ready r/>> to review. <
>>>> r/>> iñigo <
>>>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo
>>>> br/> <
>>> ZYLK.net ::
>>>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
>>>> r/>>
>>> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>>>> br/>----------------------------------------- < r/>> ----- Mensaje
>>>> original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo
>>>> Annngulo <ia...@zylk.net.INVALID>
>>> wrote:
>>>>> br/>> Hi Otto, <
>>>>> br/>> So we did this workaround, we re-forked the reepo and added
>>>>> the
>>>> changes of the nifi-record feature on a dedicated branch. I closed
>>>> the
>>> r/>> previous PR, will upload the new one on the follooowing days
>>> (we are r/>> doing some last test just in case we forrggot to copy
>>> something...)
>>>>> br/>> I will keep you informed. <
>>>>> br/>> regards, <
>>>>> iñigo
>>>>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>>>>> r/>>>
>>> br/>> <
>>>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337
>>>> br/>>
>>> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>>
>>> 48950 r/>> Erandiooo
>>>> (Bizkaia) br/>> ----------------------------------------- <
>>>>> br/>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>>> r/>>>
>>> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021,
>>> at r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>>>> wrote:
>>>>>> br/>>> Hi Otto, <
>>>>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>>>>> r/>>>>
>>> having <
>>>> difficulties at it... br/>>> br/>>> The issue is: we forked the
>>>> repo r/>>
>>> on september 2020, and started mmmaking tests and commits to our
>>> fork (on 'develop'
>>>> branch). Now I am trying to do a rebase (using git rebase -i ID)
>>>> r/>>
>>> specifying the ID of our first commit. But when the fillle is open
>>> for r/>> interactive mode, it gets <
>>>> ~900 commits on the develop branch (belonging to members of the
>>>> PLC4X
>>> r/>> community). I think this happens because beforeee opening the
>>> r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X
>>> repo, to get the updates of the code.
>>>> So, I understand that in the interactive mode file, I have to leave
>>>> r/>>
>>> all commits to 'pick' (by default), and change our cccommits of the
>>> nifi feature to 'squash'
>>>> except from the first one (which also remains as 'pick'). However,
>>>> r/>>
>>> when I tried this, many conflicts appear, almost one per each commit
>>> (comunity members'
>>>> commits)...
>>>>>> I may be doing something wrong (never did a rebase before...) and
>>>>>> I
>>>> prefered just to ask, as I dont want to break or cause any conflict
>>>> to
>>> r/>> the repo code.. If you see anything Im missing plllease let me
> know.
>>>>>> br/>>> As a workaround, I was thinking we coulld close the PR,
>>>>>> re-do
>>> r/>>>> the <
>>>> fork of the PLC4X repo, and add the changes to the code on a
>>>> dedicated
>>> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
>>> clearer...
>>>>>> What do you think?
>>>>>> br/>>> thank you, <
>>>>>> iñigo
>>>>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>>>>> r/>>>>
>>> Angulo <
>>>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
>>> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo
>>> 201-203 rrr/>> br/>>> 48950 Erandio
>>>> (Bizkaia) br/>>> ----------------------------------------- <
>>>>>> br/>>> ----- Mensaje original ----- <
>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>>>> Asunto: Re: Nifi integration record oriented processor for
>>>>>> reading
>>> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: <
>>> br/>>>
> 1.
>>>>>> Mark the PR as ready to review in github 2. rebase or squash it
>>>>>> to a
>>> r/>>>> single commit and force push to youuur branch
>>>> to clean it up
>>>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>>>> <ii...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>> Hi Otto, Chris <
>>>>>>> br/>>>> we have finally commited the uupdates on the Nifi
>>>>>>> Processor
>>> r/>>>>> to <
>>>> the Pull Request. The changes we have done are the following:
>>>>>>> - deducing avro datatypes from PlcResponse. Here we may check
>>>>>>> the
>>>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
>>>> r/>>
>>> order to see if it is the best way to do it. <
>>>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>>>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
>>>> r/>>
>>> rising errors during compilation. <
>>>>>>> - we have changed onScheduled method in
>>>>>>> Plc4xSourceRecordProcessor
>>>> (comparing to BaseProcessor), as we have included the posibility to
>>>> r/>>
>>> have an input connection into the processor, and indddicate the
>>> target r/>> addressMap through flowfile attributes. TThhe addressMap
>>> is now created in the onTrigger method.
>>>>>>> - we have tested the performance with S7 and Modbus protocols
>>>>>>> r/>>>>>
>>> (using a <
>>>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
>>>> r/>>
>>> template for both protocols, but regarding this, dooo you have any
>>> r/>> testing environment to simulate PLCs??? If that is the case, we
>>> could r/>> prepare the Processors configurratiion to match these
>>> ones (connection Strings and addressMaps).
>>>>>>> br/>>>> Please take a look at the code,, any suggestion will be
>>> r/>>>>> very <
>>>> welcome.
>>>>>>> br/>>>> iñigo <
>>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> r/>>>>> Angulo
>>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>>> r/>>
>>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
>>> r/>>
>>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>>> ------------------------------------------ <
>>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>>>> Asunto: Re: Nifi integration record oriented processor for
>>>>>>> reading
>>> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on
>>> the r/>>t;;>>> prrocessor to include the logic of
>>>> getting the values for variables from the PLC response. We tested
>>>> it r/>>
>>> with the <
>>>> S7-1200 and seems to work fine, however we would like to make some
>>>> r/>>
>>> further tests before commiting it. <
>>>>>>> br/>>>> Regarding the actual method whiich takes the datatype
>>>>>>> from
>>> r/>>>>> the <
>>>> response object, we did it in the following way:
>>>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>>>> PlcValue> responseDataStructure =
>>>> readResponse.getAsPlcValue().getStruct();
>>>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>>>> responseDataStructure.entrySet()) {
>>>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>>
>>>>>>> r/>>>>>
>>> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().
>>> a
>>>>>>> n d().intType().endUnion().noDefault();
>>>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
>>> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().a
>>> n
>>>>>>> d ().doubleType().endUnion().noDefault();
>>>> r/>>>>> }}} ... and so on for the rest of the classes on the
>>>> package
>>>> (org.apache.plc4x.java.spi.values.*)
>>>>>>> br/>>>> //the builder object is used tto build avro schema, with
>>>> desired datatypes (for example intType())
>>>>>>> }
>>>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you
>>>>>>> think
>>>> there is a better way to access PlcValues, please let us know and
>>>> we r/>>
>>> will update it. <
>>>>>>> br/>>>> We will upload the code soon soo that you can take a
>>>>>>> deeper
>>>> look.
>>>>>>> br/>>>> thank you!!
>>>>>>> iñigo
>>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> r/>>>>> Angulo
>>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>>> r/>>
>>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
>>> r/>>
>>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>>> ------------------------------------------ <
>>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>>>> Asunto: Re: Nifi integration record oriented processor for
>>>>>>> reading
>>> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>>> br/>>>>> we have been reviewingg the comments on the pull
>>>>>>>> request,
>>> r/>>>>>> and <
>>>> started to think about the approach of extracting values from
>>>> response
>>> directly.
>>>> During next week, we will work on this and make some updates to the
>>>> r/>>
>>> code with the suggestions you made. We keep you infooormed
>>>>>>>> br/>>>>> thank you, <
>>>>>>>> br/>>>>> iñigo <
>>>>>>>> br/>>>>> ------------------------------------------ br/>>>>>
>>>>>>>> Iñigo
>>>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource
>>>> br/>>>>>
>>> telf.:
>>>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
>>>> r/>>
>>> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>>>> ----------------------------------------- <
>>>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if
>>> you havee a r/>>>>>> look at the KNX protocol. This <
>>>> doesn't define the usual IEC datatypes we tried to use for all
>>>> normal
>>> r/>> PLC drivers. <
>>>>>>>> Here we have hundreds of datatypes that don't match any other
>>>> protocol. I think the PLCValue approach would be the simplest.
>>>>>>>> The one thing you have to keep in mind, is that you should
>>>>>>>> check a
>>>> PLCValue, if it's a list (Array type) or a Structure (Which sort of
>>>> r/>>
>>> relates to komplex types with a more sophisticated ssstructure).
>>>>>>>> br/>>>>> Chris <
>>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>>>> FFreitag, 23. April 2021 15:34
>>>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>>> reading
>>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>>> br/>>>>> Yes, I think the approoach you propose will be best.
>>>>>>>> By
>>> r/>>>>>> now, <
>>>> we are generating the schema ourselves. We have a record writer who
>>>> is
>>> r/>> in charge of reading PLC values. Schema is definnned previously
>>> to r/>> reading the values. We build this schema gggetting the
>>> protocol from the r/>> 'connectionString' (S7, Modbuuss) and the
>>> specified variable type from the 'PLC resource address String'
>>>> containing the list of variable to read. From this we deduce the
>>>> r/>>
>>> expected Avro datatype when reading, for instance, a word in S7 or a
>>> coil in Modbus.
>>>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother
>>>> approach r/>>
>>> will be much clearerrr and useful. Ideally, getting the actual
>>> datatype r/>> from PLCCVValue when getting the response.
>>>> Regarding this, we tried to keep the previously described 'mapping'
>>>> separated from the rest of the code, so that hopefully it can be
>>>> r/>>
>>> easily replaced.. <
>>>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a
>>>>>>>> look
>>> r/>>>>>> at <
>>>> the code and let us know what you think. We will fill the ICLA
>>>> document
>>> too.
>>>>>>>> br/>>>>> thank you <
>>>>>>>> iñigo
>>>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>>>> ----------------------------------------- < Iñigo Angulo
>>>>>>>> br/>>>>>
>>> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>> -----------------------------------------
>>>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>> reading
>>> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees
>>> from the r/>>>>>> response that wrap the <
>>>> different datatypes. So generally you shouldn't care about the
>>>> detail
>>> type.
>>>>>>>> br/>>>>> However, you can call ggetObject() which returns the
>>>>>>>> core
>>>> value the plc-value has ... so if it's the PLCValue for a Short,
>>>> r/>>
>>> getObject will return a short value. <
>>>>>>>> br/>>>>> Does that help??
>>>>>>>> br/>>>>> Chris <
>>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>>>> r/>>>>>>
>>> reading br/>>>>&&> So, you are generating the schema yourself, such
>>> r/>>>&ggtt;>> that
>>>> downstream if they inherit schema they will just get what you r/>>
>>> generate??? And you are trying to do that by the connection string?
>>> If r/>> so, a different way I could imagine doingg woould be to get
>>> the
> ’types’
>>> r/>> of the data from the responses themselves. This would be more
> generic.
>>> r/>> The flow I could imagine ( in OnTrigger
>>>> ):
>>>>>>>> br/>>>>> DO READ <
>>>>>>>> IF NOT HAS SCHEMA
>>>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>>>>> r/>>>>>>
>>> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
>>> r/>>>&ggtt;>> from the
>>>> responses.
>>>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try
>>>>>>>>> to
>>> r/>>>>>>&gggt; give
>>>> an overview. In Nifi, information packages are called Flowfiles,
>>>> and r/>>
>>> these are the actual units of information that arrre exchanged
>>> between r/>> Procesors, all along the dataflow. FFFlowfiles have two
>>> sections where r/>> we can manage <
>>>> data: Attributes and Content. In the "traditional" Nifi approach,
>>>> you
>>> r/>> work with both sections, extracting informatiiion from the
>>> Content to r/>> the Attributes and viceversa to perffform operations.
>>> This approach r/>> could have one limitation whheen you are
>>> processing batch data (lines r/>> from a CSV file foor instance),
>>> where you need to split each of the r/>> lines intto ddifferent
>>> Flowfiles. Thus, a
>>> 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process,
>>> each of them containing a single record.
>>>>>>>>> br/>>>>>> On later versioons of the product, they introduced
>>>>>>>>> the
>>>> Record oriented approach. This approach allows you to manage
>>>> multiple
>>> r/>> records on a single FFFlowfile's Content, as long as these
>>> records have all the same schema.
>>>> This means that the operations defined by the Processors are
>>>> applied r/>>
>>> simultaneously to the whole content at once. FFFollowing with the
>>> r/>> previous example, a 1000 line CSV file couuldd produce a single
>>> Flowfile r/>> with a content of <
>>>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to
>>>> r/>>
>>> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
>>> Processors uuse Writers and Readers to present this information in
>>> the r/>> desiired format (such as Avro, Json, CSV, etc). Basically,
>>> with the br/>&> record oriented approach, Nifi introduced multiple
>>> new Processors, and r/>> also included the Record version of many of
>>> the """old" ones. Using this r/>> Record approach, Nifi perfomance
>>> enhances notably, specially when working with large structured
> information.
>>>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>>>>> r/>>>>>>>
>>> Procccessor,
>>>> based on the previously existing one Plc4xSourceProcessor, to read
>>>> r/>>
>>> values from the devices. We have also included a READDDME on the
>>> r/>> plc4x/plc4j/integrations/apache-nifi module expllaaining the
>>> Processor r/>> configuration and giving an example. Mooreover, we
>>> put a nifi template r/>> with a dataflow for testiing these processors, if useful.
>>>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>> r/>>>>>>>;; that
>>>> is right. We added the writer capability to the existing
>>>> PLC4XSourceProcessor, so that it formats the output to the desired
>>> configuration in a record manner.
>>>> At the actual implementation, we did this "protocol adaptation"
>>>> from r/>>
>>> the sintax of the particular properties on Procccessor's configuration.
>>> r/>> FFFor example, from connection string 's7://IP:PORT', we
>>> extract the
>>> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro
>>> datatypes for build the record output schema.
>>>> However, here we dont have vast experience with PLC4X libraries,
>>>> and r/>>
>>> for sure there will be better ways for doing this.
>>>>>>>>> Also about the Base Processor, we were thinking that maybe the
>>> r/>>>>>>> best <
>>>> approach could be to have this Base Processor, and then implement
>>>> r/>>
>>> readers for particular protocols as Controller Serviccces. But here
>>> r/>> also, it could be very helpful to have your opppinion.
>>>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>>>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you
>>>> have
>>> r/>> defined some naming br/>&gggt;>>>> conventions, or expected
>>> structure to r/>> ffaaciliitate later work. At the br/>>>>>>
>>> present, we have a fork of r/>> the project where we have been
>>> working on br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed
>>> tthe content of our fork (fetch/merge
>>>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>>>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with
>>>> our
>>> commits?
>>>> how do you br/>>>&>>> prefer to receive the code? (we are not very
>>>> r/>>
>>> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
>>> problemss.....)
>>>>>>>>> br/>>>>>> thank you in addvance br/>>>>>> iñigo < br/>>>>>>
>>>>>>>>> br/>>>>>> ----------------------------------------- <
>>> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net ::
>>> connsultoría.openSource
>>>>>>>>> telf.: 747412337
>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>>> -----------------------------------------
>>>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>>> r/>>>>>>>
>>> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
>>> r/>>>>>>> shouuld also think of potentialllyy providing some
>>>> information on supported configuration options.
>>>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>> r/>>>>>>> these <
>>>> options and they have these datatypes and mean this"
>>>>>>>>> Additionally, the transports could too say: "I generally have
>>> r/>>>>>>> these <
>>>> options and they have these datatypes and mean this"
>>>>>>>>> br/>>>>>> I would be our StreamPipes friends would love
>>>>>>>>> something
>>>> like that? Right?
>>>>>>>>> br/>>>>>> Chris <
>>>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter
>>> on r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>>>> be happy to review your processor.
>>>>>>>>> If I understand what you are saying correctly, you have a
>>>>>>>>> single
>>>> processor which supports record writing output?
>>>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>>>> br/>>>>>> And that you haave, for configuration purposes for
>>>>>>>>> that
>>>> processor created support on a per protocol basis for configuration
>>>> r/>>
>>> and validation???
>>>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>> r/>>>>>>>;; etc,
>>>> it may be better to have a base processor, and derived processors
>>>> per
>>> r/>> protocol to handle those differences. <
>>>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the
>>>>>>>>>> Apache
>>> r/>>>>>&&>>> Nifi
>>>> integration part of the project. We have created a Record oriented
>>>> r/>>
>>> processor for reading PLC data. It is based on the prrrevious
>>> existing r/>> SourceProcessor, but works with records, uussing a
>>> Nifi Writer (such as r/>> Avro, Json, and so on) to writte data on
>>> flowfiles content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the
>>> code on our fork with thee actual PLC4X git r/>> repo about 2 weeks
>>> ago, and tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi.
>>> Alsoo, onee of our customers has recently started r/>> to use it for
>>> validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7
>>> and Modbus oover TCP. This is becaause we had to write some r/>>
>>> classes to map connectionSString annd variableList properties
>>> (sintax) r/>> of the processoor to the actual protocol, to be able
>>> to build then avro r/>> scchema for ooutput flowfile, taking into
>>> account variable datatypes, br/>> etcc. We only did this for
>>>> S7 and Modbus. I am sure that there is a better way to do this, so
>>>> at
>>> r/>> this point you maybe could take a look to find theee best
>>> solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>>
>>> br/>>>>>>> If you find br/>> this useful, we could do a ppull
>>> request to
> the
>>> main PLC4x repo.
>>> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
>>> regards,
>>>>>>>>>> iñigo
>>>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>>>> Iñigo Angulo
>>>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>>>> telf.: 747412337
>>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>>> 48950 Erandio (Bizkaia)
> > > > >>>>>> -----------------------------------------
Re: AW: AW: AW: Nifi integration record oriented processor for
reading
Posted by Gustavo Fernandez <gu...@zylk.net.INVALID>.
Hi Otto
Thanks for the new review, we try to fix everything in two weeks. We think that the most importat thing is to confirm if the way that we are inferring the avro-schema and the way that we are casting the data from PLCValue are corret way.
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Otto Fowler" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Sábado, 30 de Octubre 2021 2:15:41
> Asunto: Re: AW: AW: AW: Nifi integration record oriented processor for reading
> I just reviewed the latest change set and resolved the conversations that
> could be resolved, there are a couple of showstoppers that are left however.
>
> The top two:
>
> - you have supported expression language for the plc address property, but
> you do not evaluate the expression language when accessing the property
>
> - the caching scheme you have implemented is incorrect I think
>
> - need to change to asyncCommit
>
> I have commented on the existing items as I reviewed.
>
> Still great work
>
> From: Gustavo Fernandez <gu...@zylk.net.invalid> <gu...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <de...@plc4x.apache.org>, Gustavo
> Fernandez <gu...@zylk.net> <gu...@zylk.net>
> Date: October 25, 2021 at 11:15:00
> To: Christofer Dutz <ch...@c-ware.de> <ch...@c-ware.de>
> Cc: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: AW: AW: AW: Nifi integration record oriented processor for
> reading
>
> Hi Chris,
>
> We are making a review of pending open topics on the PR, we think most
> important issues are resolved by now.. However, we still have some doubts
> regarding the actual mapping between PlcValues (from readResponse),
> AvroSchema (to build the record schema) and the Java types. br/>At the
> present there is a mapping that is working, but we would like you to take a
> look to give your opinion, as we think there could be a better way to make
> the casting there. The code to be reviewed is
> org.apache.plc4x.nifi.util.Plc4xCommon (
> https://github.com/zylklab/plc4x/blob/c535aa3698515375792fde3bc424800466b35cab/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/util/Plc4xCommon.java)
>
>
> thank you in advance!
> --gustavo
>
> -----------------------------------------
> Gustavo Fernández
>
> ZYLK.net :: consultoría.openSource
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
>
> movil: +34 637546184
> ofic.: +34 944272119
> email: gus at zylk.net
>
> http://www.zylk.net/web/guest/web-2-0/blog
> -----------------------------------------
>
> ----- Mensaje original -----
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernández" <gu...@zylk.net>
>> Enviados: Viernes, 22 de Octubre 2021 9:17:38
>> Asunto: AW: AW: AW: Nifi integration record oriented processor for
> reading
>
>> Ping ...
>> br/>> Just wanted to check-in on you foks here ... iss there anything I
> can do to help
>> finish this?
>> br/>> Chris <
>> br/>> -----Ursprüngliche Nachricht----- <
>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
>> Gesendet: Mittwoch, 25. August 2021 12:10
>> An: dev <de...@plc4x.apache.org>
>> Betreff: Re: AW: AW: Nifi integration record oriented processor for
> reading
>> br/>> Hi, <
>> br/>> thanks Otto <
>> br/>> --gustavo <
>> br/>> br/>> ----- Mensaje original ----- <
>>> De: "Otto Fowler" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Martes, 24 de Agosto 2021 19:22:49
>>> Asunto: Re: AW: AW: Nifi integration record oriented processor for
>>> reading
>> br/>>> ok, new review up <
>>> br/>>> FFrom: Otto Fowler <ot...@gmail.com> <
> ottobackwards@gmail.com>
>>> Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
>>> Date: August 24, 2021 at 12:30:23
>>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>>> reading
>>> br/>>> Sorry, I’ve been crushed by IRL wwork, I’ll review this as soon
> as
>>> possible
>>> br/>>> FFrom: Iñigo Angulo <ia...@zylk.net.invalid>
>>> <ia...@zylk.net.invalid>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
>>> <de...@plc4x.apache.org>
>>> Date: August 19, 2021 at 02:42:02
>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
>>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>>> reading
>>> br/>>> Hi Chris, <
>>> br/>>> I also had the confirmation from Apachhe that my ICLA is filed.
> My
>>> username is inigoao.
>>> br/>>> Our collegue Alvaro has also sent it, II think in few days he
> will
>>> receive the confirmation. His username is asellart.
>>> br/>>> regards, <
>>> iñigo
>>> br/>>> ------------------------------------------ r/>Iñigo Annngulo r/>
> <
>>> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de
>>> Axpe,
>>> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
>>> r/>----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
>>> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
>>> Asunto: AW: AW: Nifi integration record oriented processor for reading
>>> br/>>> Hi Gustavo, <
>>> br/>>> great ... I had something in the back of my mind, but I wasn't
> sure
>>> anymore ... then we're safe on that side ;-)
>>> br/>>> Chris <
>>> br/>>> -----Ursprüngliche Nachricht------
>>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch,
> 18.
>>> August 2021 12:32
>>> An: dev@plc4x.apache.org
>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>> reading
>>> br/>>> Hi Chris <
>>> br/>>> I sent my ICLA a month ago. The responnse from ASF was:
>>> br/>>> [[...]
>>> This message acknowledges receipt of your ICLA, which has been filed
>>> in the Apache Software Foundation records.
>>> With this message, the PLC4X PMC has been notified that your ICLA has
>>> been filed.
>>> [...]
>>> br/>>> My apache user id is zgus <
>>> br/>>> un saludo <
>>> --gustavo
>>> br/>>> ------------------------------------------
>>> Gustavo Fernández
>>> br/>>> ZYLK.net :: consultoría.openSourrce
>>> Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> br/>>> movil: +34 637546184 <
>>> ofic.: +34 944272119
>>> email: gus at zylk.net
>>> skype: gus.zylk
>>> br/>>> http://twitter.com/zylknet <
>>> http://www.zylk.net/web/guest/web-2-0/blog
>>> -----------------------------------------
>>> br/>>> ----- Mensaje original ----- <
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: dev@plc4x.apache.org
>>>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>> br/>>>> HI Inigo, <
>>>> r/>> great news ... I hope Otto can spare some cycleees to review the
>>>> PR
>>> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>>>> r/>> As it's been quite some time ... I am pretty suuure we have an
>>>> ICLA
>>> on r/>> file from your side, correct???
>>>> r/>> Chris <
>>>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>>>> Gesendet: Mittwoch, 18. August 2021 10:46
>>>> An: dev <de...@plc4x.apache.org>
>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the
>>>> Nifi PR and made a commit r/>>
>>> including revisions for all (or most) topppics.
>>>> r/>> the last changes we did were related to reforciiing the methods
>>>> for
>>> r/>> datatype conversion (for the record schheema construction), and
>>> r/>> extending the tests. About this lasst one, we changed the r/>>
>>> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This
>>> led r/>> to renaming the test claassess package to match
>>> BasePlc4xProcessor's one r/>>
>>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
>>> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict
> on the
>>> PR, but we think it can be easily corrected when doing the merge.
>>>> r/>> At the moment, we think this nifi feature is prrretty much
>>> completed. Of r/>> course there are things that can bbee extended yet,
>>> which we will continue working on.
>>>> So please, if you could take a look and check if you miss something
>>>> r/>>
>>> would be great. Hopefully this could be soon tested by users in the
>>> r/>> community too and get some feedback :) <
>>>> r/>> thank you <
>>>> iñigo
>>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> r/>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, < r/>> thank you for the review of the PR and your
>>>> cooomments.
>>>> r/>> We have answered them, and will review the sectttions you
> suggested.
>>> I r/>> would like to know if there are any tthings you want us to
> priorize.
>>> r/>> Maybe you have some deadliinees for the next release or
>>> something, and r/>> we could try too geet things done by this time. If
>>> you miss some task we r/>> coould do here before doing the merge, please
> let us
>>> know.
>>>> r/>> iñigo <
>>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> r/>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> Great, < I’m doing the review, so you should mark it ready :)
>>>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
>>> <ia...@zylk.net.invalid>
>>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
>>> plc4x.apache.org>
>>>> Date: June 24, 2021 at 04:52:19
>>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>>> Subject: Re: Nifi integration record oriented processor for reading
>>>> r/>> Hi Otto, < r/>> I have opened the new pull request, from a
>>>> dedddicated branch in our
>>> r/>> fork, and a commit per contributor, thhhe team members that have
>>> been working on this.
>>>> r/>> Please take a look, and let me know if you wanttt me to mark it
>>>> as
>>> ready r/>> to review. <
>>>> r/>> iñigo <
>>>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo
>>>> br/> <
>>> ZYLK.net ::
>>>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
>>>> r/>>
>>> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>>>> br/>----------------------------------------- < r/>> ----- Mensaje
>>>> original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo
>>>> Annngulo <ia...@zylk.net.INVALID>
>>> wrote:
>>>>> br/>> Hi Otto, <
>>>>> br/>> So we did this workaround, we re-forked the reepo and added
>>>>> the
>>>> changes of the nifi-record feature on a dedicated branch. I closed
>>>> the
>>> r/>> previous PR, will upload the new one on the follooowing days (we
>>> are r/>> doing some last test just in case we forrggot to copy
>>> something...)
>>>>> br/>> I will keep you informed. <
>>>>> br/>> regards, <
>>>>> iñigo
>>>>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>>>>> r/>>>
>>> br/>> <
>>>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
>>> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>>
>>> 48950 r/>> Erandiooo
>>>> (Bizkaia) br/>> ----------------------------------------- <
>>>>> br/>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>>> r/>>>
>>> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
>>> r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>>>> wrote:
>>>>>> br/>>> Hi Otto, <
>>>>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>>>>> r/>>>>
>>> having <
>>>> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
>>>> r/>>
>>> on september 2020, and started mmmaking tests and commits to our fork
>>> (on 'develop'
>>>> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
>>> specifying the ID of our first commit. But when the fillle is open for
>>> r/>> interactive mode, it gets <
>>>> ~900 commits on the develop branch (belonging to members of the PLC4X
>>> r/>> community). I think this happens because beforeee opening the
>>> r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X
>>> repo, to get the updates of the code.
>>>> So, I understand that in the interactive mode file, I have to leave
>>>> r/>>
>>> all commits to 'pick' (by default), and change our cccommits of the
>>> nifi feature to 'squash'
>>>> except from the first one (which also remains as 'pick'). However,
>>>> r/>>
>>> when I tried this, many conflicts appear, almost one per each commit
>>> (comunity members'
>>>> commits)...
>>>>>> I may be doing something wrong (never did a rebase before...) and I
>>>> prefered just to ask, as I dont want to break or cause any conflict
>>>> to
>>> r/>> the repo code.. If you see anything Im missing plllease let me
> know.
>>>>>> br/>>> As a workaround, I was thinking we coulld close the PR,
>>>>>> re-do
>>> r/>>>> the <
>>>> fork of the PLC4X repo, and add the changes to the code on a
>>>> dedicated
>>> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
>>> clearer...
>>>>>> What do you think?
>>>>>> br/>>> thank you, <
>>>>>> iñigo
>>>>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>>>>> r/>>>>
>>> Angulo <
>>>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
>>> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo
>>> 201-203 rrr/>> br/>>> 48950 Erandio
>>>> (Bizkaia) br/>>> ----------------------------------------- <
>>>>>> br/>>> ----- Mensaje original ----- <
>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>>
> 1.
>>>>>> Mark the PR as ready to review in github 2. rebase or squash it to
>>>>>> a
>>> r/>>>> single commit and force push to youuur branch
>>>> to clean it up
>>>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>>>> <ii...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>> Hi Otto, Chris <
>>>>>>> br/>>>> we have finally commited the uupdates on the Nifi
>>>>>>> Processor
>>> r/>>>>> to <
>>>> the Pull Request. The changes we have done are the following:
>>>>>>> - deducing avro datatypes from PlcResponse. Here we may check the
>>>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
>>>> r/>>
>>> order to see if it is the best way to do it. <
>>>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>>>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
>>>> r/>>
>>> rising errors during compilation. <
>>>>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
>>>> (comparing to BaseProcessor), as we have included the posibility to
>>>> r/>>
>>> have an input connection into the processor, and indddicate the target
>>> r/>> addressMap through flowfile attributes. TThhe addressMap is now
>>> created in the onTrigger method.
>>>>>>> - we have tested the performance with S7 and Modbus protocols
>>>>>>> r/>>>>>
>>> (using a <
>>>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
>>>> r/>>
>>> template for both protocols, but regarding this, dooo you have any
>>> r/>> testing environment to simulate PLCs??? If that is the case, we
>>> could r/>> prepare the Processors configurratiion to match these ones
>>> (connection Strings and addressMaps).
>>>>>>> br/>>>> Please take a look at the code,, any suggestion will be
>>> r/>>>>> very <
>>>> welcome.
>>>>>>> br/>>>> iñigo <
>>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> r/>>>>> Angulo
>>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>>> r/>>
>>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>>> ------------------------------------------ <
>>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>>> r/>>t;;>>> prrocessor to include the logic of
>>>> getting the values for variables from the PLC response. We tested it
>>>> r/>>
>>> with the <
>>>> S7-1200 and seems to work fine, however we would like to make some
>>>> r/>>
>>> further tests before commiting it. <
>>>>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>>> r/>>>>> the <
>>>> response object, we did it in the following way:
>>>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>>>> PlcValue> responseDataStructure =
>>>> readResponse.getAsPlcValue().getStruct();
>>>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>>>> responseDataStructure.entrySet()) {
>>>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
>>> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>>>>> n d().intType().endUnion().noDefault();
>>>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
>>> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>>>>> d ().doubleType().endUnion().noDefault();
>>>> r/>>>>> }}} ... and so on for the rest of the classes on the package
>>>> (org.apache.plc4x.java.spi.values.*)
>>>>>>> br/>>>> //the builder object is used tto build avro schema, with
>>>> desired datatypes (for example intType())
>>>>>>> }
>>>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you
>>>>>>> think
>>>> there is a better way to access PlcValues, please let us know and we
>>>> r/>>
>>> will update it. <
>>>>>>> br/>>>> We will upload the code soon soo that you can take a
>>>>>>> deeper
>>>> look.
>>>>>>> br/>>>> thank you!!
>>>>>>> iñigo
>>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> r/>>>>> Angulo
>>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>>> r/>>
>>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>>> ------------------------------------------ <
>>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>>> br/>>>>> we have been reviewingg the comments on the pull
>>>>>>>> request,
>>> r/>>>>>> and <
>>>> started to think about the approach of extracting values from
>>>> response
>>> directly.
>>>> During next week, we will work on this and make some updates to the
>>>> r/>>
>>> code with the suggestions you made. We keep you infooormed
>>>>>>>> br/>>>>> thank you, <
>>>>>>>> br/>>>>> iñigo <
>>>>>>>> br/>>>>> ------------------------------------------ br/>>>>>
>>>>>>>> Iñigo
>>>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
>>> telf.:
>>>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
>>>> r/>>
>>> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>>>> ----------------------------------------- <
>>>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you
>>> havee a r/>>>>>> look at the KNX protocol. This <
>>>> doesn't define the usual IEC datatypes we tried to use for all normal
>>> r/>> PLC drivers. <
>>>>>>>> Here we have hundreds of datatypes that don't match any other
>>>> protocol. I think the PLCValue approach would be the simplest.
>>>>>>>> The one thing you have to keep in mind, is that you should check
>>>>>>>> a
>>>> PLCValue, if it's a list (Array type) or a Structure (Which sort of
>>>> r/>>
>>> relates to komplex types with a more sophisticated ssstructure).
>>>>>>>> br/>>>>> Chris <
>>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>>>> FFreitag, 23. April 2021 15:34
>>>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>>> reading
>>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>>> r/>>>>>> now, <
>>>> we are generating the schema ourselves. We have a record writer who
>>>> is
>>> r/>> in charge of reading PLC values. Schema is definnned previously
>>> to r/>> reading the values. We build this schema gggetting the
>>> protocol from the r/>> 'connectionString' (S7, Modbuuss) and the
>>> specified variable type from the 'PLC resource address String'
>>>> containing the list of variable to read. From this we deduce the r/>>
>>> expected Avro datatype when reading, for instance, a word in S7 or a
>>> coil in Modbus.
>>>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
>>>> r/>>
>>> will be much clearerrr and useful. Ideally, getting the actual
>>> datatype r/>> from PLCCVValue when getting the response.
>>>> Regarding this, we tried to keep the previously described 'mapping'
>>>> separated from the rest of the code, so that hopefully it can be r/>>
>>> easily replaced.. <
>>>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>>> r/>>>>>> at <
>>>> the code and let us know what you think. We will fill the ICLA
>>>> document
>>> too.
>>>>>>>> br/>>>>> thank you <
>>>>>>>> iñigo
>>>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>>> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>> -----------------------------------------
>>>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>> reading
>>> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees
>>> from the r/>>>>>> response that wrap the <
>>>> different datatypes. So generally you shouldn't care about the detail
>>> type.
>>>>>>>> br/>>>>> However, you can call ggetObject() which returns the
>>>>>>>> core
>>>> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
>>> getObject will return a short value. <
>>>>>>>> br/>>>>> Does that help??
>>>>>>>> br/>>>>> Chris <
>>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>>>> r/>>>>>>
>>> reading br/>>>>&&> So, you are generating the schema yourself, such
>>> r/>>>&ggtt;>> that
>>>> downstream if they inherit schema they will just get what you r/>>
>>> generate??? And you are trying to do that by the connection string? If
>>> r/>> so, a different way I could imagine doingg woould be to get the
> ’types’
>>> r/>> of the data from the responses themselves. This would be more
> generic.
>>> r/>> The flow I could imagine ( in OnTrigger
>>>> ):
>>>>>>>> br/>>>>> DO READ <
>>>>>>>> IF NOT HAS SCHEMA
>>>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>>>>> r/>>>>>>
>>> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
>>> r/>>>&ggtt;>> from the
>>>> responses.
>>>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>>> r/>>>>>>&gggt; give
>>>> an overview. In Nifi, information packages are called Flowfiles, and
>>>> r/>>
>>> these are the actual units of information that arrre exchanged between
>>> r/>> Procesors, all along the dataflow. FFFlowfiles have two sections
>>> where r/>> we can manage <
>>>> data: Attributes and Content. In the "traditional" Nifi approach, you
>>> r/>> work with both sections, extracting informatiiion from the
>>> Content to r/>> the Attributes and viceversa to perffform operations.
>>> This approach r/>> could have one limitation whheen you are processing
>>> batch data (lines r/>> from a CSV file foor instance), where you need
>>> to split each of the r/>> lines intto ddifferent Flowfiles. Thus, a
>>> 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process, each
>>> of them containing a single record.
>>>>>>>>> br/>>>>>> On later versioons of the product, they introduced the
>>>> Record oriented approach. This approach allows you to manage multiple
>>> r/>> records on a single FFFlowfile's Content, as long as these
>>> records have all the same schema.
>>>> This means that the operations defined by the Processors are applied
>>>> r/>>
>>> simultaneously to the whole content at once. FFFollowing with the r/>>
>>> previous example, a 1000 line CSV file couuldd produce a single
>>> Flowfile r/>> with a content of <
>>>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
>>> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
>>> Processors uuse Writers and Readers to present this information in the
>>> r/>> desiired format (such as Avro, Json, CSV, etc). Basically, with
>>> the br/>&> record oriented approach, Nifi introduced multiple new
>>> Processors, and r/>> also included the Record version of many of the
>>> """old" ones. Using this r/>> Record approach, Nifi perfomance
>>> enhances notably, specially when working with large structured
> information.
>>>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>>>>> r/>>>>>>>
>>> Procccessor,
>>>> based on the previously existing one Plc4xSourceProcessor, to read
>>>> r/>>
>>> values from the devices. We have also included a READDDME on the r/>>
>>> plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor
>>> r/>> configuration and giving an example. Mooreover, we put a nifi
>>> template r/>> with a dataflow for testiing these processors, if useful.
>>>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>> r/>>>>>>>;; that
>>>> is right. We added the writer capability to the existing
>>>> PLC4XSourceProcessor, so that it formats the output to the desired
>>> configuration in a record manner.
>>>> At the actual implementation, we did this "protocol adaptation" from
>>>> r/>>
>>> the sintax of the particular properties on Procccessor's configuration.
>>> r/>> FFFor example, from connection string 's7://IP:PORT', we extract
>>> the
>>> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro
>>> datatypes for build the record output schema.
>>>> However, here we dont have vast experience with PLC4X libraries, and
>>>> r/>>
>>> for sure there will be better ways for doing this.
>>>>>>>>> Also about the Base Processor, we were thinking that maybe the
>>> r/>>>>>>> best <
>>>> approach could be to have this Base Processor, and then implement
>>>> r/>>
>>> readers for particular protocols as Controller Serviccces. But here
>>> r/>> also, it could be very helpful to have your opppinion.
>>>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>>>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
>>> r/>> defined some naming br/>&gggt;>>>> conventions, or expected
>>> structure to r/>> ffaaciliitate later work. At the br/>>>>>> present,
>>> we have a fork of r/>> the project where we have been working on
>>> br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed tthe content
>>> of our fork (fetch/merge
>>>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>>>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with
>>>> our
>>> commits?
>>>> how do you br/>>>&>>> prefer to receive the code? (we are not very
>>>> r/>>
>>> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
>>> problemss.....)
>>>>>>>>> br/>>>>>> thank you in addvance
>>>>>>>>> br/>>>>>> iñigo <
>>>>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net ::
>>> connsultoría.openSource
>>>>>>>>> telf.: 747412337
>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>>> -----------------------------------------
>>>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>>> r/>>>>>>>
>>> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
>>> r/>>>>>>> shouuld also think of potentialllyy providing some
>>>> information on supported configuration options.
>>>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>> r/>>>>>>> these <
>>>> options and they have these datatypes and mean this"
>>>>>>>>> Additionally, the transports could too say: "I generally have
>>> r/>>>>>>> these <
>>>> options and they have these datatypes and mean this"
>>>>>>>>> br/>>>>>> I would be our StreamPipes friends would love
>>>>>>>>> something
>>>> like that? Right?
>>>>>>>>> br/>>>>>> Chris <
>>>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter
>>> on r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>>>> be happy to review your processor.
>>>>>>>>> If I understand what you are saying correctly, you have a single
>>>> processor which supports record writing output?
>>>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>>>> br/>>>>>> And that you haave, for configuration purposes for
>>>>>>>>> that
>>>> processor created support on a per protocol basis for configuration
>>>> r/>>
>>> and validation???
>>>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>> r/>>>>>>>;; etc,
>>>> it may be better to have a base processor, and derived processors per
>>> r/>> protocol to handle those differences. <
>>>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>>> r/>>>>>&&>>> Nifi
>>>> integration part of the project. We have created a Record oriented
>>>> r/>>
>>> processor for reading PLC data. It is based on the prrrevious existing
>>> r/>> SourceProcessor, but works with records, uussing a Nifi Writer
>>> (such as r/>> Avro, Json, and so on) to writte data on flowfiles
>>> content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the code on our
>>> fork with thee actual PLC4X git r/>> repo about 2 weeks ago, and
>>> tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi.
>>> Alsoo, onee of our customers has recently started r/>> to use it for
>>> validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7
>>> and Modbus oover TCP. This is becaause we had to write some r/>>
>>> classes to map connectionSString annd variableList properties (sintax)
>>> r/>> of the processoor to the actual protocol, to be able to build
>>> then avro r/>> scchema for ooutput flowfile, taking into account
>>> variable datatypes, br/>> etcc. We only did this for
>>>> S7 and Modbus. I am sure that there is a better way to do this, so at
>>> r/>> this point you maybe could take a look to find theee best
>>> solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>>
>>> br/>>>>>>> If you find br/>> this useful, we could do a ppull request to
> the
>>> main PLC4x repo.
>>> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
>>> regards,
>>>>>>>>>> iñigo
>>>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>>>> Iñigo Angulo
>>>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>>>> telf.: 747412337
>>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>>> 48950 Erandio (Bizkaia)
> > > > >>>>>> -----------------------------------------
Re: AW: AW: AW: Nifi integration record oriented processor for reading
Posted by Otto Fowler <ot...@gmail.com>.
I just reviewed the latest change set and resolved the conversations that
could be resolved, there are a couple of showstoppers that are left however.
The top two:
- you have supported expression language for the plc address property, but
you do not evaluate the expression language when accessing the property
- the caching scheme you have implemented is incorrect I think
- need to change to asyncCommit
I have commented on the existing items as I reviewed.
Still great work
From: Gustavo Fernandez <gu...@zylk.net.invalid> <gu...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
<de...@plc4x.apache.org>, Gustavo
Fernandez <gu...@zylk.net> <gu...@zylk.net>
Date: October 25, 2021 at 11:15:00
To: Christofer Dutz <ch...@c-ware.de> <ch...@c-ware.de>
Cc: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: AW: AW: AW: Nifi integration record oriented processor for
reading
Hi Chris,
We are making a review of pending open topics on the PR, we think most
important issues are resolved by now.. However, we still have some doubts
regarding the actual mapping between PlcValues (from readResponse),
AvroSchema (to build the record schema) and the Java types. br/>At the
present there is a mapping that is working, but we would like you to take a
look to give your opinion, as we think there could be a better way to make
the casting there. The code to be reviewed is
org.apache.plc4x.nifi.util.Plc4xCommon (
https://github.com/zylklab/plc4x/blob/c535aa3698515375792fde3bc424800466b35cab/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/util/Plc4xCommon.java)
thank you in advance!
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernández" <gu...@zylk.net>
> Enviados: Viernes, 22 de Octubre 2021 9:17:38
> Asunto: AW: AW: AW: Nifi integration record oriented processor for
reading
> Ping ...
> br/>> Just wanted to check-in on you foks here ... iss there anything I
can do to help
> finish this?
> br/>> Chris <
> br/>> -----Ursprüngliche Nachricht----- <
> Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
> Gesendet: Mittwoch, 25. August 2021 12:10
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: AW: AW: Nifi integration record oriented processor for
reading
> br/>> Hi, <
> br/>> thanks Otto <
> br/>> --gustavo <
> br/>> br/>> ----- Mensaje original ----- <
>> De: "Otto Fowler" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Martes, 24 de Agosto 2021 19:22:49
>> Asunto: Re: AW: AW: Nifi integration record oriented processor for
>> reading
> br/>>> ok, new review up <
>> br/>>> FFrom: Otto Fowler <ot...@gmail.com> <
ottobackwards@gmail.com>
>> Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
>> Date: August 24, 2021 at 12:30:23
>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>> reading
>> br/>>> Sorry, I’ve been crushed by IRL wwork, I’ll review this as soon
as
>> possible
>> br/>>> FFrom: Iñigo Angulo <ia...@zylk.net.invalid>
>> <ia...@zylk.net.invalid>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
>> <de...@plc4x.apache.org>
>> Date: August 19, 2021 at 02:42:02
>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>> reading
>> br/>>> Hi Chris, <
>> br/>>> I also had the confirmation from Apachhe that my ICLA is filed.
My
>> username is inigoao.
>> br/>>> Our collegue Alvaro has also sent it, II think in few days he
will
>> receive the confirmation. His username is asellart.
>> br/>>> regards, <
>> iñigo
>> br/>>> ------------------------------------------ r/>Iñigo Annngulo r/>
<
>> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de
>> Axpe,
>> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
>> r/>----------------------------------------- <
>> br/>>> ----- Mensaje original ----- <
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
>> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
>> Asunto: AW: AW: Nifi integration record oriented processor for reading
>> br/>>> Hi Gustavo, <
>> br/>>> great ... I had something in the back of my mind, but I wasn't
sure
>> anymore ... then we're safe on that side ;-)
>> br/>>> Chris <
>> br/>>> -----Ursprüngliche Nachricht------
>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch,
18.
>> August 2021 12:32
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Nifi integration record oriented processor for
>> reading
>> br/>>> Hi Chris <
>> br/>>> I sent my ICLA a month ago. The responnse from ASF was:
>> br/>>> [[...]
>> This message acknowledges receipt of your ICLA, which has been filed
>> in the Apache Software Foundation records.
>> With this message, the PLC4X PMC has been notified that your ICLA has
>> been filed.
>> [...]
>> br/>>> My apache user id is zgus <
>> br/>>> un saludo <
>> --gustavo
>> br/>>> ------------------------------------------
>> Gustavo Fernández
>> br/>>> ZYLK.net :: consultoría.openSourrce
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>> br/>>> movil: +34 637546184 <
>> ofic.: +34 944272119
>> email: gus at zylk.net
>> skype: gus.zylk
>> br/>>> http://twitter.com/zylknet <
>> http://www.zylk.net/web/guest/web-2-0/blog
>> -----------------------------------------
>> br/>>> ----- Mensaje original ----- <
>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>> Para: dev@plc4x.apache.org
>>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>>> Asunto: AW: Nifi integration record oriented processor for reading
>> br/>>>> HI Inigo, <
>>> r/>> great news ... I hope Otto can spare some cycleees to review the
>>> PR
>> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>>> r/>> As it's been quite some time ... I am pretty suuure we have an
>>> ICLA
>> on r/>> file from your side, correct???
>>> r/>> Chris <
>>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>>> Gesendet: Mittwoch, 18. August 2021 10:46
>>> An: dev <de...@plc4x.apache.org>
>>> Betreff: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the
>>> Nifi PR and made a commit r/>>
>> including revisions for all (or most) topppics.
>>> r/>> the last changes we did were related to reforciiing the methods
>>> for
>> r/>> datatype conversion (for the record schheema construction), and
>> r/>> extending the tests. About this lasst one, we changed the r/>>
>> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This
>> led r/>> to renaming the test claassess package to match
>> BasePlc4xProcessor's one r/>>
>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
>> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict
on the
>> PR, but we think it can be easily corrected when doing the merge.
>>> r/>> At the moment, we think this nifi feature is prrretty much
>> completed. Of r/>> course there are things that can bbee extended yet,
>> which we will continue working on.
>>> So please, if you could take a look and check if you miss something
>>> r/>>
>> would be great. Hopefully this could be soon tested by users in the
>> r/>> community too and get some feedback :) <
>>> r/>> thank you <
>>> iñigo
>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>> ZYLK.net :: consultoría.openSource <
>>> telf.: 747412337
>>> Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> -----------------------------------------
>>> r/>> ----- Mensaje original ----- <
>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, < r/>> thank you for the review of the PR and your
>>> cooomments.
>>> r/>> We have answered them, and will review the sectttions you
suggested.
>> I r/>> would like to know if there are any tthings you want us to
priorize.
>> r/>> Maybe you have some deadliinees for the next release or
>> something, and r/>> we could try too geet things done by this time. If
>> you miss some task we r/>> coould do here before doing the merge, please
let us
>> know.
>>> r/>> iñigo <
>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>> ZYLK.net :: consultoría.openSource <
>>> telf.: 747412337
>>> Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> -----------------------------------------
>>> r/>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> Great, < I’m doing the review, so you should mark it ready :)
>>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
>> <ia...@zylk.net.invalid>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
>> plc4x.apache.org>
>>> Date: June 24, 2021 at 04:52:19
>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Subject: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, < r/>> I have opened the new pull request, from a
>>> dedddicated branch in our
>> r/>> fork, and a commit per contributor, thhhe team members that have
>> been working on this.
>>> r/>> Please take a look, and let me know if you wanttt me to mark it
>>> as
>> ready r/>> to review. <
>>> r/>> iñigo <
>>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo
>>> br/> <
>> ZYLK.net ::
>>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
>>> r/>>
>> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>>> br/>----------------------------------------- < r/>> ----- Mensaje
>>> original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo
>>> Annngulo <ia...@zylk.net.INVALID>
>> wrote:
>>>> br/>> Hi Otto, <
>>>> br/>> So we did this workaround, we re-forked the reepo and added
>>>> the
>>> changes of the nifi-record feature on a dedicated branch. I closed
>>> the
>> r/>> previous PR, will upload the new one on the follooowing days (we
>> are r/>> doing some last test just in case we forrggot to copy
>> something...)
>>>> br/>> I will keep you informed. <
>>>> br/>> regards, <
>>>> iñigo
>>>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>>>> r/>>>
>> br/>> <
>>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
>> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>>
>> 48950 r/>> Erandiooo
>>> (Bizkaia) br/>> ----------------------------------------- <
>>>> br/>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>>>
>> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
>> r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>>> wrote:
>>>>> br/>>> Hi Otto, <
>>>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>>>> r/>>>>
>> having <
>>> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
>>> r/>>
>> on september 2020, and started mmmaking tests and commits to our fork
>> (on 'develop'
>>> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
>> specifying the ID of our first commit. But when the fillle is open for
>> r/>> interactive mode, it gets <
>>> ~900 commits on the develop branch (belonging to members of the PLC4X
>> r/>> community). I think this happens because beforeee opening the
>> r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X
>> repo, to get the updates of the code.
>>> So, I understand that in the interactive mode file, I have to leave
>>> r/>>
>> all commits to 'pick' (by default), and change our cccommits of the
>> nifi feature to 'squash'
>>> except from the first one (which also remains as 'pick'). However,
>>> r/>>
>> when I tried this, many conflicts appear, almost one per each commit
>> (comunity members'
>>> commits)...
>>>>> I may be doing something wrong (never did a rebase before...) and I
>>> prefered just to ask, as I dont want to break or cause any conflict
>>> to
>> r/>> the repo code.. If you see anything Im missing plllease let me
know.
>>>>> br/>>> As a workaround, I was thinking we coulld close the PR,
>>>>> re-do
>> r/>>>> the <
>>> fork of the PLC4X repo, and add the changes to the code on a
>>> dedicated
>> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
>> clearer...
>>>>> What do you think?
>>>>> br/>>> thank you, <
>>>>> iñigo
>>>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>>>> r/>>>>
>> Angulo <
>>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
>> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo
>> 201-203 rrr/>> br/>>> 48950 Erandio
>>> (Bizkaia) br/>>> ----------------------------------------- <
>>>>> br/>>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>>
1.
>>>>> Mark the PR as ready to review in github 2. rebase or squash it to
>>>>> a
>> r/>>>> single commit and force push to youuur branch
>>> to clean it up
>>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>>> <ii...@zylk.net.INVALID> wrote:
>>>>>> br/>>>> Hi Otto, Chris <
>>>>>> br/>>>> we have finally commited the uupdates on the Nifi
>>>>>> Processor
>> r/>>>>> to <
>>> the Pull Request. The changes we have done are the following:
>>>>>> - deducing avro datatypes from PlcResponse. Here we may check the
>>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
>>> r/>>
>> order to see if it is the best way to do it. <
>>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
>>> r/>>
>> rising errors during compilation. <
>>>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
>>> (comparing to BaseProcessor), as we have included the posibility to
>>> r/>>
>> have an input connection into the processor, and indddicate the target
>> r/>> addressMap through flowfile attributes. TThhe addressMap is now
>> created in the onTrigger method.
>>>>>> - we have tested the performance with S7 and Modbus protocols
>>>>>> r/>>>>>
>> (using a <
>>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
>>> r/>>
>> template for both protocols, but regarding this, dooo you have any
>> r/>> testing environment to simulate PLCs??? If that is the case, we
>> could r/>> prepare the Processors configurratiion to match these ones
>> (connection Strings and addressMaps).
>>>>>> br/>>>> Please take a look at the code,, any suggestion will be
>> r/>>>>> very <
>>> welcome.
>>>>>> br/>>>> iñigo <
>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>> r/>>>>> Angulo
>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>> r/>>
>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>> ------------------------------------------ <
>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>> r/>>t;;>>> prrocessor to include the logic of
>>> getting the values for variables from the PLC response. We tested it
>>> r/>>
>> with the <
>>> S7-1200 and seems to work fine, however we would like to make some
>>> r/>>
>> further tests before commiting it. <
>>>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>> r/>>>>> the <
>>> response object, we did it in the following way:
>>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>>> PlcValue> responseDataStructure =
>>> readResponse.getAsPlcValue().getStruct();
>>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>>> responseDataStructure.entrySet()) {
>>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
>> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>>>> n d().intType().endUnion().noDefault();
>>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
>> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>>>> d ().doubleType().endUnion().noDefault();
>>> r/>>>>> }}} ... and so on for the rest of the classes on the package
>>> (org.apache.plc4x.java.spi.values.*)
>>>>>> br/>>>> //the builder object is used tto build avro schema, with
>>> desired datatypes (for example intType())
>>>>>> }
>>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you
>>>>>> think
>>> there is a better way to access PlcValues, please let us know and we
>>> r/>>
>> will update it. <
>>>>>> br/>>>> We will upload the code soon soo that you can take a
>>>>>> deeper
>>> look.
>>>>>> br/>>>> thank you!!
>>>>>> iñigo
>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>> r/>>>>> Angulo
>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>> r/>>
>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>> ------------------------------------------ <
>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>> br/>>>>> we have been reviewingg the comments on the pull
>>>>>>> request,
>> r/>>>>>> and <
>>> started to think about the approach of extracting values from
>>> response
>> directly.
>>> During next week, we will work on this and make some updates to the
>>> r/>>
>> code with the suggestions you made. We keep you infooormed
>>>>>>> br/>>>>> thank you, <
>>>>>>> br/>>>>> iñigo <
>>>>>>> br/>>>>> ------------------------------------------ br/>>>>>
>>>>>>> Iñigo
>>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
>> telf.:
>>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
>>> r/>>
>> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>>> ----------------------------------------- <
>>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you
>> havee a r/>>>>>> look at the KNX protocol. This <
>>> doesn't define the usual IEC datatypes we tried to use for all normal
>> r/>> PLC drivers. <
>>>>>>> Here we have hundreds of datatypes that don't match any other
>>> protocol. I think the PLCValue approach would be the simplest.
>>>>>>> The one thing you have to keep in mind, is that you should check
>>>>>>> a
>>> PLCValue, if it's a list (Array type) or a Structure (Which sort of
>>> r/>>
>> relates to komplex types with a more sophisticated ssstructure).
>>>>>>> br/>>>>> Chris <
>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>>> FFreitag, 23. April 2021 15:34
>>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>> reading
>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>> r/>>>>>> now, <
>>> we are generating the schema ourselves. We have a record writer who
>>> is
>> r/>> in charge of reading PLC values. Schema is definnned previously
>> to r/>> reading the values. We build this schema gggetting the
>> protocol from the r/>> 'connectionString' (S7, Modbuuss) and the
>> specified variable type from the 'PLC resource address String'
>>> containing the list of variable to read. From this we deduce the r/>>
>> expected Avro datatype when reading, for instance, a word in S7 or a
>> coil in Modbus.
>>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
>>> r/>>
>> will be much clearerrr and useful. Ideally, getting the actual
>> datatype r/>> from PLCCVValue when getting the response.
>>> Regarding this, we tried to keep the previously described 'mapping'
>>> separated from the rest of the code, so that hopefully it can be r/>>
>> easily replaced.. <
>>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>> r/>>>>>> at <
>>> the code and let us know what you think. We will fill the ICLA
>>> document
>> too.
>>>>>>> br/>>>>> thank you <
>>>>>>> iñigo
>>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>> -----------------------------------------
>>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>> reading
>> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees
>> from the r/>>>>>> response that wrap the <
>>> different datatypes. So generally you shouldn't care about the detail
>> type.
>>>>>>> br/>>>>> However, you can call ggetObject() which returns the
>>>>>>> core
>>> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
>> getObject will return a short value. <
>>>>>>> br/>>>>> Does that help??
>>>>>>> br/>>>>> Chris <
>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>>> r/>>>>>>
>> reading br/>>>>&&> So, you are generating the schema yourself, such
>> r/>>>&ggtt;>> that
>>> downstream if they inherit schema they will just get what you r/>>
>> generate??? And you are trying to do that by the connection string? If
>> r/>> so, a different way I could imagine doingg woould be to get the
’types’
>> r/>> of the data from the responses themselves. This would be more
generic.
>> r/>> The flow I could imagine ( in OnTrigger
>>> ):
>>>>>>> br/>>>>> DO READ <
>>>>>>> IF NOT HAS SCHEMA
>>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>>>> r/>>>>>>
>> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
>> r/>>>&ggtt;>> from the
>>> responses.
>>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>> r/>>>>>>&gggt; give
>>> an overview. In Nifi, information packages are called Flowfiles, and
>>> r/>>
>> these are the actual units of information that arrre exchanged between
>> r/>> Procesors, all along the dataflow. FFFlowfiles have two sections
>> where r/>> we can manage <
>>> data: Attributes and Content. In the "traditional" Nifi approach, you
>> r/>> work with both sections, extracting informatiiion from the
>> Content to r/>> the Attributes and viceversa to perffform operations.
>> This approach r/>> could have one limitation whheen you are processing
>> batch data (lines r/>> from a CSV file foor instance), where you need
>> to split each of the r/>> lines intto ddifferent Flowfiles. Thus, a
>> 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process, each
>> of them containing a single record.
>>>>>>>> br/>>>>>> On later versioons of the product, they introduced the
>>> Record oriented approach. This approach allows you to manage multiple
>> r/>> records on a single FFFlowfile's Content, as long as these
>> records have all the same schema.
>>> This means that the operations defined by the Processors are applied
>>> r/>>
>> simultaneously to the whole content at once. FFFollowing with the r/>>
>> previous example, a 1000 line CSV file couuldd produce a single
>> Flowfile r/>> with a content of <
>>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
>> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
>> Processors uuse Writers and Readers to present this information in the
>> r/>> desiired format (such as Avro, Json, CSV, etc). Basically, with
>> the br/>&> record oriented approach, Nifi introduced multiple new
>> Processors, and r/>> also included the Record version of many of the
>> """old" ones. Using this r/>> Record approach, Nifi perfomance
>> enhances notably, specially when working with large structured
information.
>>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>>>> r/>>>>>>>
>> Procccessor,
>>> based on the previously existing one Plc4xSourceProcessor, to read
>>> r/>>
>> values from the devices. We have also included a READDDME on the r/>>
>> plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor
>> r/>> configuration and giving an example. Mooreover, we put a nifi
>> template r/>> with a dataflow for testiing these processors, if useful.
>>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>> r/>>>>>>>;; that
>>> is right. We added the writer capability to the existing
>>> PLC4XSourceProcessor, so that it formats the output to the desired
>> configuration in a record manner.
>>> At the actual implementation, we did this "protocol adaptation" from
>>> r/>>
>> the sintax of the particular properties on Procccessor's configuration.
>> r/>> FFFor example, from connection string 's7://IP:PORT', we extract
>> the
>> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro
>> datatypes for build the record output schema.
>>> However, here we dont have vast experience with PLC4X libraries, and
>>> r/>>
>> for sure there will be better ways for doing this.
>>>>>>>> Also about the Base Processor, we were thinking that maybe the
>> r/>>>>>>> best <
>>> approach could be to have this Base Processor, and then implement
>>> r/>>
>> readers for particular protocols as Controller Serviccces. But here
>> r/>> also, it could be very helpful to have your opppinion.
>>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
>> r/>> defined some naming br/>&gggt;>>>> conventions, or expected
>> structure to r/>> ffaaciliitate later work. At the br/>>>>>> present,
>> we have a fork of r/>> the project where we have been working on
>> br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed tthe content
>> of our fork (fetch/merge
>>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with
>>> our
>> commits?
>>> how do you br/>>>&>>> prefer to receive the code? (we are not very
>>> r/>>
>> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
>> problemss.....)
>>>>>>>> br/>>>>>> thank you in addvance
>>>>>>>> br/>>>>>> iñigo <
>>>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net ::
>> connsultoría.openSource
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>> -----------------------------------------
>>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>> r/>>>>>>>
>> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
>> r/>>>>>>> shouuld also think of potentialllyy providing some
>>> information on supported configuration options.
>>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>> r/>>>>>>> these <
>>> options and they have these datatypes and mean this"
>>>>>>>> Additionally, the transports could too say: "I generally have
>> r/>>>>>>> these <
>>> options and they have these datatypes and mean this"
>>>>>>>> br/>>>>>> I would be our StreamPipes friends would love
>>>>>>>> something
>>> like that? Right?
>>>>>>>> br/>>>>>> Chris <
>>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter
>> on r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>>> be happy to review your processor.
>>>>>>>> If I understand what you are saying correctly, you have a single
>>> processor which supports record writing output?
>>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>>> br/>>>>>> And that you haave, for configuration purposes for
>>>>>>>> that
>>> processor created support on a per protocol basis for configuration
>>> r/>>
>> and validation???
>>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>> r/>>>>>>>;; etc,
>>> it may be better to have a base processor, and derived processors per
>> r/>> protocol to handle those differences. <
>>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>> r/>>>>>&&>>> Nifi
>>> integration part of the project. We have created a Record oriented
>>> r/>>
>> processor for reading PLC data. It is based on the prrrevious existing
>> r/>> SourceProcessor, but works with records, uussing a Nifi Writer
>> (such as r/>> Avro, Json, and so on) to writte data on flowfiles
>> content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the code on our
>> fork with thee actual PLC4X git r/>> repo about 2 weeks ago, and
>> tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi.
>> Alsoo, onee of our customers has recently started r/>> to use it for
>> validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7
>> and Modbus oover TCP. This is becaause we had to write some r/>>
>> classes to map connectionSString annd variableList properties (sintax)
>> r/>> of the processoor to the actual protocol, to be able to build
>> then avro r/>> scchema for ooutput flowfile, taking into account
>> variable datatypes, br/>> etcc. We only did this for
>>> S7 and Modbus. I am sure that there is a better way to do this, so at
>> r/>> this point you maybe could take a look to find theee best
>> solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>>
>> br/>>>>>>> If you find br/>> this useful, we could do a ppull request to
the
>> main PLC4x repo.
>> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
>> regards,
>>>>>>>>> iñigo
>>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>>> Iñigo Angulo
>>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>>> telf.: 747412337
>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>> 48950 Erandio (Bizkaia)
> > > >>>>>> -----------------------------------------
Re: AW: AW: AW: Nifi integration record oriented processor for
reading
Posted by Gustavo Fernandez <gu...@zylk.net.INVALID>.
Hi Chris,
We are making a review of pending open topics on the PR, we think most important issues are resolved by now.. However, we still have some doubts regarding the actual mapping between PlcValues (from readResponse), AvroSchema (to build the record schema) and the Java types.
At the present there is a mapping that is working, but we would like you to take a look to give your opinion, as we think there could be a better way to make the casting there. The code to be reviewed is org.apache.plc4x.nifi.util.Plc4xCommon (https://github.com/zylklab/plc4x/blob/c535aa3698515375792fde3bc424800466b35cab/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/util/Plc4xCommon.java)
thank you in advance!
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernández" <gu...@zylk.net>
> Enviados: Viernes, 22 de Octubre 2021 9:17:38
> Asunto: AW: AW: AW: Nifi integration record oriented processor for reading
> Ping ...
>
> Just wanted to check-in on you foks here ... is there anything I can do to help
> finish this?
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
> Gesendet: Mittwoch, 25. August 2021 12:10
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: AW: AW: Nifi integration record oriented processor for reading
>
> Hi,
>
> thanks Otto
>
> --gustavo
>
>
> ----- Mensaje original -----
>> De: "Otto Fowler" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Martes, 24 de Agosto 2021 19:22:49
>> Asunto: Re: AW: AW: Nifi integration record oriented processor for
>> reading
>
>> ok, new review up
>>
>> From: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
>> Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
>> Date: August 24, 2021 at 12:30:23
>> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>> reading
>>
>> Sorry, I’ve been crushed by IRL work, I’ll review this as soon as
>> possible
>>
>> From: Iñigo Angulo <ia...@zylk.net.invalid>
>> <ia...@zylk.net.invalid>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
>> <de...@plc4x.apache.org>
>> Date: August 19, 2021 at 02:42:02
>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
>> Subject: Re: AW: AW: Nifi integration record oriented processor for
>> reading
>>
>> Hi Chris,
>>
>> I also had the confirmation from Apache that my ICLA is filed. My
>> username is inigoao.
>>
>> Our collegue Alvaro has also sent it, I think in few days he will
>> receive the confirmation. His username is asellart.
>>
>> regards,
>> iñigo
>>
>> ----------------------------------------- r/>Iñigo Annngulo r/> <
>> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de
>> Axpe,
>> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
>> r/>----------------------------------------- <
>>
>> ----- Mensaje original -----
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
>> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
>> Asunto: AW: AW: Nifi integration record oriented processor for reading
>>
>> Hi Gustavo,
>>
>> great ... I had something in the back of my mind, but I wasn't sure
>> anymore ... then we're safe on that side ;-)
>>
>> Chris
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch, 18.
>> August 2021 12:32
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Nifi integration record oriented processor for
>> reading
>>
>> Hi Chris
>>
>> I sent my ICLA a month ago. The response from ASF was:
>>
>> [...]
>> This message acknowledges receipt of your ICLA, which has been filed
>> in the Apache Software Foundation records.
>> With this message, the PLC4X PMC has been notified that your ICLA has
>> been filed.
>> [...]
>>
>> My apache user id is zgus
>>
>> un saludo
>> --gustavo
>>
>> -----------------------------------------
>> Gustavo Fernández
>>
>> ZYLK.net :: consultoría.openSource
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>>
>> movil: +34 637546184
>> ofic.: +34 944272119
>> email: gus at zylk.net
>> skype: gus.zylk
>>
>> http://twitter.com/zylknet
>> http://www.zylk.net/web/guest/web-2-0/blog
>> -----------------------------------------
>>
>> ----- Mensaje original -----
>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>> Para: dev@plc4x.apache.org
>>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>>> Asunto: AW: Nifi integration record oriented processor for reading
>>
>>> HI Inigo,
>>> r/>> great news ... I hope Otto can spare some cycleees to review the
>>> PR
>> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>>> r/>> As it's been quite some time ... I am pretty suuure we have an
>>> ICLA
>> on r/>> file from your side, correct???
>>> r/>> Chris <
>>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>>> Gesendet: Mittwoch, 18. August 2021 10:46
>>> An: dev <de...@plc4x.apache.org>
>>> Betreff: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the
>>> Nifi PR and made a commit r/>>
>> including revisions for all (or most) topppics.
>>> r/>> the last changes we did were related to reforciiing the methods
>>> for
>> r/>> datatype conversion (for the record schheema construction), and
>> r/>> extending the tests. About this lasst one, we changed the r/>>
>> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This
>> led r/>> to renaming the test claassess package to match
>> BasePlc4xProcessor's one r/>>
>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
>> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict on the
>> PR, but we think it can be easily corrected when doing the merge.
>>> r/>> At the moment, we think this nifi feature is prrretty much
>> completed. Of r/>> course there are things that can bbee extended yet,
>> which we will continue working on.
>>> So please, if you could take a look and check if you miss something
>>> r/>>
>> would be great. Hopefully this could be soon tested by users in the
>> r/>> community too and get some feedback :) <
>>> r/>> thank you <
>>> iñigo
>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>> ZYLK.net :: consultoría.openSource <
>>> telf.: 747412337
>>> Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> -----------------------------------------
>>> r/>> ----- Mensaje original ----- <
>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, < r/>> thank you for the review of the PR and your
>>> cooomments.
>>> r/>> We have answered them, and will review the sectttions you suggested.
>> I r/>> would like to know if there are any tthings you want us to priorize.
>> r/>> Maybe you have some deadliinees for the next release or
>> something, and r/>> we could try too geet things done by this time. If
>> you miss some task we r/>> coould do here before doing the merge, please let us
>> know.
>>> r/>> iñigo <
>>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>>> ZYLK.net :: consultoría.openSource <
>>> telf.: 747412337
>>> Ribera de Axpe, 11
>>> Edificio A, modulo 201-203
>>> 48950 Erandio (Bizkaia)
>>> -----------------------------------------
>>> r/>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> Great, < I’m doing the review, so you should mark it ready :)
>>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
>> <ia...@zylk.net.invalid>
>>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
>> plc4x.apache.org>
>>> Date: June 24, 2021 at 04:52:19
>>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>>> Subject: Re: Nifi integration record oriented processor for reading
>>> r/>> Hi Otto, < r/>> I have opened the new pull request, from a
>>> dedddicated branch in our
>> r/>> fork, and a commit per contributor, thhhe team members that have
>> been working on this.
>>> r/>> Please take a look, and let me know if you wanttt me to mark it
>>> as
>> ready r/>> to review. <
>>> r/>> iñigo <
>>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo
>>> br/> <
>> ZYLK.net ::
>>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
>>> r/>>
>> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>>> br/>----------------------------------------- < r/>> ----- Mensaje
>>> original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo
>>> Annngulo <ia...@zylk.net.INVALID>
>> wrote:
>>>> br/>> Hi Otto, <
>>>> br/>> So we did this workaround, we re-forked the reepo and added
>>>> the
>>> changes of the nifi-record feature on a dedicated branch. I closed
>>> the
>> r/>> previous PR, will upload the new one on the follooowing days (we
>> are r/>> doing some last test just in case we forrggot to copy
>> something...)
>>>> br/>> I will keep you informed. <
>>>> br/>> regards, <
>>>> iñigo
>>>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>>>> r/>>>
>> br/>> <
>>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
>> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>>
>> 48950 r/>> Erandiooo
>>> (Bizkaia) br/>> ----------------------------------------- <
>>>> br/>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> r/>>>
>> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
>> r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>>> wrote:
>>>>> br/>>> Hi Otto, <
>>>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>>>> r/>>>>
>> having <
>>> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
>>> r/>>
>> on september 2020, and started mmmaking tests and commits to our fork
>> (on 'develop'
>>> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
>> specifying the ID of our first commit. But when the fillle is open for
>> r/>> interactive mode, it gets <
>>> ~900 commits on the develop branch (belonging to members of the PLC4X
>> r/>> community). I think this happens because beforeee opening the
>> r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X
>> repo, to get the updates of the code.
>>> So, I understand that in the interactive mode file, I have to leave
>>> r/>>
>> all commits to 'pick' (by default), and change our cccommits of the
>> nifi feature to 'squash'
>>> except from the first one (which also remains as 'pick'). However,
>>> r/>>
>> when I tried this, many conflicts appear, almost one per each commit
>> (comunity members'
>>> commits)...
>>>>> I may be doing something wrong (never did a rebase before...) and I
>>> prefered just to ask, as I dont want to break or cause any conflict
>>> to
>> r/>> the repo code.. If you see anything Im missing plllease let me know.
>>>>> br/>>> As a workaround, I was thinking we coulld close the PR,
>>>>> re-do
>> r/>>>> the <
>>> fork of the PLC4X repo, and add the changes to the code on a
>>> dedicated
>> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
>> clearer...
>>>>> What do you think?
>>>>> br/>>> thank you, <
>>>>> iñigo
>>>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>>>> r/>>>>
>> Angulo <
>>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
>> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo
>> 201-203 rrr/>> br/>>> 48950 Erandio
>>> (Bizkaia) br/>>> ----------------------------------------- <
>>>>> br/>>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>> 1.
>>>>> Mark the PR as ready to review in github 2. rebase or squash it to
>>>>> a
>> r/>>>> single commit and force push to youuur branch
>>> to clean it up
>>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>>> <ii...@zylk.net.INVALID> wrote:
>>>>>> br/>>>> Hi Otto, Chris <
>>>>>> br/>>>> we have finally commited the uupdates on the Nifi
>>>>>> Processor
>> r/>>>>> to <
>>> the Pull Request. The changes we have done are the following:
>>>>>> - deducing avro datatypes from PlcResponse. Here we may check the
>>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
>>> r/>>
>> order to see if it is the best way to do it. <
>>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
>>> r/>>
>> rising errors during compilation. <
>>>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
>>> (comparing to BaseProcessor), as we have included the posibility to
>>> r/>>
>> have an input connection into the processor, and indddicate the target
>> r/>> addressMap through flowfile attributes. TThhe addressMap is now
>> created in the onTrigger method.
>>>>>> - we have tested the performance with S7 and Modbus protocols
>>>>>> r/>>>>>
>> (using a <
>>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
>>> r/>>
>> template for both protocols, but regarding this, dooo you have any
>> r/>> testing environment to simulate PLCs??? If that is the case, we
>> could r/>> prepare the Processors configurratiion to match these ones
>> (connection Strings and addressMaps).
>>>>>> br/>>>> Please take a look at the code,, any suggestion will be
>> r/>>>>> very <
>>> welcome.
>>>>>> br/>>>> iñigo <
>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>> r/>>>>> Angulo
>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>> r/>>
>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>> ------------------------------------------ <
>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>> r/>>t;;>>> prrocessor to include the logic of
>>> getting the values for variables from the PLC response. We tested it
>>> r/>>
>> with the <
>>> S7-1200 and seems to work fine, however we would like to make some
>>> r/>>
>> further tests before commiting it. <
>>>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>> r/>>>>> the <
>>> response object, we did it in the following way:
>>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>>> PlcValue> responseDataStructure =
>>> readResponse.getAsPlcValue().getStruct();
>>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>>> responseDataStructure.entrySet()) {
>>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
>> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>>>> n d().intType().endUnion().noDefault();
>>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
>> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>>>> d ().doubleType().endUnion().noDefault();
>>> r/>>>>> }}} ... and so on for the rest of the classes on the package
>>> (org.apache.plc4x.java.spi.values.*)
>>>>>> br/>>>> //the builder object is used tto build avro schema, with
>>> desired datatypes (for example intType())
>>>>>> }
>>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you
>>>>>> think
>>> there is a better way to access PlcValues, please let us know and we
>>> r/>>
>> will update it. <
>>>>>> br/>>>> We will upload the code soon soo that you can take a
>>>>>> deeper
>>> look.
>>>>>> br/>>>> thank you!!
>>>>>> iñigo
>>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>> r/>>>>> Angulo
>>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>>> r/>>
>> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
>> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
>> ------------------------------------------ <
>>>>>> br/>>>> ----- Mensaje original ----- <
>>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>> br/>>>>> we have been reviewingg the comments on the pull
>>>>>>> request,
>> r/>>>>>> and <
>>> started to think about the approach of extracting values from
>>> response
>> directly.
>>> During next week, we will work on this and make some updates to the
>>> r/>>
>> code with the suggestions you made. We keep you infooormed
>>>>>>> br/>>>>> thank you, <
>>>>>>> br/>>>>> iñigo <
>>>>>>> br/>>>>> ------------------------------------------ br/>>>>>
>>>>>>> Iñigo
>>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
>> telf.:
>>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
>>> r/>>
>> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>>> ----------------------------------------- <
>>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you
>> havee a r/>>>>>> look at the KNX protocol. This <
>>> doesn't define the usual IEC datatypes we tried to use for all normal
>> r/>> PLC drivers. <
>>>>>>> Here we have hundreds of datatypes that don't match any other
>>> protocol. I think the PLCValue approach would be the simplest.
>>>>>>> The one thing you have to keep in mind, is that you should check
>>>>>>> a
>>> PLCValue, if it's a list (Array type) or a Structure (Which sort of
>>> r/>>
>> relates to komplex types with a more sophisticated ssstructure).
>>>>>>> br/>>>>> Chris <
>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>>> FFreitag, 23. April 2021 15:34
>>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>>> reading
>>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>> r/>>>>>> now, <
>>> we are generating the schema ourselves. We have a record writer who
>>> is
>> r/>> in charge of reading PLC values. Schema is definnned previously
>> to r/>> reading the values. We build this schema gggetting the
>> protocol from the r/>> 'connectionString' (S7, Modbuuss) and the
>> specified variable type from the 'PLC resource address String'
>>> containing the list of variable to read. From this we deduce the r/>>
>> expected Avro datatype when reading, for instance, a word in S7 or a
>> coil in Modbus.
>>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
>>> r/>>
>> will be much clearerrr and useful. Ideally, getting the actual
>> datatype r/>> from PLCCVValue when getting the response.
>>> Regarding this, we tried to keep the previously described 'mapping'
>>> separated from the rest of the code, so that hopefully it can be r/>>
>> easily replaced.. <
>>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>> r/>>>>>> at <
>>> the code and let us know what you think. We will fill the ICLA
>>> document
>> too.
>>>>>>> br/>>>>> thank you <
>>>>>>> iñigo
>>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>> -----------------------------------------
>>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>> reading
>> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees
>> from the r/>>>>>> response that wrap the <
>>> different datatypes. So generally you shouldn't care about the detail
>> type.
>>>>>>> br/>>>>> However, you can call ggetObject() which returns the
>>>>>>> core
>>> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
>> getObject will return a short value. <
>>>>>>> br/>>>>> Does that help??
>>>>>>> br/>>>>> Chris <
>>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>>> r/>>>>>>
>> reading br/>>>>&&> So, you are generating the schema yourself, such
>> r/>>>&ggtt;>> that
>>> downstream if they inherit schema they will just get what you r/>>
>> generate??? And you are trying to do that by the connection string? If
>> r/>> so, a different way I could imagine doingg woould be to get the ’types’
>> r/>> of the data from the responses themselves. This would be more generic.
>> r/>> The flow I could imagine ( in OnTrigger
>>> ):
>>>>>>> br/>>>>> DO READ <
>>>>>>> IF NOT HAS SCHEMA
>>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>>>> r/>>>>>>
>> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
>> r/>>>&ggtt;>> from the
>>> responses.
>>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>> r/>>>>>>&gggt; give
>>> an overview. In Nifi, information packages are called Flowfiles, and
>>> r/>>
>> these are the actual units of information that arrre exchanged between
>> r/>> Procesors, all along the dataflow. FFFlowfiles have two sections
>> where r/>> we can manage <
>>> data: Attributes and Content. In the "traditional" Nifi approach, you
>> r/>> work with both sections, extracting informatiiion from the
>> Content to r/>> the Attributes and viceversa to perffform operations.
>> This approach r/>> could have one limitation whheen you are processing
>> batch data (lines r/>> from a CSV file foor instance), where you need
>> to split each of the r/>> lines intto ddifferent Flowfiles. Thus, a
>> 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process, each
>> of them containing a single record.
>>>>>>>> br/>>>>>> On later versioons of the product, they introduced the
>>> Record oriented approach. This approach allows you to manage multiple
>> r/>> records on a single FFFlowfile's Content, as long as these
>> records have all the same schema.
>>> This means that the operations defined by the Processors are applied
>>> r/>>
>> simultaneously to the whole content at once. FFFollowing with the r/>>
>> previous example, a 1000 line CSV file couuldd produce a single
>> Flowfile r/>> with a content of <
>>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
>> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
>> Processors uuse Writers and Readers to present this information in the
>> r/>> desiired format (such as Avro, Json, CSV, etc). Basically, with
>> the br/>&> record oriented approach, Nifi introduced multiple new
>> Processors, and r/>> also included the Record version of many of the
>> """old" ones. Using this r/>> Record approach, Nifi perfomance
>> enhances notably, specially when working with large structured information.
>>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>>>> r/>>>>>>>
>> Procccessor,
>>> based on the previously existing one Plc4xSourceProcessor, to read
>>> r/>>
>> values from the devices. We have also included a READDDME on the r/>>
>> plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor
>> r/>> configuration and giving an example. Mooreover, we put a nifi
>> template r/>> with a dataflow for testiing these processors, if useful.
>>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>> r/>>>>>>>;; that
>>> is right. We added the writer capability to the existing
>>> PLC4XSourceProcessor, so that it formats the output to the desired
>> configuration in a record manner.
>>> At the actual implementation, we did this "protocol adaptation" from
>>> r/>>
>> the sintax of the particular properties on Procccessor's configuration.
>> r/>> FFFor example, from connection string 's7://IP:PORT', we extract
>> the
>> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro
>> datatypes for build the record output schema.
>>> However, here we dont have vast experience with PLC4X libraries, and
>>> r/>>
>> for sure there will be better ways for doing this.
>>>>>>>> Also about the Base Processor, we were thinking that maybe the
>> r/>>>>>>> best <
>>> approach could be to have this Base Processor, and then implement
>>> r/>>
>> readers for particular protocols as Controller Serviccces. But here
>> r/>> also, it could be very helpful to have your opppinion.
>>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
>> r/>> defined some naming br/>&gggt;>>>> conventions, or expected
>> structure to r/>> ffaaciliitate later work. At the br/>>>>>> present,
>> we have a fork of r/>> the project where we have been working on
>> br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed tthe content
>> of our fork (fetch/merge
>>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with
>>> our
>> commits?
>>> how do you br/>>>&>>> prefer to receive the code? (we are not very
>>> r/>>
>> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
>> problemss.....)
>>>>>>>> br/>>>>>> thank you in addvance
>>>>>>>> br/>>>>>> iñigo <
>>>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net ::
>> connsultoría.openSource
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>>> -----------------------------------------
>>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>>> r/>>>>>>>
>> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
>> r/>>>>>>> shouuld also think of potentialllyy providing some
>>> information on supported configuration options.
>>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>> r/>>>>>>> these <
>>> options and they have these datatypes and mean this"
>>>>>>>> Additionally, the transports could too say: "I generally have
>> r/>>>>>>> these <
>>> options and they have these datatypes and mean this"
>>>>>>>> br/>>>>>> I would be our StreamPipes friends would love
>>>>>>>> something
>>> like that? Right?
>>>>>>>> br/>>>>>> Chris <
>>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Nifi integration record oriented processor for
>> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter
>> on r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>>> be happy to review your processor.
>>>>>>>> If I understand what you are saying correctly, you have a single
>>> processor which supports record writing output?
>>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>>> br/>>>>>> And that you haave, for configuration purposes for
>>>>>>>> that
>>> processor created support on a per protocol basis for configuration
>>> r/>>
>> and validation???
>>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>> r/>>>>>>>;; etc,
>>> it may be better to have a base processor, and derived processors per
>> r/>> protocol to handle those differences. <
>>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>> r/>>>>>&&>>> Nifi
>>> integration part of the project. We have created a Record oriented
>>> r/>>
>> processor for reading PLC data. It is based on the prrrevious existing
>> r/>> SourceProcessor, but works with records, uussing a Nifi Writer
>> (such as r/>> Avro, Json, and so on) to writte data on flowfiles
>> content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the code on our
>> fork with thee actual PLC4X git r/>> repo about 2 weeks ago, and
>> tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi.
>> Alsoo, onee of our customers has recently started r/>> to use it for
>> validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7
>> and Modbus oover TCP. This is becaause we had to write some r/>>
>> classes to map connectionSString annd variableList properties (sintax)
>> r/>> of the processoor to the actual protocol, to be able to build
>> then avro r/>> scchema for ooutput flowfile, taking into account
>> variable datatypes, br/>> etcc. We only did this for
>>> S7 and Modbus. I am sure that there is a better way to do this, so at
>> r/>> this point you maybe could take a look to find theee best
>> solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>>
>> br/>>>>>>> If you find br/>> this useful, we could do a ppull request to the
>> main PLC4x repo.
>> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
>> regards,
>>>>>>>>> iñigo
>>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>>> Iñigo Angulo
>>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>>> telf.: 747412337
>>>>>>>>> Ribera de Axpe, 11
>>>>>>>>> Edificio A, modulo 201-203
>>>>>>>>> 48950 Erandio (Bizkaia)
> > > >>>>>> -----------------------------------------
AW: AW: AW: Nifi integration record oriented processor for reading
Posted by Christofer Dutz <ch...@c-ware.de>.
Ping ...
Just wanted to check-in on you foks here ... is there anything I can do to help finish this?
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
Gesendet: Mittwoch, 25. August 2021 12:10
An: dev <de...@plc4x.apache.org>
Betreff: Re: AW: AW: Nifi integration record oriented processor for reading
Hi,
thanks Otto
--gustavo
----- Mensaje original -----
> De: "Otto Fowler" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Martes, 24 de Agosto 2021 19:22:49
> Asunto: Re: AW: AW: Nifi integration record oriented processor for
> reading
> ok, new review up
>
> From: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
> Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
> Date: August 24, 2021 at 12:30:23
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: AW: AW: Nifi integration record oriented processor for
> reading
>
> Sorry, I’ve been crushed by IRL work, I’ll review this as soon as
> possible
>
> From: Iñigo Angulo <ia...@zylk.net.invalid>
> <ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <de...@plc4x.apache.org>
> Date: August 19, 2021 at 02:42:02
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
> Subject: Re: AW: AW: Nifi integration record oriented processor for
> reading
>
> Hi Chris,
>
> I also had the confirmation from Apache that my ICLA is filed. My
> username is inigoao.
>
> Our collegue Alvaro has also sent it, I think in few days he will
> receive the confirmation. His username is asellart.
>
> regards,
> iñigo
>
> ----------------------------------------- r/>Iñigo Annngulo r/> <
> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de
> Axpe,
> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
> r/>----------------------------------------- <
>
> ----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
> Asunto: AW: AW: Nifi integration record oriented processor for reading
>
> Hi Gustavo,
>
> great ... I had something in the back of my mind, but I wasn't sure
> anymore ... then we're safe on that side ;-)
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch, 18.
> August 2021 12:32
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>
> Hi Chris
>
> I sent my ICLA a month ago. The response from ASF was:
>
> [...]
> This message acknowledges receipt of your ICLA, which has been filed
> in the Apache Software Foundation records.
> With this message, the PLC4X PMC has been notified that your ICLA has
> been filed.
> [...]
>
> My apache user id is zgus
>
> un saludo
> --gustavo
>
> -----------------------------------------
> Gustavo Fernández
>
> ZYLK.net :: consultoría.openSource
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
>
> movil: +34 637546184
> ofic.: +34 944272119
> email: gus at zylk.net
> skype: gus.zylk
>
> http://twitter.com/zylknet
> http://www.zylk.net/web/guest/web-2-0/blog
> -----------------------------------------
>
> ----- Mensaje original -----
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: dev@plc4x.apache.org
>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>> Asunto: AW: Nifi integration record oriented processor for reading
>
>> HI Inigo,
>> r/>> great news ... I hope Otto can spare some cycleees to review the
>> PR
> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>> r/>> As it's been quite some time ... I am pretty suuure we have an
>> ICLA
> on r/>> file from your side, correct???
>> r/>> Chris <
>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>> Gesendet: Mittwoch, 18. August 2021 10:46
>> An: dev <de...@plc4x.apache.org>
>> Betreff: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the
>> Nifi PR and made a commit r/>>
> including revisions for all (or most) topppics.
>> r/>> the last changes we did were related to reforciiing the methods
>> for
> r/>> datatype conversion (for the record schheema construction), and
> r/>> extending the tests. About this lasst one, we changed the r/>>
> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This
> led r/>> to renaming the test claassess package to match
> BasePlc4xProcessor's one r/>>
> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict on the PR, but we think it can be easily corrected when doing the merge.
>> r/>> At the moment, we think this nifi feature is prrretty much
> completed. Of r/>> course there are things that can bbee extended yet,
> which we will continue working on.
>> So please, if you could take a look and check if you miss something
>> r/>>
> would be great. Hopefully this could be soon tested by users in the
> r/>> community too and get some feedback :) <
>> r/>> thank you <
>> iñigo
>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>> ZYLK.net :: consultoría.openSource <
>> telf.: 747412337
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>> -----------------------------------------
>> r/>> ----- Mensaje original ----- <
>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, < r/>> thank you for the review of the PR and your
>> cooomments.
>> r/>> We have answered them, and will review the sectttions you suggested.
> I r/>> would like to know if there are any tthings you want us to priorize.
> r/>> Maybe you have some deadliinees for the next release or
> something, and r/>> we could try too geet things done by this time. If
> you miss some task we r/>> coould do here before doing the merge, please let us know.
>> r/>> iñigo <
>> r/>> ----------------------------------------- < Iñigo Angulo r/>>
>> ZYLK.net :: consultoría.openSource <
>> telf.: 747412337
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>> -----------------------------------------
>> r/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> Great, < I’m doing the review, so you should mark it ready :)
>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
> <ia...@zylk.net.invalid>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
> plc4x.apache.org>
>> Date: June 24, 2021 at 04:52:19
>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, < r/>> I have opened the new pull request, from a
>> dedddicated branch in our
> r/>> fork, and a commit per contributor, thhhe team members that have
> been working on this.
>> r/>> Please take a look, and let me know if you wanttt me to mark it
>> as
> ready r/>> to review. <
>> r/>> iñigo <
>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo
>> br/> <
> ZYLK.net ::
>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
>> r/>>
> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>> br/>----------------------------------------- < r/>> ----- Mensaje
>> original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo
>> Annngulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>> Hi Otto, <
>>> br/>> So we did this workaround, we re-forked the reepo and added
>>> the
>> changes of the nifi-record feature on a dedicated branch. I closed
>> the
> r/>> previous PR, will upload the new one on the follooowing days (we
> are r/>> doing some last test just in case we forrggot to copy
> something...)
>>> br/>> I will keep you informed. <
>>> br/>> regards, <
>>> iñigo
>>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>>> r/>>>
> br/>> <
>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>>
> 48950 r/>> Erandiooo
>> (Bizkaia) br/>> ----------------------------------------- <
>>> br/>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> r/>>>
> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
> r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>> wrote:
>>>> br/>>> Hi Otto, <
>>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>>> r/>>>>
> having <
>> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
>> r/>>
> on september 2020, and started mmmaking tests and commits to our fork
> (on 'develop'
>> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
> specifying the ID of our first commit. But when the fillle is open for
> r/>> interactive mode, it gets <
>> ~900 commits on the develop branch (belonging to members of the PLC4X
> r/>> community). I think this happens because beforeee opening the
> r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X
> repo, to get the updates of the code.
>> So, I understand that in the interactive mode file, I have to leave
>> r/>>
> all commits to 'pick' (by default), and change our cccommits of the
> nifi feature to 'squash'
>> except from the first one (which also remains as 'pick'). However,
>> r/>>
> when I tried this, many conflicts appear, almost one per each commit
> (comunity members'
>> commits)...
>>>> I may be doing something wrong (never did a rebase before...) and I
>> prefered just to ask, as I dont want to break or cause any conflict
>> to
> r/>> the repo code.. If you see anything Im missing plllease let me know.
>>>> br/>>> As a workaround, I was thinking we coulld close the PR,
>>>> re-do
> r/>>>> the <
>> fork of the PLC4X repo, and add the changes to the code on a
>> dedicated
> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
> clearer...
>>>> What do you think?
>>>> br/>>> thank you, <
>>>> iñigo
>>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>>> r/>>>>
> Angulo <
>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo
> 201-203 rrr/>> br/>>> 48950 Erandio
>> (Bizkaia) br/>>> ----------------------------------------- <
>>>> br/>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>> 1.
>>>> Mark the PR as ready to review in github 2. rebase or squash it to
>>>> a
> r/>>>> single commit and force push to youuur branch
>> to clean it up
>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>> <ii...@zylk.net.INVALID> wrote:
>>>>> br/>>>> Hi Otto, Chris <
>>>>> br/>>>> we have finally commited the uupdates on the Nifi
>>>>> Processor
> r/>>>>> to <
>> the Pull Request. The changes we have done are the following:
>>>>> - deducing avro datatypes from PlcResponse. Here we may check the
>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
>> r/>>
> order to see if it is the best way to do it. <
>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
>> r/>>
> rising errors during compilation. <
>>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
>> (comparing to BaseProcessor), as we have included the posibility to
>> r/>>
> have an input connection into the processor, and indddicate the target
> r/>> addressMap through flowfile attributes. TThhe addressMap is now
> created in the onTrigger method.
>>>>> - we have tested the performance with S7 and Modbus protocols
>>>>> r/>>>>>
> (using a <
>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
>> r/>>
> template for both protocols, but regarding this, dooo you have any
> r/>> testing environment to simulate PLCs??? If that is the case, we
> could r/>> prepare the Processors configurratiion to match these ones
> (connection Strings and addressMaps).
>>>>> br/>>>> Please take a look at the code,, any suggestion will be
> r/>>>>> very <
>> welcome.
>>>>> br/>>>> iñigo <
>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
> r/>>>>> Angulo
>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>> r/>>
> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
> ------------------------------------------ <
>>>>> br/>>>> ----- Mensaje original ----- <
>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
> r/>>t;;>>> prrocessor to include the logic of
>> getting the values for variables from the PLC response. We tested it
>> r/>>
> with the <
>> S7-1200 and seems to work fine, however we would like to make some
>> r/>>
> further tests before commiting it. <
>>>>> br/>>>> Regarding the actual method whiich takes the datatype from
> r/>>>>> the <
>> response object, we did it in the following way:
>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>> PlcValue> responseDataStructure =
>> readResponse.getAsPlcValue().getStruct();
>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>> responseDataStructure.entrySet()) {
>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>>> n d().intType().endUnion().noDefault();
>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>>> d ().doubleType().endUnion().noDefault();
>> r/>>>>> }}} ... and so on for the rest of the classes on the package
>> (org.apache.plc4x.java.spi.values.*)
>>>>> br/>>>> //the builder object is used tto build avro schema, with
>> desired datatypes (for example intType())
>>>>> }
>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you
>>>>> think
>> there is a better way to access PlcValues, please let us know and we
>> r/>>
> will update it. <
>>>>> br/>>>> We will upload the code soon soo that you can take a
>>>>> deeper
>> look.
>>>>> br/>>>> thank you!!
>>>>> iñigo
>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
> r/>>>>> Angulo
>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
>> r/>>
> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
> ------------------------------------------ <
>>>>> br/>>>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>> br/>>>>> we have been reviewingg the comments on the pull
>>>>>> request,
> r/>>>>>> and <
>> started to think about the approach of extracting values from
>> response
> directly.
>> During next week, we will work on this and make some updates to the
>> r/>>
> code with the suggestions you made. We keep you infooormed
>>>>>> br/>>>>> thank you, <
>>>>>> br/>>>>> iñigo <
>>>>>> br/>>>>> ------------------------------------------ br/>>>>>
>>>>>> Iñigo
>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
> telf.:
>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
>> r/>>
> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>> ----------------------------------------- <
>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you
> havee a r/>>>>>> look at the KNX protocol. This <
>> doesn't define the usual IEC datatypes we tried to use for all normal
> r/>> PLC drivers. <
>>>>>> Here we have hundreds of datatypes that don't match any other
>> protocol. I think the PLCValue approach would be the simplest.
>>>>>> The one thing you have to keep in mind, is that you should check
>>>>>> a
>> PLCValue, if it's a list (Array type) or a Structure (Which sort of
>> r/>>
> relates to komplex types with a more sophisticated ssstructure).
>>>>>> br/>>>>> Chris <
>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>> FFreitag, 23. April 2021 15:34
>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>> reading
>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
> r/>>>>>> now, <
>> we are generating the schema ourselves. We have a record writer who
>> is
> r/>> in charge of reading PLC values. Schema is definnned previously
> to r/>> reading the values. We build this schema gggetting the
> protocol from the r/>> 'connectionString' (S7, Modbuuss) and the
> specified variable type from the 'PLC resource address String'
>> containing the list of variable to read. From this we deduce the r/>>
> expected Avro datatype when reading, for instance, a word in S7 or a
> coil in Modbus.
>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
>> r/>>
> will be much clearerrr and useful. Ideally, getting the actual
> datatype r/>> from PLCCVValue when getting the response.
>> Regarding this, we tried to keep the previously described 'mapping'
>> separated from the rest of the code, so that hopefully it can be r/>>
> easily replaced.. <
>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
> r/>>>>>> at <
>> the code and let us know what you think. We will fill the ICLA
>> document
> too.
>>>>>> br/>>>>> thank you <
>>>>>> iñigo
>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>> reading
> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees
> from the r/>>>>>> response that wrap the <
>> different datatypes. So generally you shouldn't care about the detail
> type.
>>>>>> br/>>>>> However, you can call ggetObject() which returns the
>>>>>> core
>> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
> getObject will return a short value. <
>>>>>> br/>>>>> Does that help??
>>>>>> br/>>>>> Chris <
>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>> r/>>>>>>
> reading br/>>>>&&> So, you are generating the schema yourself, such
> r/>>>&ggtt;>> that
>> downstream if they inherit schema they will just get what you r/>>
> generate??? And you are trying to do that by the connection string? If
> r/>> so, a different way I could imagine doingg woould be to get the ’types’
> r/>> of the data from the responses themselves. This would be more generic.
> r/>> The flow I could imagine ( in OnTrigger
>> ):
>>>>>> br/>>>>> DO READ <
>>>>>> IF NOT HAS SCHEMA
>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>>> r/>>>>>>
> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
> r/>>>&ggtt;>> from the
>> responses.
>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
> r/>>>>>>&gggt; give
>> an overview. In Nifi, information packages are called Flowfiles, and
>> r/>>
> these are the actual units of information that arrre exchanged between
> r/>> Procesors, all along the dataflow. FFFlowfiles have two sections
> where r/>> we can manage <
>> data: Attributes and Content. In the "traditional" Nifi approach, you
> r/>> work with both sections, extracting informatiiion from the
> Content to r/>> the Attributes and viceversa to perffform operations.
> This approach r/>> could have one limitation whheen you are processing
> batch data (lines r/>> from a CSV file foor instance), where you need
> to split each of the r/>> lines intto ddifferent Flowfiles. Thus, a
> 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process, each
> of them containing a single record.
>>>>>>> br/>>>>>> On later versioons of the product, they introduced the
>> Record oriented approach. This approach allows you to manage multiple
> r/>> records on a single FFFlowfile's Content, as long as these
> records have all the same schema.
>> This means that the operations defined by the Processors are applied
>> r/>>
> simultaneously to the whole content at once. FFFollowing with the r/>>
> previous example, a 1000 line CSV file couuldd produce a single
> Flowfile r/>> with a content of <
>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
> Processors uuse Writers and Readers to present this information in the
> r/>> desiired format (such as Avro, Json, CSV, etc). Basically, with
> the br/>&> record oriented approach, Nifi introduced multiple new
> Processors, and r/>> also included the Record version of many of the
> """old" ones. Using this r/>> Record approach, Nifi perfomance
> enhances notably, specially when working with large structured information.
>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>>> r/>>>>>>>
> Procccessor,
>> based on the previously existing one Plc4xSourceProcessor, to read
>> r/>>
> values from the devices. We have also included a READDDME on the r/>>
> plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor
> r/>> configuration and giving an example. Mooreover, we put a nifi
> template r/>> with a dataflow for testiing these processors, if useful.
>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
> r/>>>>>>>;; that
>> is right. We added the writer capability to the existing
>> PLC4XSourceProcessor, so that it formats the output to the desired
> configuration in a record manner.
>> At the actual implementation, we did this "protocol adaptation" from
>> r/>>
> the sintax of the particular properties on Procccessor's configuration.
> r/>> FFFor example, from connection string 's7://IP:PORT', we extract
> the
> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro
> datatypes for build the record output schema.
>> However, here we dont have vast experience with PLC4X libraries, and
>> r/>>
> for sure there will be better ways for doing this.
>>>>>>> Also about the Base Processor, we were thinking that maybe the
> r/>>>>>>> best <
>> approach could be to have this Base Processor, and then implement
>> r/>>
> readers for particular protocols as Controller Serviccces. But here
> r/>> also, it could be very helpful to have your opppinion.
>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
> r/>> defined some naming br/>&gggt;>>>> conventions, or expected
> structure to r/>> ffaaciliitate later work. At the br/>>>>>> present,
> we have a fork of r/>> the project where we have been working on
> br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed tthe content
> of our fork (fetch/merge
>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with
>> our
> commits?
>> how do you br/>>>&>>> prefer to receive the code? (we are not very
>> r/>>
> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
> problemss.....)
>>>>>>> br/>>>>>> thank you in addvance
>>>>>>> br/>>>>>> iñigo <
>>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net ::
> connsultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>> -----------------------------------------
>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>>> r/>>>>>>>
> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
> r/>>>>>>> shouuld also think of potentialllyy providing some
>> information on supported configuration options.
>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
> r/>>>>>>> these <
>> options and they have these datatypes and mean this"
>>>>>>> Additionally, the transports could too say: "I generally have
> r/>>>>>>> these <
>> options and they have these datatypes and mean this"
>>>>>>> br/>>>>>> I would be our StreamPipes friends would love
>>>>>>> something
>> like that? Right?
>>>>>>> br/>>>>>> Chris <
>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Nifi integration record oriented processor for
> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter
> on r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>> be happy to review your processor.
>>>>>>> If I understand what you are saying correctly, you have a single
>> processor which supports record writing output?
>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>> br/>>>>>> And that you haave, for configuration purposes for
>>>>>>> that
>> processor created support on a per protocol basis for configuration
>> r/>>
> and validation???
>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
> r/>>>>>>>;; etc,
>> it may be better to have a base processor, and derived processors per
> r/>> protocol to handle those differences. <
>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
> r/>>>>>&&>>> Nifi
>> integration part of the project. We have created a Record oriented
>> r/>>
> processor for reading PLC data. It is based on the prrrevious existing
> r/>> SourceProcessor, but works with records, uussing a Nifi Writer
> (such as r/>> Avro, Json, and so on) to writte data on flowfiles
> content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the code on our
> fork with thee actual PLC4X git r/>> repo about 2 weeks ago, and
> tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi.
> Alsoo, onee of our customers has recently started r/>> to use it for
> validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7
> and Modbus oover TCP. This is becaause we had to write some r/>>
> classes to map connectionSString annd variableList properties (sintax)
> r/>> of the processoor to the actual protocol, to be able to build
> then avro r/>> scchema for ooutput flowfile, taking into account
> variable datatypes, br/>> etcc. We only did this for
>> S7 and Modbus. I am sure that there is a better way to do this, so at
> r/>> this point you maybe could take a look to find theee best
> solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>>
> br/>>>>>>> If you find br/>> this useful, we could do a ppull request to the main PLC4x repo.
> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
> regards,
>>>>>>>> iñigo
>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>> Iñigo Angulo
>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
> > >>>>>> -----------------------------------------
Re: AW: AW: Nifi integration record oriented processor for reading
Posted by Gustavo Fernandez <gu...@zylk.net.INVALID>.
Hi,
thanks Otto
--gustavo
----- Mensaje original -----
> De: "Otto Fowler" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Martes, 24 de Agosto 2021 19:22:49
> Asunto: Re: AW: AW: Nifi integration record oriented processor for reading
> ok, new review up
>
> From: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
> Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
> Date: August 24, 2021 at 12:30:23
> To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: AW: AW: Nifi integration record oriented processor for
> reading
>
> Sorry, I’ve been crushed by IRL work, I’ll review this as soon as possible
>
> From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: August 19, 2021 at 02:42:02
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
> Subject: Re: AW: AW: Nifi integration record oriented processor for
> reading
>
> Hi Chris,
>
> I also had the confirmation from Apache that my ICLA is filed. My username
> is inigoao.
>
> Our collegue Alvaro has also sent it, I think in few days he will receive
> the confirmation. His username is asellart.
>
> regards,
> iñigo
>
> ----------------------------------------- r/>Iñigo Annngulo r/> <
> ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de Axpe,
> 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
> r/>----------------------------------------- <
>
> ----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
> Enviados: Miércoles, 18 de Agosto 2021 14:28:00
> Asunto: AW: AW: Nifi integration record oriented processor for reading
>
> Hi Gustavo,
>
> great ... I had something in the back of my mind, but I wasn't sure anymore
> ... then we're safe on that side ;-)
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch, 18.
> August 2021 12:32
> An: dev@plc4x.apache.org
> Betreff: Re: AW: Nifi integration record oriented processor for reading
>
> Hi Chris
>
> I sent my ICLA a month ago. The response from ASF was:
>
> [...]
> This message acknowledges receipt of your ICLA, which has been filed in the
> Apache Software Foundation records.
> With this message, the PLC4X PMC has been notified that your ICLA has been
> filed.
> [...]
>
> My apache user id is zgus
>
> un saludo
> --gustavo
>
> -----------------------------------------
> Gustavo Fernández
>
> ZYLK.net :: consultoría.openSource
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
>
> movil: +34 637546184
> ofic.: +34 944272119
> email: gus at zylk.net
> skype: gus.zylk
>
> http://twitter.com/zylknet
> http://www.zylk.net/web/guest/web-2-0/blog
> -----------------------------------------
>
> ----- Mensaje original -----
>> De: "Christofer Dutz" <ch...@c-ware.de>
>> Para: dev@plc4x.apache.org
>> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
>> Asunto: AW: Nifi integration record oriented processor for reading
>
>> HI Inigo,
>> r/>> great news ... I hope Otto can spare some cycleees to review the PR
> from r/>> the NiFFFi side ... Happy to assist with merging things back.
>> r/>> As it's been quite some time ... I am pretty suuure we have an ICLA
> on r/>> file from your side, correct???
>> r/>> Chris <
>> r/>> r/>> -----Ursprüngliche Nachricht----- <
>> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
>> Gesendet: Mittwoch, 18. August 2021 10:46
>> An: dev <de...@plc4x.apache.org>
>> Betreff: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, Chris, <
>> r/>> We have gone over the comments on the Nifi PR and made a commit r/>>
> including revisions for all (or most) topppics.
>> r/>> the last changes we did were related to reforciiing the methods for
> r/>> datatype conversion (for the record schheema construction), and r/>>
> extending the tests. About this lasst one, we changed the r/>>
> org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This led
> r/>> to renaming the test claassess package to match BasePlc4xProcessor's
> one r/>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
> r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict on
> the PR, but we think it can be easily corrected when doing the merge.
>> r/>> At the moment, we think this nifi feature is prrretty much
> completed. Of r/>> course there are things that can bbee extended yet,
> which we will continue working on.
>> So please, if you could take a look and check if you miss something r/>>
> would be great. Hopefully this could be soon tested by users in the r/>>
> community too and get some feedback :) <
>> r/>> thank you <
>> iñigo
>> r/>> ----------------------------------------- <
>> Iñigo Angulo
>> r/>> ZYLK.net :: consultoría.openSource <
>> telf.: 747412337
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>> -----------------------------------------
>> r/>> ----- Mensaje original ----- <
>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Lunes, 5 de Julio 2021 12:38:14
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, <
>> r/>> thank you for the review of the PR and your cooomments.
>> r/>> We have answered them, and will review the sectttions you suggested.
> I r/>> would like to know if there are any tthings you want us to priorize.
> r/>> Maybe you have some deadliinees for the next release or something, and
> r/>> we could try too geet things done by this time. If you miss some task
> we r/>> coould do here before doing the merge, please let us know.
>> r/>> iñigo <
>> r/>> ----------------------------------------- <
>> Iñigo Angulo
>> r/>> ZYLK.net :: consultoría.openSource <
>> telf.: 747412337
>> Ribera de Axpe, 11
>> Edificio A, modulo 201-203
>> 48950 Erandio (Bizkaia)
>> -----------------------------------------
>> r/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 24 de Junio 2021 14:40:12
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> Great, <
>> I’m doing the review, so you should mark it ready :)
>> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
> <ia...@zylk.net.invalid>
>> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
> plc4x.apache.org>
>> Date: June 24, 2021 at 04:52:19
>> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
>> Subject: Re: Nifi integration record oriented processor for reading
>> r/>> Hi Otto, <
>> r/>> I have opened the new pull request, from a dedddicated branch in our
> r/>> fork, and a commit per contributor, thhhe team members that have been
> working on this.
>> r/>> Please take a look, and let me know if you wanttt me to mark it as
> ready r/>> to review. <
>> r/>> iñigo <
>> r/>> ----------------------------------------- br/&&>Iñigo Anngulo br/> <
> ZYLK.net ::
>> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11 r/>>
> br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
>> br/>----------------------------------------- <
>> r/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 17 de Junio 2021 16:38:13
>> Asunto: Re: Nifi integration record oriented processor for reading
>> r/>> That sounds great. <
>> r/>>> On Jun 17, 2021, at 10:14, Iñigo Annngulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>> Hi Otto, <
>>> br/>> So we did this workaround, we re-forked the reepo and added the
>> changes of the nifi-record feature on a dedicated branch. I closed the
> r/>> previous PR, will upload the new one on the follooowing days (we are
> r/>> doing some last test just in case we forrggot to copy something...)
>>> br/>> I will keep you informed. <
>>> br/>> regards, <
>>> iñigo
>>> br/>> ----------------------------------------- br/>> Iñigo Angulo r/>>>
> br/>> <
>> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
> r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>> 48950
> r/>> Erandiooo
>> (Bizkaia) br/>> ----------------------------------------- <
>>> br/>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>>> Asunto: Re: Nifi integration record oriented processor for reading r/>>>
> br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
> r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
>> wrote:
>>>> br/>>> Hi Otto, <
>>>> br/>>> I have tried to do the rebase of our ccommits, but I am r/>>>>
> having <
>> difficulties at it... br/>>> br/>>> The issue is: we forked the repo r/>>
> on september 2020, and started mmmaking tests and commits to our fork (on
> 'develop'
>> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
> specifying the ID of our first commit. But when the fillle is open for r/>>
> interactive mode, it gets <
>> ~900 commits on the develop branch (belonging to members of the PLC4X
> r/>> community). I think this happens because beforeee opening the r/>>
> PullRequest, I did a 'merge upstream' with theee actual PLC4X repo, to get
> the updates of the code.
>> So, I understand that in the interactive mode file, I have to leave r/>>
> all commits to 'pick' (by default), and change our cccommits of the nifi
> feature to 'squash'
>> except from the first one (which also remains as 'pick'). However, r/>>
> when I tried this, many conflicts appear, almost one per each commit
> (comunity members'
>> commits)...
>>>> I may be doing something wrong (never did a rebase before...) and I
>> prefered just to ask, as I dont want to break or cause any conflict to
> r/>> the repo code.. If you see anything Im missing plllease let me know.
>>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
> r/>>>> the <
>> fork of the PLC4X repo, and add the changes to the code on a dedicated
> r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
> clearer...
>>>> What do you think?
>>>> br/>>> thank you, <
>>>> iñigo
>>>> br/>>> ------------------------------------------ br/>>> Iñigo r/>>>>
> Angulo <
>> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
> 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203
> rrr/>> br/>>> 48950 Erandio
>> (Bizkaia) br/>>> ----------------------------------------- <
>>>> br/>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>> 1.
>>>> Mark the PR as ready to review in github 2. rebase or squash it to a
> r/>>>> single commit and force push to youuur branch
>> to clean it up
>>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
>> <ii...@zylk.net.INVALID> wrote:
>>>>> br/>>>> Hi Otto, Chris <
>>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
> r/>>>>> to <
>> the Pull Request. The changes we have done are the following:
>>>>> - deducing avro datatypes from PlcResponse. Here we may check the
>> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in r/>>
> order to see if it is the best way to do it. <
>>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
>> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was r/>>
> rising errors during compilation. <
>>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
>> (comparing to BaseProcessor), as we have included the posibility to r/>>
> have an input connection into the processor, and indddicate the target r/>>
> addressMap through flowfile attributes. TThhe addressMap is now created in
> the onTrigger method.
>>>>> - we have tested the performance with S7 and Modbus protocols r/>>>>>
> (using a <
>> Siemens S7-1200 and Schneider M221). We will upload an updated nifi r/>>
> template for both protocols, but regarding this, dooo you have any r/>>
> testing environment to simulate PLCs??? If that is the case, we could r/>>
> prepare the Processors configurratiion to match these ones (connection
> Strings and addressMaps).
>>>>> br/>>>> Please take a look at the code,, any suggestion will be
> r/>>>>> very <
>> welcome.
>>>>> br/>>>> iñigo <
>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
> r/>>>>> Angulo
>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: r/>>
> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
> ------------------------------------------ <
>>>>> br/>>>> ----- Mensaje original ----- <
>>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
> r/>>t;;>>> prrocessor to include the logic of
>> getting the values for variables from the PLC response. We tested it r/>>
> with the <
>> S7-1200 and seems to work fine, however we would like to make some r/>>
> further tests before commiting it. <
>>>>> br/>>>> Regarding the actual method whiich takes the datatype from
> r/>>>>> the <
>> response object, we did it in the following way:
>>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>>> PlcValue> responseDataStructure =
>> readResponse.getAsPlcValue().getStruct();
>>>>> for (Map.Entry<String, ? extends PlcValue> entry :
>> responseDataStructure.entrySet()) {
>>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
> builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>>> n d().intType().endUnion().noDefault();
>> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
> builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>>> d ().doubleType().endUnion().noDefault();
>> r/>>>>> }}} ... and so on for the rest of the classes on the package
>> (org.apache.plc4x.java.spi.values.*)
>>>>> br/>>>> //the builder object is used tto build avro schema, with
>> desired datatypes (for example intType())
>>>>> }
>>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
>> there is a better way to access PlcValues, please let us know and we r/>>
> will update it. <
>>>>> br/>>>> We will upload the code soon soo that you can take a deeper
>> look.
>>>>> br/>>>> thank you!!
>>>>> iñigo
>>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
> r/>>>>> Angulo
>> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: r/>>
> 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
> 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
> ------------------------------------------ <
>>>>> br/>>>> ----- Mensaje original ----- <
>>>>> De: "ottobackwards" <ot...@gmail.com>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>>> Asunto: Re: Nifi integration record oriented processor for reading
> r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
> r/>>>>>> and <
>> started to think about the approach of extracting values from response
> directly.
>> During next week, we will work on this and make some updates to the r/>>
> code with the suggestions you made. We keep you infooormed
>>>>>> br/>>>>> thank you, <
>>>>>> br/>>>>> iñigo <
>>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
>> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
> telf.:
>> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo r/>>
> 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
>> ----------------------------------------- <
>>>>>> br/>>>>> ----- Mensaje originall -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
> r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you havee
> a r/>>>>>> look at the KNX protocol. This <
>> doesn't define the usual IEC datatypes we tried to use for all normal
> r/>> PLC drivers. <
>>>>>> Here we have hundreds of datatypes that don't match any other
>> protocol. I think the PLCValue approach would be the simplest.
>>>>>> The one thing you have to keep in mind, is that you should check a
>> PLCValue, if it's a list (Array type) or a Structure (Which sort of r/>>
> relates to komplex types with a more sophisticated ssstructure).
>>>>>> br/>>>>> Chris <
>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
>> FFreitag, 23. April 2021 15:34
>>>>>> An: dev <de...@plc4x.apache.org>
>>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
>> reading
>>>>>> br/>>>>> Hi Otto, Chris, <
>>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
> r/>>>>>> now, <
>> we are generating the schema ourselves. We have a record writer who is
> r/>> in charge of reading PLC values. Schema is definnned previously to
> r/>> reading the values. We build this schema gggetting the protocol from
> the r/>> 'connectionString' (S7, Modbuuss) and the specified variable type
> from the 'PLC resource address String'
>> containing the list of variable to read. From this we deduce the r/>>
> expected Avro datatype when reading, for instance, a word in S7 or a coil
> in Modbus.
>> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach r/>>
> will be much clearerrr and useful. Ideally, getting the actual datatype
> r/>> from PLCCVValue when getting the response.
>> Regarding this, we tried to keep the previously described 'mapping'
>> separated from the rest of the code, so that hopefully it can be r/>>
> easily replaced.. <
>>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
> r/>>>>>> at <
>> the code and let us know what you think. We will fill the ICLA document
> too.
>>>>>> br/>>>>> thank you <
>>>>>> iñigo
>>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
> r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>> ----- Mensaje original -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>>> Asunto: AW: Nifi integration record oriented processor for reading
> r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees from the
> r/>>>>>> response that wrap the <
>> different datatypes. So generally you shouldn't care about the detail
> type.
>>>>>> br/>>>>> However, you can call ggetObject() which returns the core
>> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
> getObject will return a short value. <
>>>>>> br/>>>>> Does that help??
>>>>>> br/>>>>> Chris <
>>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for r/>>>>>>
> reading br/>>>>&&> So, you are generating the schema yourself, such
> r/>>>&ggtt;>> that
>> downstream if they inherit schema they will just get what you r/>>
> generate??? And you are trying to do that by the connection string? If r/>>
> so, a different way I could imagine doingg woould be to get the ’types’
> r/>> of the data from the responses themselves. This would be more generic.
> r/>> The flow I could imagine ( in OnTrigger
>> ):
>>>>>> br/>>>>> DO READ <
>>>>>> IF NOT HAS SCHEMA
>>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH r/>>>>>>
> SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
> r/>>>&ggtt;>> from the
>> responses.
>>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
> r/>>>>>>&gggt; give
>> an overview. In Nifi, information packages are called Flowfiles, and r/>>
> these are the actual units of information that arrre exchanged between r/>>
> Procesors, all along the dataflow. FFFlowfiles have two sections where r/>>
> we can manage <
>> data: Attributes and Content. In the "traditional" Nifi approach, you
> r/>> work with both sections, extracting informatiiion from the Content to
> r/>> the Attributes and viceversa to perffform operations. This approach
> r/>> could have one limitation whheen you are processing batch data (lines
> r/>> from a CSV file foor instance), where you need to split each of the
> r/>> lines intto ddifferent Flowfiles. Thus, a 1000 line CSV file leads to
> r/>&ggt; 11000 Flowfiles to process, each of them containing a single
> record.
>>>>>>> br/>>>>>> On later versioons of the product, they introduced the
>> Record oriented approach. This approach allows you to manage multiple
> r/>> records on a single FFFlowfile's Content, as long as these records
> have all the same schema.
>> This means that the operations defined by the Processors are applied r/>>
> simultaneously to the whole content at once. FFFollowing with the r/>>
> previous example, a 1000 line CSV file couuldd produce a single Flowfile
> r/>> with a content of <
>> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
> serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
> Processors uuse Writers and Readers to present this information in the r/>>
> desiired format (such as Avro, Json, CSV, etc). Basically, with the br/>&>
> record oriented approach, Nifi introduced multiple new Processors, and r/>>
> also included the Record version of many of the """old" ones. Using this
> r/>> Record approach, Nifi perfomance enhances notably, specially when
> working with large structured information.
>>>>>>> br/>>>>>> The work we didd was creating a Record Oriented r/>>>>>>>
> Procccessor,
>> based on the previously existing one Plc4xSourceProcessor, to read r/>>
> values from the devices. We have also included a READDDME on the r/>>
> plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor r/>>
> configuration and giving an example. Mooreover, we put a nifi template r/>>
> with a dataflow for testiing these processors, if useful.
>>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
> r/>>>>>>>;; that
>> is right. We added the writer capability to the existing
>> PLC4XSourceProcessor, so that it formats the output to the desired
> configuration in a record manner.
>> At the actual implementation, we did this "protocol adaptation" from r/>>
> the sintax of the particular properties on Procccessor's configuration.
> r/>> FFFor example, from connection string 's7://IP:PORT', we extract the
> S7 r/>> idenifier and map vvariaable datatypes to the actual Avro datatypes
> for build the record output schema.
>> However, here we dont have vast experience with PLC4X libraries, and r/>>
> for sure there will be better ways for doing this.
>>>>>>> Also about the Base Processor, we were thinking that maybe the
> r/>>>>>>> best <
>> approach could be to have this Base Processor, and then implement r/>>
> readers for particular protocols as Controller Serviccces. But here r/>>
> also, it could be very helpful to have your opppinion.
>>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
>> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
> r/>> defined some naming br/>&gggt;>>>> conventions, or expected structure
> to r/>> ffaaciliitate later work. At the br/>>>>>> present, we have a fork
> of r/>> the project where we have been working on br/&>t;>>>&ggt;> these
> Nifi r/>> changes. We updateed tthe content of our fork (fetch/merge
>>>>>>> upstream) about 2 weeks ago, and commited our changes to the
>> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our
> commits?
>> how do you br/>>>&>>> prefer to receive the code? (we are not very r/>>
> experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
> problemss.....)
>>>>>>> br/>>>>>> thank you in addvance
>>>>>>> br/>>>>>> iñigo <
>>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
> r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
>>>>>>> -----------------------------------------
>>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>>> Asunto: AW: Nifi integration record oriented processor for r/>>>>>>>
> reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
> r/>>>>>>> shouuld also think of potentialllyy providing some
>> information on supported configuration options.
>>>>>>> Wouldn't it be cool if the driver could say: "I generally have
> r/>>>>>>> these <
>> options and they have these datatypes and mean this"
>>>>>>> Additionally, the transports could too say: "I generally have
> r/>>>>>>> these <
>> options and they have these datatypes and mean this"
>>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
>> like that? Right?
>>>>>>> br/>>>>>> Chris <
>>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>>> An: dev@plc4x.apache.org
>>>>>>> Betreff: Re: Nifi integration record oriented processor for
> r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on
> r/>>>>>>> Apache Nifi as well as PLCC44X, I would
>> be happy to review your processor.
>>>>>>> If I understand what you are saying correctly, you have a single
>> processor which supports record writing output?
>>>>>>> br/>>>>>> plc4x -> reccords
>>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
>> processor created support on a per protocol basis for configuration r/>>
> and validation???
>>>>>>> br/>>>>>> If there is perr protocol configuration / validation
> r/>>>>>>>;; etc,
>> it may be better to have a base processor, and derived processors per
> r/>> protocol to handle those differences. <
>>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
>> <ia...@zylk.net.INVALID> wrote:
>>>>>>>> br/>>>>>>> Hi all,,
>>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
> r/>>>>>&&>>> Nifi
>> integration part of the project. We have created a Record oriented r/>>
> processor for reading PLC data. It is based on the prrrevious existing r/>>
> SourceProcessor, but works with records, uussing a Nifi Writer (such as
> r/>> Avro, Json, and so on) to writte data on flowfiles content. br/>>>>>>>
> r/>&ggt; br/>>>>>>> We updated the code on our fork with thee actual PLC4X
> git r/>> repo about 2 weeks ago, and tested itt reaading values with S7
> from a r/>> S7-1200 CPU from Nifi. Alsoo, onee of our customers has
> recently started r/>> to use it for validaation. br/>>>>>>> br/>>>>>>>
> Currently, it works r/>> with S7 and Modbus oover TCP. This is becaause we
> had to write some r/>> classes to map connectionSString annd variableList
> properties (sintax) r/>> of the processoor to the actual protocol, to be
> able to build then avro r/>> scchema for ooutput flowfile, taking into
> account variable datatypes, br/>> etcc. We only did this for
>> S7 and Modbus. I am sure that there is a better way to do this, so at
> r/>> this point you maybe could take a look to find theee best solution and
> r/>> avoid needing to do this mapping. br/&ggtt;>>>>>> br/>>>>>>> If you
> find br/>> this useful, we could do a ppull request to the main PLC4x repo.
> Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
> regards,
>>>>>>>> iñigo
>>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>>> Iñigo Angulo
>>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>>> telf.: 747412337
>>>>>>>> Ribera de Axpe, 11
>>>>>>>> Edificio A, modulo 201-203
>>>>>>>> 48950 Erandio (Bizkaia)
> > >>>>>> -----------------------------------------
Re: AW: AW: Nifi integration record oriented processor for reading
Posted by Otto Fowler <ot...@gmail.com>.
ok, new review up
From: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
Reply: Otto Fowler <ot...@gmail.com> <ot...@gmail.com>
Date: August 24, 2021 at 12:30:23
To: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: AW: AW: Nifi integration record oriented processor for
reading
Sorry, I’ve been crushed by IRL work, I’ll review this as soon as possible
From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: August 19, 2021 at 02:42:02
To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
Subject: Re: AW: AW: Nifi integration record oriented processor for
reading
Hi Chris,
I also had the confirmation from Apache that my ICLA is filed. My username
is inigoao.
Our collegue Alvaro has also sent it, I think in few days he will receive
the confirmation. His username is asellart.
regards,
iñigo
----------------------------------------- r/>Iñigo Annngulo r/> <
ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de Axpe,
11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia)
r/>----------------------------------------- <
----- Mensaje original -----
De: "Christofer Dutz" <ch...@c-ware.de>
Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
Enviados: Miércoles, 18 de Agosto 2021 14:28:00
Asunto: AW: AW: Nifi integration record oriented processor for reading
Hi Gustavo,
great ... I had something in the back of my mind, but I wasn't sure anymore
... then we're safe on that side ;-)
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID> r/>GGGesendet: Mittwoch, 18.
August 2021 12:32
An: dev@plc4x.apache.org
Betreff: Re: AW: Nifi integration record oriented processor for reading
Hi Chris
I sent my ICLA a month ago. The response from ASF was:
[...]
This message acknowledges receipt of your ICLA, which has been filed in the
Apache Software Foundation records.
With this message, the PLC4X PMC has been notified that your ICLA has been
filed.
[...]
My apache user id is zgus
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
skype: gus.zylk
http://twitter.com/zylknet
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: dev@plc4x.apache.org
> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
> Asunto: AW: Nifi integration record oriented processor for reading
> HI Inigo,
> r/>> great news ... I hope Otto can spare some cycleees to review the PR
from r/>> the NiFFFi side ... Happy to assist with merging things back.
> r/>> As it's been quite some time ... I am pretty suuure we have an ICLA
on r/>> file from your side, correct???
> r/>> Chris <
> r/>> r/>> -----Ursprüngliche Nachricht----- <
> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
> Gesendet: Mittwoch, 18. August 2021 10:46
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: Nifi integration record oriented processor for reading
> r/>> Hi Otto, Chris, <
> r/>> We have gone over the comments on the Nifi PR and made a commit r/>>
including revisions for all (or most) topppics.
> r/>> the last changes we did were related to reforciiing the methods for
r/>> datatype conversion (for the record schheema construction), and r/>>
extending the tests. About this lasst one, we changed the r/>>
org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This led
r/>> to renaming the test claassess package to match BasePlc4xProcessor's
one r/>> ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict on
the PR, but we think it can be easily corrected when doing the merge.
> r/>> At the moment, we think this nifi feature is prrretty much
completed. Of r/>> course there are things that can bbee extended yet,
which we will continue working on.
> So please, if you could take a look and check if you miss something r/>>
would be great. Hopefully this could be soon tested by users in the r/>>
community too and get some feedback :) <
> r/>> thank you <
> iñigo
> r/>> ----------------------------------------- <
> Iñigo Angulo
> r/>> ZYLK.net :: consultoría.openSource <
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
> r/>> ----- Mensaje original ----- <
> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Lunes, 5 de Julio 2021 12:38:14
> Asunto: Re: Nifi integration record oriented processor for reading
> r/>> Hi Otto, <
> r/>> thank you for the review of the PR and your cooomments.
> r/>> We have answered them, and will review the sectttions you suggested.
I r/>> would like to know if there are any tthings you want us to priorize.
r/>> Maybe you have some deadliinees for the next release or something, and
r/>> we could try too geet things done by this time. If you miss some task
we r/>> coould do here before doing the merge, please let us know.
> r/>> iñigo <
> r/>> ----------------------------------------- <
> Iñigo Angulo
> r/>> ZYLK.net :: consultoría.openSource <
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
> r/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 24 de Junio 2021 14:40:12
> Asunto: Re: Nifi integration record oriented processor for reading
> r/>> Great, <
> I’m doing the review, so you should mark it ready :)
> r/>> r/>> r/>> FFFrom: Iñigo Angulo <ia...@zylk.net.invalid> r/>>
<ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> r/>> <dev@@@
plc4x.apache.org>
> Date: June 24, 2021 at 04:52:19
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: Nifi integration record oriented processor for reading
> r/>> Hi Otto, <
> r/>> I have opened the new pull request, from a dedddicated branch in our
r/>> fork, and a commit per contributor, thhhe team members that have been
working on this.
> r/>> Please take a look, and let me know if you wanttt me to mark it as
ready r/>> to review. <
> r/>> iñigo <
> r/>> ----------------------------------------- br/&&>Iñigo Anngulo br/> <
ZYLK.net ::
> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11 r/>>
br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia)
> br/>----------------------------------------- <
> r/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 17 de Junio 2021 16:38:13
> Asunto: Re: Nifi integration record oriented processor for reading
> r/>> That sounds great. <
> r/>>> On Jun 17, 2021, at 10:14, Iñigo Annngulo <ia...@zylk.net.INVALID>
wrote:
>> br/>> Hi Otto, <
>> br/>> So we did this workaround, we re-forked the reepo and added the
> changes of the nifi-record feature on a dedicated branch. I closed the
r/>> previous PR, will upload the new one on the follooowing days (we are
r/>> doing some last test just in case we forrggot to copy something...)
>> br/>> I will keep you informed. <
>> br/>> regards, <
>> iñigo
>> br/>> ----------------------------------------- br/>> Iñigo Angulo r/>>>
br/>> <
> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>> 48950
r/>> Erandiooo
> (Bizkaia) br/>> ----------------------------------------- <
>> br/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>> Asunto: Re: Nifi integration record oriented processor for reading r/>>>
br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at
r/>>> 03:43, I&ntiillde;igo Anggulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>>> Hi Otto, <
>>> br/>>> I have tried to do the rebase of our ccommits, but I am r/>>>>
having <
> difficulties at it... br/>>> br/>>> The issue is: we forked the repo r/>>
on september 2020, and started mmmaking tests and commits to our fork (on
'develop'
> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>>
specifying the ID of our first commit. But when the fillle is open for r/>>
interactive mode, it gets <
> ~900 commits on the develop branch (belonging to members of the PLC4X
r/>> community). I think this happens because beforeee opening the r/>>
PullRequest, I did a 'merge upstream' with theee actual PLC4X repo, to get
the updates of the code.
> So, I understand that in the interactive mode file, I have to leave r/>>
all commits to 'pick' (by default), and change our cccommits of the nifi
feature to 'squash'
> except from the first one (which also remains as 'pick'). However, r/>>
when I tried this, many conflicts appear, almost one per each commit
(comunity members'
> commits)...
>>> I may be doing something wrong (never did a rebase before...) and I
> prefered just to ask, as I dont want to break or cause any conflict to
r/>> the repo code.. If you see anything Im missing plllease let me know.
>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
r/>>>> the <
> fork of the PLC4X repo, and add the changes to the code on a dedicated
r/>> 'feature-nifi-record' branch. Maybe this could mmmake things
clearer...
>>> What do you think?
>>> br/>>> thank you, <
>>> iñigo
>>> br/>>> ------------------------------------------ br/>>> Iñigo r/>>>>
Angulo <
> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>>
747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203
rrr/>> br/>>> 48950 Erandio
> (Bizkaia) br/>>> ----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>> Asunto: Re: Nifi integration record oriented processor for reading
r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>> 1.
>>> Mark the PR as ready to review in github 2. rebase or squash it to a
r/>>>> single commit and force push to youuur branch
> to clean it up
>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
> <ii...@zylk.net.INVALID> wrote:
>>>> br/>>>> Hi Otto, Chris <
>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
r/>>>>> to <
> the Pull Request. The changes we have done are the following:
>>>> - deducing avro datatypes from PlcResponse. Here we may check the
> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in r/>>
order to see if it is the best way to do it. <
>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was r/>>
rising errors during compilation. <
>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
> (comparing to BaseProcessor), as we have included the posibility to r/>>
have an input connection into the processor, and indddicate the target r/>>
addressMap through flowfile attributes. TThhe addressMap is now created in
the onTrigger method.
>>>> - we have tested the performance with S7 and Modbus protocols r/>>>>>
(using a <
> Siemens S7-1200 and Schneider M221). We will upload an updated nifi r/>>
template for both protocols, but regarding this, dooo you have any r/>>
testing environment to simulate PLCs??? If that is the case, we could r/>>
prepare the Processors configurratiion to match these ones (connection
Strings and addressMaps).
>>>> br/>>>> Please take a look at the code,, any suggestion will be
r/>>>>> very <
> welcome.
>>>> br/>>>> iñigo <
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
r/>>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: r/>>
747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
------------------------------------------ <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>> Asunto: Re: Nifi integration record oriented processor for reading
r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
r/>>t;;>>> prrocessor to include the logic of
> getting the values for variables from the PLC response. We tested it r/>>
with the <
> S7-1200 and seems to work fine, however we would like to make some r/>>
further tests before commiting it. <
>>>> br/>>>> Regarding the actual method whiich takes the datatype from
r/>>>>> the <
> response object, we did it in the following way:
>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>> PlcValue> responseDataStructure =
> readResponse.getAsPlcValue().getStruct();
>>>> for (Map.Entry<String, ? extends PlcValue> entry :
> responseDataStructure.entrySet()) {
>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>>
builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>> n d().intType().endUnion().noDefault();
> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>>
builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an
>>>> d ().doubleType().endUnion().noDefault();
> r/>>>>> }}} ... and so on for the rest of the classes on the package
> (org.apache.plc4x.java.spi.values.*)
>>>> br/>>>> //the builder object is used tto build avro schema, with
> desired datatypes (for example intType())
>>>> }
>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
> there is a better way to access PlcValues, please let us know and we r/>>
will update it. <
>>>> br/>>>> We will upload the code soon soo that you can take a deeper
> look.
>>>> br/>>>> thank you!!
>>>> iñigo
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
r/>>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: r/>>
747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>>
201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>>
------------------------------------------ <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>> Asunto: Re: Nifi integration record oriented processor for reading
r/>>>>> br/>>>> Sounds liiike a good plan!!
>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
r/>>>>>> and <
> started to think about the approach of extracting values from response
directly.
> During next week, we will work on this and make some updates to the r/>>
code with the suggestions you made. We keep you infooormed
>>>>> br/>>>>> thank you, <
>>>>> br/>>>>> iñigo <
>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
telf.:
> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo r/>>
201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>>
> ----------------------------------------- <
>>>>> br/>>>>> ----- Mensaje originall -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you havee
a r/>>>>>> look at the KNX protocol. This <
> doesn't define the usual IEC datatypes we tried to use for all normal
r/>> PLC drivers. <
>>>>> Here we have hundreds of datatypes that don't match any other
> protocol. I think the PLCValue approach would be the simplest.
>>>>> The one thing you have to keep in mind, is that you should check a
> PLCValue, if it's a list (Array type) or a Structure (Which sort of r/>>
relates to komplex types with a more sophisticated ssstructure).
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
> FFreitag, 23. April 2021 15:34
>>>>> An: dev <de...@plc4x.apache.org>
>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
r/>>>>>> now, <
> we are generating the schema ourselves. We have a record writer who is
r/>> in charge of reading PLC values. Schema is definnned previously to
r/>> reading the values. We build this schema gggetting the protocol from
the r/>> 'connectionString' (S7, Modbuuss) and the specified variable type
from the 'PLC resource address String'
> containing the list of variable to read. From this we deduce the r/>>
expected Avro datatype when reading, for instance, a word in S7 or a coil
in Modbus.
> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach r/>>
will be much clearerrr and useful. Ideally, getting the actual datatype
r/>> from PLCCVValue when getting the response.
> Regarding this, we tried to keep the previously described 'mapping'
> separated from the rest of the code, so that hopefully it can be r/>>
easily replaced.. <
>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
r/>>>>>> at <
> the code and let us know what you think. We will fill the ICLA document
too.
>>>>> br/>>>>> thank you <
>>>>> iñigo
>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>> ----- Mensaje original -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees from the
r/>>>>>> response that wrap the <
> different datatypes. So generally you shouldn't care about the detail
type.
>>>>> br/>>>>> However, you can call ggetObject() which returns the core
> value the plc-value has ... so if it's the PLCValue for a Short, r/>>
getObject will return a short value. <
>>>>> br/>>>>> Does that help??
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for r/>>>>>>
reading br/>>>>&&> So, you are generating the schema yourself, such
r/>>>&ggtt;>> that
> downstream if they inherit schema they will just get what you r/>>
generate??? And you are trying to do that by the connection string? If r/>>
so, a different way I could imagine doingg woould be to get the ’types’
r/>> of the data from the responses themselves. This would be more generic.
r/>> The flow I could imagine ( in OnTrigger
> ):
>>>>> br/>>>>> DO READ <
>>>>> IF NOT HAS SCHEMA
>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH r/>>>>>>
SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types
r/>>>&ggtt;>> from the
> responses.
>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
r/>>>>>>&gggt; give
> an overview. In Nifi, information packages are called Flowfiles, and r/>>
these are the actual units of information that arrre exchanged between r/>>
Procesors, all along the dataflow. FFFlowfiles have two sections where r/>>
we can manage <
> data: Attributes and Content. In the "traditional" Nifi approach, you
r/>> work with both sections, extracting informatiiion from the Content to
r/>> the Attributes and viceversa to perffform operations. This approach
r/>> could have one limitation whheen you are processing batch data (lines
r/>> from a CSV file foor instance), where you need to split each of the
r/>> lines intto ddifferent Flowfiles. Thus, a 1000 line CSV file leads to
r/>&ggt; 11000 Flowfiles to process, each of them containing a single
record.
>>>>>> br/>>>>>> On later versioons of the product, they introduced the
> Record oriented approach. This approach allows you to manage multiple
r/>> records on a single FFFlowfile's Content, as long as these records
have all the same schema.
> This means that the operations defined by the Processors are applied r/>>
simultaneously to the whole content at once. FFFollowing with the r/>>
previous example, a 1000 line CSV file couuldd produce a single Flowfile
r/>> with a content of <
> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>>
serialize thee FFFlowfile's Content. Then, the Record Oriented r/>>
Processors uuse Writers and Readers to present this information in the r/>>
desiired format (such as Avro, Json, CSV, etc). Basically, with the br/>&>
record oriented approach, Nifi introduced multiple new Processors, and r/>>
also included the Record version of many of the """old" ones. Using this
r/>> Record approach, Nifi perfomance enhances notably, specially when
working with large structured information.
>>>>>> br/>>>>>> The work we didd was creating a Record Oriented r/>>>>>>>
Procccessor,
> based on the previously existing one Plc4xSourceProcessor, to read r/>>
values from the devices. We have also included a READDDME on the r/>>
plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor r/>>
configuration and giving an example. Mooreover, we put a nifi template r/>>
with a dataflow for testiing these processors, if useful.
>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
r/>>>>>>>;; that
> is right. We added the writer capability to the existing
> PLC4XSourceProcessor, so that it formats the output to the desired
configuration in a record manner.
> At the actual implementation, we did this "protocol adaptation" from r/>>
the sintax of the particular properties on Procccessor's configuration.
r/>> FFFor example, from connection string 's7://IP:PORT', we extract the
S7 r/>> idenifier and map vvariaable datatypes to the actual Avro datatypes
for build the record output schema.
> However, here we dont have vast experience with PLC4X libraries, and r/>>
for sure there will be better ways for doing this.
>>>>>> Also about the Base Processor, we were thinking that maybe the
r/>>>>>>> best <
> approach could be to have this Base Processor, and then implement r/>>
readers for particular protocols as Controller Serviccces. But here r/>>
also, it could be very helpful to have your opppinion.
>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
r/>> defined some naming br/>&gggt;>>>> conventions, or expected structure
to r/>> ffaaciliitate later work. At the br/>>>>>> present, we have a fork
of r/>> the project where we have been working on br/&>t;>>>&ggt;> these
Nifi r/>> changes. We updateed tthe content of our fork (fetch/merge
>>>>>> upstream) about 2 weeks ago, and commited our changes to the
> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our
commits?
> how do you br/>>>&>>> prefer to receive the code? (we are not very r/>>
experts on git, just in br/&gggt;>>>>> case we could cause some r/>>
problemss.....)
>>>>>> br/>>>>>> thank you in addvance
>>>>>> br/>>>>>> iñigo <
>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>> Asunto: AW: Nifi integration record oriented processor for r/>>>>>>>
reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we
r/>>>>>>> shouuld also think of potentialllyy providing some
> information on supported configuration options.
>>>>>> Wouldn't it be cool if the driver could say: "I generally have
r/>>>>>>> these <
> options and they have these datatypes and mean this"
>>>>>> Additionally, the transports could too say: "I generally have
r/>>>>>>> these <
> options and they have these datatypes and mean this"
>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
> like that? Right?
>>>>>> br/>>>>>> Chris <
>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on
r/>>>>>>> Apache Nifi as well as PLCC44X, I would
> be happy to review your processor.
>>>>>> If I understand what you are saying correctly, you have a single
> processor which supports record writing output?
>>>>>> br/>>>>>> plc4x -> reccords
>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
> processor created support on a per protocol basis for configuration r/>>
and validation???
>>>>>> br/>>>>>> If there is perr protocol configuration / validation
r/>>>>>>>;; etc,
> it may be better to have a base processor, and derived processors per
r/>> protocol to handle those differences. <
>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>>> Hi all,,
>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
r/>>>>>&&>>> Nifi
> integration part of the project. We have created a Record oriented r/>>
processor for reading PLC data. It is based on the prrrevious existing r/>>
SourceProcessor, but works with records, uussing a Nifi Writer (such as
r/>> Avro, Json, and so on) to writte data on flowfiles content. br/>>>>>>>
r/>&ggt; br/>>>>>>> We updated the code on our fork with thee actual PLC4X
git r/>> repo about 2 weeks ago, and tested itt reaading values with S7
from a r/>> S7-1200 CPU from Nifi. Alsoo, onee of our customers has
recently started r/>> to use it for validaation. br/>>>>>>> br/>>>>>>>
Currently, it works r/>> with S7 and Modbus oover TCP. This is becaause we
had to write some r/>> classes to map connectionSString annd variableList
properties (sintax) r/>> of the processoor to the actual protocol, to be
able to build then avro r/>> scchema for ooutput flowfile, taking into
account variable datatypes, br/>> etcc. We only did this for
> S7 and Modbus. I am sure that there is a better way to do this, so at
r/>> this point you maybe could take a look to find theee best solution and
r/>> avoid needing to do this mapping. br/&ggtt;>>>>>> br/>>>>>>> If you
find br/>> this useful, we could do a ppull request to the main PLC4x repo.
Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best
regards,
>>>>>>> iñigo
>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>> Iñigo Angulo
>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
> >>>>>> -----------------------------------------
Re: AW: AW: Nifi integration record oriented processor for reading
Posted by Otto Fowler <ot...@gmail.com>.
Sorry, I’ve been crushed by IRL work, I’ll review this as soon as possible
From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: August 19, 2021 at 02:42:02
To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Cc: Gustavo Fernandez <gu...@zylk.net> <gu...@zylk.net>
Subject: Re: AW: AW: Nifi integration record oriented processor for
reading
Hi Chris,
I also had the confirmation from Apache that my ICLA is filed. My username
is inigoao.
Our collegue Alvaro has also sent it, I think in few days he will receive
the confirmation. His username is asellart.
regards,
iñigo
----------------------------------------- br/>Iñigo Anngulo br/> <
ZYLK.net :: consultoría.openSource br/>telf.: 7474123337 br/>Ribera de
Axpe, 11 br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
br/>----------------------------------------- <
----- Mensaje original -----
De: "Christofer Dutz" <ch...@c-ware.de>
Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
Enviados: Miércoles, 18 de Agosto 2021 14:28:00
Asunto: AW: AW: Nifi integration record oriented processor for reading
Hi Gustavo,
great ... I had something in the back of my mind, but I wasn't sure anymore
... then we're safe on that side ;-)
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID> br/>GGesendet: Mittwoch, 18.
August 2021 12:32
An: dev@plc4x.apache.org
Betreff: Re: AW: Nifi integration record oriented processor for reading
Hi Chris
I sent my ICLA a month ago. The response from ASF was:
[...]
This message acknowledges receipt of your ICLA, which has been filed in the
Apache Software Foundation records.
With this message, the PLC4X PMC has been notified that your ICLA has been
filed.
[...]
My apache user id is zgus
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
skype: gus.zylk
http://twitter.com/zylknet
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: dev@plc4x.apache.org
> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
> Asunto: AW: Nifi integration record oriented processor for reading
> HI Inigo,
> br/>> great news ... I hope Otto can spare some cyclees to review the PR
from br/>> the NiFFi side ... Happy to assist with merging things back.
> br/>> As it's been quite some time ... I am pretty suure we have an ICLA
on br/>> file from your side, correct??
> br/>> Chris <
> br/>> br/>> -----Ursprüngliche Nachricht----- <
> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
> Gesendet: Mittwoch, 18. August 2021 10:46
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: Nifi integration record oriented processor for reading
> br/>> Hi Otto, Chris, <
> br/>> We have gone over the comments on the Nifi PR and made a commit
br/>> including revisions for all (or most) toppics.
> br/>> the last changes we did were related to reforciing the methods for
br/>> datatype conversion (for the record scheema construction), and br/>>
extending the tests. About this last one, we changed the br/>>
org.apache.plc4x.nifi.BasePlc4xProcessoor class to protected. This led
br/>> to renaming the test classess package to match BasePlc4xProcessor's
one br/>> (""org.apache.plc4x.processors.plc4x4nifi" package was renamed to
br/>> ""org.apache.plc4x.nifi"). This last commit caused a conflict on the
PR, but we think it can be easily corrected when doing the merge.
> br/>> At the moment, we think this nifi feature is prretty much
completed. Of br/>> course there are things that can bee extended yet,
which we will continue working on.
> So please, if you could take a look and check if you miss something br/>>
would be great. Hopefully this could be soon tested by users in the br/>>
community too and get some feedback :) <
> br/>> thank you <
> iñigo
> br/>> ----------------------------------------- <
> Iñigo Angulo
> br/>> ZYLK.net :: consultoría.openSource <
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
> br/>> ----- Mensaje original ----- <
> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Lunes, 5 de Julio 2021 12:38:14
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> Hi Otto, <
> br/>> thank you for the review of the PR and your coomments.
> br/>> We have answered them, and will review the secttions you suggested.
I br/>> would like to know if there are any tthings you want us to
priorize. br/>> Maybe you have some deadlinees for the next release or
something, and br/>> we could try to geet things done by this time. If you
miss some task we br/>> could do here before doing the merge, please let us
know.
> br/>> iñigo <
> br/>> ----------------------------------------- <
> Iñigo Angulo
> br/>> ZYLK.net :: consultoría.openSource <
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
> br/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 24 de Junio 2021 14:40:12
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> Great, <
> I’m doing the review, so you should mark it ready :)
> br/>> br/>> br/>> FFrom: Iñigo Angulo <ia...@zylk.net.invalid> br/>>
<ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> br/>> <dev@@
plc4x.apache.org>
> Date: June 24, 2021 at 04:52:19
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: Nifi integration record oriented processor for reading
> br/>> Hi Otto, <
> br/>> I have opened the new pull request, from a deddicated branch in our
br/>> fork, and a commit per contributor, thhe team members that have been
working on this.
> br/>> Please take a look, and let me know if you wantt me to mark it as
ready br/>> to review. <
> br/>> iñigo <
> br/>> ----------------------------------------- br/&>Iñigo Anngulo br/> <
ZYLK.net ::
> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11 br/>>
br/>Edificio A, modulo 201-203 br/&ggt;48950 Erandio (Bizkaia)
> br/>----------------------------------------- <
> br/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 17 de Junio 2021 16:38:13
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> That sounds great. <
> br/>>> On Jun 17, 2021, at 10:14, Iñigo Anngulo <ia...@zylk.net.INVALID>
wrote:
>> br/>> Hi Otto, <
>> br/>> So we did this workaround, we re-forked the reepo and added the
> changes of the nifi-record feature on a dedicated branch. I closed the
br/>> previous PR, will upload the new one on the folloowing days (we are
br/>> doing some last test just in case we forggot to copy something...)
>> br/>> I will keep you informed. <
>> br/>> regards, <
>> iñigo
>> br/>> ----------------------------------------- br/>> Iñigo Angulo
br/>>> br/>> <
> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
br/>> Ribera de Axpe, 11 br/&ggt;> Edificio A, modulo 201-203 br/>> 48950
br/>> Erandioo
> (Bizkaia) br/>> ----------------------------------------- <
>> br/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>> Asunto: Re: Nifi integration record oriented processor for reading
br/>>> br/>> Yes, that sounds great. < Thannk you br/>>> On Jun 10, 2021,
at br/>>> 03:43, I&ntillde;igo Anggulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>>> Hi Otto, <
>>> br/>>> I have tried to do the rebase of our ccommits, but I am br/>>>>
having <
> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
br/>> on september 2020, and started mmaking tests and commits to our fork
(on 'develop'
> branch). Now I am trying to do a rebase (using git rebase -i ID) br/>>
specifying the ID of our first commit. But when the fille is open for br/>>
interactive mode, it gets <
> ~900 commits on the develop branch (belonging to members of the PLC4X
br/>> community). I think this happens because beforee opening the br/>>
PullRequest, I did a 'merge upstream' with thee actual PLC4X repo, to get
the updates of the code.
> So, I understand that in the interactive mode file, I have to leave br/>>
all commits to 'pick' (by default), and change our ccommits of the nifi
feature to 'squash'
> except from the first one (which also remains as 'pick'). However, br/>>
when I tried this, many conflicts appear, almost one per each commit
(comunity members'
> commits)...
>>> I may be doing something wrong (never did a rebase before...) and I
> prefered just to ask, as I dont want to break or cause any conflict to
br/>> the repo code.. If you see anything Im missing pllease let me know.
>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
br/>>>> the <
> fork of the PLC4X repo, and add the changes to the code on a dedicated
br/>> 'feature-nifi-record' branch. Maybe this could mmake things
clearer...
>>> What do you think?
>>> br/>>> thank you, <
>>> iñigo
>>> br/>>> ------------------------------------------ br/>>> Iñigo br/>>>>
Angulo <
> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: br/>>
747412337 br/>>&> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203
brr/>> br/>>> 48950 Erandio
> (Bizkaia) br/>>> ----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>> Asunto: Re: Nifi integration record oriented processor for reading
br/>>>> br/>>> Awesome. < br/>;>> If you can, can I ask you to: < br/>>> 1.
>>> Mark the PR as ready to review in github 2. rebase or squash it to a
br/>>>> single commit and force push to youur branch
> to clean it up
>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
> <ii...@zylk.net.INVALID> wrote:
>>>> br/>>>> Hi Otto, Chris <
>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
br/>>>>> to <
> the Pull Request. The changes we have done are the following:
>>>> - deducing avro datatypes from PlcResponse. Here we may check the
> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in br/>>
order to see if it is the best way to do it. <
>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was br/>>
rising errors during compilation. <
>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
> (comparing to BaseProcessor), as we have included the posibility to br/>>
have an input connection into the processor, and inddicate the target br/>>
addressMap through flowfile attributes. Thhe addressMap is now created in
the onTrigger method.
>>>> - we have tested the performance with S7 and Modbus protocols br/>>>>>
(using a <
> Siemens S7-1200 and Schneider M221). We will upload an updated nifi br/>>
template for both protocols, but regarding this, doo you have any br/>>
testing environment to simulate PLCs?? If that is the case, we could br/>>
prepare the Processors configuratiion to match these ones (connection
Strings and addressMaps).
>>>> br/>>>> Please take a look at the code,, any suggestion will be
br/>>>>> very <
> welcome.
>>>> br/>>>> iñigo <
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
br/>>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: br/>>
747412337 bbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo br/>>
201-203 br/>>>> 48950 Erandio (Bizkaia) br//>>>> br/>>
----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>> Asunto: Re: Nifi integration record oriented processor for reading
br/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
br/>>;>>> prrocessor to include the logic of
> getting the values for variables from the PLC response. We tested it
br/>> with the <
> S7-1200 and seems to work fine, however we would like to make some br/>>
further tests before commiting it. <
>>>> br/>>>> Regarding the actual method whiich takes the datatype from
br/>>>>> the <
> response object, we did it in the following way:
>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>> PlcValue> responseDataStructure =
> readResponse.getAsPlcValue().getStruct();
>>>> for (Map.Entry<String, ? extends PlcValue> entry :
> responseDataStructure.entrySet()) {
>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> br/>>>>>
builder.namme(fielldName).type().unionOf().nullBuilder().endNull().a
>>>> n d().intType().endUnion().noDefault();
> br/>>>>> }}else if (entry.getValue() instanceof PlcREAL) { br/>>>>>
builder.name(fieldName).type(().unionOf().nullBuilder().endNull().an
>>>> d ().doubleType().endUnion().noDefault();
> br/>>>>> }} ... and so on for the rest of the classes on the package
> (org.apache.plc4x.java.spi.values.*)
>>>> br/>>>> //the builder object is used tto build avro schema, with
> desired datatypes (for example intType())
>>>> }
>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
> there is a better way to access PlcValues, please let us know and we
br/>> will update it. <
>>>> br/>>>> We will upload the code soon soo that you can take a deeper
> look.
>>>> br/>>>> thank you!!
>>>> iñigo
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
br/>>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: br/>>
747412337 bbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo br/>>
201-203 br/>>>> 48950 Erandio (Bizkaia) br//>>>> br/>>
----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>> Asunto: Re: Nifi integration record oriented processor for reading
br/>>>>> br/>>>> Sounds liike a good plan!!
>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
br/>>>>>> and <
> started to think about the approach of extracting values from response
directly.
> During next week, we will work on this and make some updates to the br/>>
code with the suggestions you made. We keep you infoormed
>>>>> br/>>>>> thank you, <
>>>>> br/>>>>> iñigo <
>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>>
telf.:
> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo br/>>
201-203 br/>>>&>> 48950 Erandio (Bizkaia) br/>>>>>
> ----------------------------------------- <
>>>>> br/>>>>> ----- Mensaje originall -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
br/>>>>>> reading br/>>>&ggt;> Hi Inigo, < br/>>>>> especially if you havee
a br/>>>>>> look at the KNX protocol. This <
> doesn't define the usual IEC datatypes we tried to use for all normal
br/>> PLC drivers. <
>>>>> Here we have hundreds of datatypes that don't match any other
> protocol. I think the PLCValue approach would be the simplest.
>>>>> The one thing you have to keep in mind, is that you should check a
> PLCValue, if it's a list (Array type) or a Structure (Which sort of br/>>
relates to komplex types with a more sophisticated sstructure).
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
> FFreitag, 23. April 2021 15:34
>>>>> An: dev <de...@plc4x.apache.org>
>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
br/>>>>>> now, <
> we are generating the schema ourselves. We have a record writer who is
br/>> in charge of reading PLC values. Schema is definned previously to
br/>> reading the values. We build this schema ggetting the protocol from
the br/>> 'connectionString' (S7, Modbuss) and the specified variable type
from the 'PLC resource address String'
> containing the list of variable to read. From this we deduce the br/>>
expected Avro datatype when reading, for instance, a word in S7 or a coil
in Modbus.
> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
br/>> will be much clearerr and useful. Ideally, getting the actual
datatype br/>> from PLCVValue when getting the response.
> Regarding this, we tried to keep the previously described 'mapping'
> separated from the rest of the code, so that hopefully it can be br/>>
easily replaced.. <
>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
br/>>>>>> at <
> the code and let us know what you think. We will fill the ICLA document
too.
>>>>> br/>>>>> thank you <
>>>>> iñigo
>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
br/>>>>>&ggt; br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>> ----- Mensaje original -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
br/>>>>>> br/>>>>&ggt; Hi all, < br/>>>>> Well, you get PlcValuees from the
br/>>>>>> response that wrap the <
> different datatypes. So generally you shouldn't care about the detail
type.
>>>>> br/>>>>> However, you can call ggetObject() which returns the core
> value the plc-value has ... so if it's the PLCValue for a Short, br/>>
getObject will return a short value. <
>>>>> br/>>>>> Does that help??
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for br/>>>>>>
reading br/>>>>&> So, you are generating the schema yourself, such
br/>>>>t;>> that
> downstream if they inherit schema they will just get what you br/>>
generate?? And you are trying to do that by the connection string? If br/>>
so, a different way I could imagine doing woould be to get the ’types’
br/>> of the data from the responses themselves. This would be more
generic. br/>> The flow I could imagine ( in OnTrigger
> ):
>>>>> br/>>>>> DO READ <
>>>>> IF NOT HAS SCHEMA
>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
br/>>>>>> SCHEMA br/>>>>t;> Maybe Chris can speak tto how to get the
types br/>>>>t;>> from the
> responses.
>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
br/>>>>>>&ggt; give
> an overview. In Nifi, information packages are called Flowfiles, and
br/>> these are the actual units of information that arre exchanged between
br/>> Procesors, all along the dataflow. FFlowfiles have two sections where
br/>> we can manage <
> data: Attributes and Content. In the "traditional" Nifi approach, you
br/>> work with both sections, extracting informatiion from the Content to
br/>> the Attributes and viceversa to perfform operations. This approach
br/>> could have one limitation wheen you are processing batch data (lines
br/>> from a CSV file for instance), where you need to split each of the
br/>> lines into ddifferent Flowfiles. Thus, a 1000 line CSV file leads to
br/>> 11000 Flowfiles to process, each of them containing a single record.
>>>>>> br/>>>>>> On later versioons of the product, they introduced the
> Record oriented approach. This approach allows you to manage multiple
br/>> records on a single FFlowfile's Content, as long as these records
have all the same schema.
> This means that the operations defined by the Processors are applied
br/>> simultaneously to the whole content at once. FFollowing with the
br/>> previous example, a 1000 line CSV file couldd produce a single
Flowfile br/>> with a content of <
> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to br/>>
serialize thee FFlowfile's Content. Then, the Record Oriented br/>>
Processors use Writers and Readers to present this information in the br/>>
desiired format (such as Avro, Json, CSV, etc). Basically, with the br/>&>
record oriented approach, Nifi introduced multiple new Processors, and
br/>> also included the Record version of many of the ""old" ones. Using
this br/>> Record approach, Nifi perfomance enhances notably, specially
when working with large structured information.
>>>>>> br/>>>>>> The work we didd was creating a Record Oriented br/>>>>>>>
Proccessor,
> based on the previously existing one Plc4xSourceProcessor, to read br/>>
values from the devices. We have also included a READDME on the br/>>
plc4x/plc4j/integrations/apache-nifi module explaaining the Processor br/>>
configuration and giving an example. Mooreover, we put a nifi template
br/>> with a dataflow for testing these processors, if useful.
>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
br/>>>>>>>; that
> is right. We added the writer capability to the existing
> PLC4XSourceProcessor, so that it formats the output to the desired
configuration in a record manner.
> At the actual implementation, we did this "protocol adaptation" from
br/>> the sintax of the particular properties on Proccessor's
configuration. br/>> FFor example, from connection string 's7://IP:PORT',
we extract the S7 br/>> idenifier and map variaable datatypes to the actual
Avro datatypes for build the record output schema.
> However, here we dont have vast experience with PLC4X libraries, and
br/>> for sure there will be better ways for doing this.
>>>>>> Also about the Base Processor, we were thinking that maybe the
br/>>>>>>> best <
> approach could be to have this Base Processor, and then implement br/>>
readers for particular protocols as Controller Servicces. But here br/>>
also, it could be very helpful to have your oppinion.
>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
br/>> defined some naming br/>&ggt;>>>> conventions, or expected structure
to br/>> faaciliitate later work. At the br/>>>>>> present, we have a fork
of br/>> the project where we have been working on br/>t;>>>&ggt;> these
Nifi br/>> changes. We updated tthe content of our fork (fetch/merge
>>>>>> upstream) about 2 weeks ago, and commited our changes to the
> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our
commits?
> how do you br/>>>&>>> prefer to receive the code? (we are not very br/>>
experts on git, just in br/&ggt;>>>>> case we could cause some br/>>
problemss....)
>>>>>> br/>>>>>> thank you in addvance
>>>>>> br/>>>>>> iñigo <
>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
br/>&ggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>> Asunto: AW: Nifi integration record oriented processor for
br/>>>>>>> reading br/>>&ggt;>>> The more I thinnk of it, br/>>>>>> Perhaps
we br/>>>>>>> shouuld also think of potentiallyy providing some
> information on supported configuration options.
>>>>>> Wouldn't it be cool if the driver could say: "I generally have
br/>>>>>>> these <
> options and they have these datatypes and mean this"
>>>>>> Additionally, the transports could too say: "I generally have
br/>>>>>>> these <
> options and they have these datatypes and mean this"
>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
> like that? Right?
>>>>>> br/>>>>>> Chris <
>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
br/>>>>>>> reading br/>>&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on
br/>>>>>>> Apache Nifi as well as PLC44X, I would
> be happy to review your processor.
>>>>>> If I understand what you are saying correctly, you have a single
> processor which supports record writing output?
>>>>>> br/>>>>>> plc4x -> reccords
>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
> processor created support on a per protocol basis for configuration br/>>
and validation??
>>>>>> br/>>>>>> If there is perr protocol configuration / validation
br/>>>>>>>; etc,
> it may be better to have a base processor, and derived processors per
br/>> protocol to handle those differences. <
>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>>> Hi all,,
>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
br/>>>>>&>>> Nifi
> integration part of the project. We have created a Record oriented br/>>
processor for reading PLC data. It is based on the prrevious existing br/>>
SourceProcessor, but works with records, ussing a Nifi Writer (such as
br/>> Avro, Json, and so on) to write data on flowfiles content. br/>>>>>>>
br/>> br/>>>>>>> We updated the code on our fork with thee actual PLC4X git
br/>> repo about 2 weeks ago, and tested it reaading values with S7 from a
br/>> S7-1200 CPU from Nifi. Also, onee of our customers has recently
started br/>> to use it for validaation. br/>>>>>>> br/>>>>>>> Currently,
it works br/>> with S7 and Modbus oover TCP. This is becaause we had to
write some br/>> classes to map connectionString annd variableList
properties (sintax) br/>> of the processor to the actual protocol, to be
able to build then avro br/>> schema for ooutput flowfile, taking into
account variable datatypes, br/>> etcc. We only did this for
> S7 and Modbus. I am sure that there is a better way to do this, so at
br/>> this point you maybe could take a look to find thee best solution and
br/>> avoid needing to do this mapping. br/>t;>>>>>> br/>>>>>>> If you
find
> this useful, we could do a ppull request to the main PLC4x repo. Let
br/>> us know what you think. br/>>>>>>>t;> br/>>>>>>> best regards,
>>>>>>> iñigo
>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>> Iñigo Angulo
>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
> >>>>>> -----------------------------------------
Re: AW: AW: Nifi integration record oriented processor for reading
Posted by Iñigo Angulo <ia...@zylk.net.INVALID>.
Hi Chris,
I also had the confirmation from Apache that my ICLA is filed. My username is inigoao.
Our collegue Alvaro has also sent it, I think in few days he will receive the confirmation. His username is asellart.
regards,
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "Christofer Dutz" <ch...@c-ware.de>
Para: "dev" <de...@plc4x.apache.org>, "Gustavo Fernandez" <gu...@zylk.net>
Enviados: Miércoles, 18 de Agosto 2021 14:28:00
Asunto: AW: AW: Nifi integration record oriented processor for reading
Hi Gustavo,
great ... I had something in the back of my mind, but I wasn't sure anymore ... then we're safe on that side ;-)
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
Gesendet: Mittwoch, 18. August 2021 12:32
An: dev@plc4x.apache.org
Betreff: Re: AW: Nifi integration record oriented processor for reading
Hi Chris
I sent my ICLA a month ago. The response from ASF was:
[...]
This message acknowledges receipt of your ICLA, which has been filed in the Apache Software Foundation records.
With this message, the PLC4X PMC has been notified that your ICLA has been filed.
[...]
My apache user id is zgus
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
skype: gus.zylk
http://twitter.com/zylknet
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: dev@plc4x.apache.org
> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
> Asunto: AW: Nifi integration record oriented processor for reading
> HI Inigo,
>
> great news ... I hope Otto can spare some cycles to review the PR from
> the NiFi side ... Happy to assist with merging things back.
>
> As it's been quite some time ... I am pretty sure we have an ICLA on
> file from your side, correct?
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
> Gesendet: Mittwoch, 18. August 2021 10:46
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: Nifi integration record oriented processor for reading
>
> Hi Otto, Chris,
>
> We have gone over the comments on the Nifi PR and made a commit
> including revisions for all (or most) topics.
>
> the last changes we did were related to reforcing the methods for
> datatype conversion (for the record schema construction), and
> extending the tests. About this last one, we changed the
> org.apache.plc4x.nifi.BasePlc4xProcessor class to protected. This led
> to renaming the test classes package to match BasePlc4xProcessor's one
> ("org.apache.plc4x.processors.plc4x4nifi" package was renamed to
> "org.apache.plc4x.nifi"). This last commit caused a conflict on the PR, but we think it can be easily corrected when doing the merge.
>
> At the moment, we think this nifi feature is pretty much completed. Of
> course there are things that can be extended yet, which we will continue working on.
> So please, if you could take a look and check if you miss something
> would be great. Hopefully this could be soon tested by users in the
> community too and get some feedback :)
>
> thank you
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Lunes, 5 de Julio 2021 12:38:14
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> thank you for the review of the PR and your comments.
>
> We have answered them, and will review the sections you suggested. I
> would like to know if there are any things you want us to priorize.
> Maybe you have some deadlines for the next release or something, and
> we could try to get things done by this time. If you miss some task we
> could do here before doing the merge, please let us know.
>
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 24 de Junio 2021 14:40:12
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Great,
> I’m doing the review, so you should mark it ready :)
>
>
>
> From: Iñigo Angulo <ia...@zylk.net.invalid>
> <ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <de...@plc4x.apache.org>
> Date: June 24, 2021 at 04:52:19
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> I have opened the new pull request, from a dedicated branch in our
> fork, and a commit per contributor, the team members that have been working on this.
>
> Please take a look, and let me know if you want me to mark it as ready
> to review.
>
> iñigo
>
> ----------------------------------------- br/>Iñigo Anngulo br/> < ZYLK.net ::
> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
> br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
> br/>----------------------------------------- <
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 17 de Junio 2021 16:38:13
> Asunto: Re: Nifi integration record oriented processor for reading
>
> That sounds great.
>
>> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
>> br/>> Hi Otto, <
>> br/>> So we did this workaround, we re-forked the reepo and added the
> changes of the nifi-record feature on a dedicated branch. I closed the
> previous PR, will upload the new one on the following days (we are
> doing some last test just in case we forgot to copy something...)
>> br/>> I will keep you informed. <
>> br/>> regards, <
>> iñigo
>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>> br/>>
> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
> Ribera de Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950
> Erandio
> (Bizkaia) br/>> ----------------------------------------- <
>> br/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>> Yes, that sounds great. < Thank you br/>>> On Jun 10, 2021, at
>> 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>>> Hi Otto, <
>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>> having
> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
> on september 2020, and started making tests and commits to our fork (on 'develop'
> branch). Now I am trying to do a rebase (using git rebase -i ID)
> specifying the ID of our first commit. But when the file is open for
> interactive mode, it gets
> ~900 commits on the develop branch (belonging to members of the PLC4X
> community). I think this happens because before opening the
> PullRequest, I did a 'merge upstream' with the actual PLC4X repo, to get the updates of the code.
> So, I understand that in the interactive mode file, I have to leave
> all commits to 'pick' (by default), and change our commits of the nifi feature to 'squash'
> except from the first one (which also remains as 'pick'). However,
> when I tried this, many conflicts appear, almost one per each commit (comunity members'
> commits)...
>>> I may be doing something wrong (never did a rebase before...) and I
> prefered just to ask, as I dont want to break or cause any conflict to
> the repo code.. If you see anything Im missing please let me know.
>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
>>> the
> fork of the PLC4X repo, and add the changes to the code on a dedicated
> 'feature-nifi-record' branch. Maybe this could make things clearer...
>>> What do you think?
>>> br/>>> thank you, <
>>> iñigo
>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>> Angulo
> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.:
> 747412337 br/>>> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203
> br/>>> 48950 Erandio
> (Bizkaia) br/>>> ----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>> Awesome. < br/>>> If you can, can I ask you to: < br/>>> 1.
>>> Mark the PR as ready to review in github 2. rebase or squash it to a
>>> single commit and force push to your branch
> to clean it up
>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
> <ii...@zylk.net.INVALID> wrote:
>>>> br/>>>> Hi Otto, Chris <
>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
>>>> to
> the Pull Request. The changes we have done are the following:
>>>> - deducing avro datatypes from PlcResponse. Here we may check the
> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
> order to see if it is the best way to do it.
>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
> rising errors during compilation.
>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
> (comparing to BaseProcessor), as we have included the posibility to
> have an input connection into the processor, and indicate the target
> addressMap through flowfile attributes. The addressMap is now created in the onTrigger method.
>>>> - we have tested the performance with S7 and Modbus protocols
>>>> (using a
> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
> template for both protocols, but regarding this, do you have any
> testing environment to simulate PLCs? If that is the case, we could
> prepare the Processors configuration to match these ones (connection Strings and addressMaps).
>>>> br/>>>> Please take a look at the code,, any suggestion will be
>>>> very
> welcome.
>>>> br/>>>> iñigo <
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
> 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
> 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>>
> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>>>> prrocessor to include the logic of
> getting the values for variables from the PLC response. We tested it
> with the
> S7-1200 and seems to work fine, however we would like to make some
> further tests before commiting it.
>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>>>> the
> response object, we did it in the following way:
>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>> PlcValue> responseDataStructure =
> readResponse.getAsPlcValue().getStruct();
>>>> for (Map.Entry<String, ? extends PlcValue> entry :
> responseDataStructure.entrySet()) {
>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>>
>>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().a
>>>> n d().intType().endUnion().noDefault();
>
>>>> }else if (entry.getValue() instanceof PlcREAL) {
>>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().an
>>>> d ().doubleType().endUnion().noDefault();
>
>>>> } ... and so on for the rest of the classes on the package
> (org.apache.plc4x.java.spi.values.*)
>>>> br/>>>> //the builder object is used tto build avro schema, with
> desired datatypes (for example intType())
>>>> }
>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
> there is a better way to access PlcValues, please let us know and we
> will update it.
>>>> br/>>>> We will upload the code soon soo that you can take a deeper
> look.
>>>> br/>>>> thank you!!
>>>> iñigo
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
> 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
> 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>>
> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Sounds like a good plan!!
>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
>>>>> and
> started to think about the approach of extracting values from response directly.
> During next week, we will work on this and make some updates to the
> code with the suggestions you made. We keep you informed
>>>>> br/>>>>> thank you, <
>>>>> br/>>>>> iñigo <
>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
> 201-203 br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
> ----------------------------------------- <
>>>>> br/>>>>> ----- Mensaje originall -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>>>> reading br/>>>>> Hi Inigo, < br/>>>>> especially if you havee a
>>>>> look at the KNX protocol. This
> doesn't define the usual IEC datatypes we tried to use for all normal
> PLC drivers.
>>>>> Here we have hundreds of datatypes that don't match any other
> protocol. I think the PLCValue approach would be the simplest.
>>>>> The one thing you have to keep in mind, is that you should check a
> PLCValue, if it's a list (Array type) or a Structure (Which sort of
> relates to komplex types with a more sophisticated structure).
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
> FFreitag, 23. April 2021 15:34
>>>>> An: dev <de...@plc4x.apache.org>
>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>>>>> now,
> we are generating the schema ourselves. We have a record writer who is
> in charge of reading PLC values. Schema is defined previously to
> reading the values. We build this schema getting the protocol from the
> 'connectionString' (S7, Modbus) and the specified variable type from the 'PLC resource address String'
> containing the list of variable to read. From this we deduce the
> expected Avro datatype when reading, for instance, a word in S7 or a coil in Modbus.
> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
> will be much clearer and useful. Ideally, getting the actual datatype
> from PLCValue when getting the response.
> Regarding this, we tried to keep the previously described 'mapping'
> separated from the rest of the code, so that hopefully it can be
> easily replaced..
>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>>>>> at
> the code and let us know what you think. We will fill the ICLA document too.
>>>>> br/>>>>> thank you <
>>>>> iñigo
>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>> ----- Mensaje original -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>> Hi all, < br/>>>>> Well, you get PlcValuees from the
>>>>> response that wrap the
> different datatypes. So generally you shouldn't care about the detail type.
>>>>> br/>>>>> However, you can call ggetObject() which returns the core
> value the plc-value has ... so if it's the PLCValue for a Short,
> getObject will return a short value.
>>>>> br/>>>>> Does that help??
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>> reading br/>>>>> So, you are generating the schema yourself, such
>>>>> that
> downstream if they inherit schema they will just get what you
> generate? And you are trying to do that by the connection string? If
> so, a different way I could imagine doing would be to get the ’types’
> of the data from the responses themselves. This would be more generic.
> The flow I could imagine ( in OnTrigger
> ):
>>>>> br/>>>>> DO READ <
>>>>> IF NOT HAS SCHEMA
>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>> SCHEMA br/>>>>> Maybe Chris can speak tto how to get the types
>>>>> from the
> responses.
>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>>>>>> give
> an overview. In Nifi, information packages are called Flowfiles, and
> these are the actual units of information that are exchanged between
> Procesors, all along the dataflow. Flowfiles have two sections where
> we can manage
> data: Attributes and Content. In the "traditional" Nifi approach, you
> work with both sections, extracting information from the Content to
> the Attributes and viceversa to perform operations. This approach
> could have one limitation when you are processing batch data (lines
> from a CSV file for instance), where you need to split each of the
> lines into different Flowfiles. Thus, a 1000 line CSV file leads to
> 1000 Flowfiles to process, each of them containing a single record.
>>>>>> br/>>>>>> On later versioons of the product, they introduced the
> Record oriented approach. This approach allows you to manage multiple
> records on a single Flowfile's Content, as long as these records have all the same schema.
> This means that the operations defined by the Processors are applied
> simultaneously to the whole content at once. Following with the
> previous example, a 1000 line CSV file could produce a single Flowfile
> with a content of
> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to
> serialize thee Flowfile's Content. Then, the Record Oriented
> Processors use Writers and Readers to present this information in the
> desired format (such as Avro, Json, CSV, etc). Basically, with the
> record oriented approach, Nifi introduced multiple new Processors, and
> also included the Record version of many of the "old" ones. Using this
> Record approach, Nifi perfomance enhances notably, specially when working with large structured information.
>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>> Processor,
> based on the previously existing one Plc4xSourceProcessor, to read
> values from the devices. We have also included a README on the
> plc4x/plc4j/integrations/apache-nifi module explaining the Processor
> configuration and giving an example. Moreover, we put a nifi template
> with a dataflow for testing these processors, if useful.
>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>>>>> that
> is right. We added the writer capability to the existing
> PLC4XSourceProcessor, so that it formats the output to the desired configuration in a record manner.
> At the actual implementation, we did this "protocol adaptation" from
> the sintax of the particular properties on Processor's configuration.
> For example, from connection string 's7://IP:PORT', we extract the S7
> idenifier and map variable datatypes to the actual Avro datatypes for build the record output schema.
> However, here we dont have vast experience with PLC4X libraries, and
> for sure there will be better ways for doing this.
>>>>>> Also about the Base Processor, we were thinking that maybe the
>>>>>> best
> approach could be to have this Base Processor, and then implement
> readers for particular protocols as Controller Services. But here
> also, it could be very helpful to have your opinion.
>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
> defined some naming br/>>>>>> conventions, or expected structure to
> faciliitate later work. At the br/>>>>>> present, we have a fork of
> the project where we have been working on br/>>>>&ggt;> these Nifi
> changes. We updated the content of our fork (fetch/merge
>>>>>> upstream) about 2 weeks ago, and commited our changes to the
> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our commits?
> how do you br/>>>&>>> prefer to receive the code? (we are not very
> experts on git, just in br/>>>>>> case we could cause some
> problemss...)
>>>>>> br/>>>>>> thank you in addvance
>>>>>> br/>>>>>> iñigo <
>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>> reading br/>>>>>> The more I thinnk of it, br/>>>>>> Perhaps we
>>>>>> shouuld also think of potentially providing some
> information on supported configuration options.
>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> Additionally, the transports could too say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
> like that? Right?
>>>>>> br/>>>>>> Chris <
>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>> reading br/>>>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on
>>>>>> Apache Nifi as well as PLC4X, I would
> be happy to review your processor.
>>>>>> If I understand what you are saying correctly, you have a single
> processor which supports record writing output?
>>>>>> br/>>>>>> plc4x -> reccords
>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
> processor created support on a per protocol basis for configuration
> and validation?
>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>>>>> etc,
> it may be better to have a base processor, and derived processors per
> protocol to handle those differences.
>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>>> Hi all,,
>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>>>>>>> Nifi
> integration part of the project. We have created a Record oriented
> processor for reading PLC data. It is based on the previous existing
> SourceProcessor, but works with records, using a Nifi Writer (such as
> Avro, Json, and so on) to write data on flowfiles content. br/>>>>>>>
> br/>>>>>>> We updated the code on our fork with thee actual PLC4X git
> repo about 2 weeks ago, and tested it reading values with S7 from a
> S7-1200 CPU from Nifi. Also, one of our customers has recently started
> to use it for validation. br/>>>>>>> br/>>>>>>> Currently, it works
> with S7 and Modbus oover TCP. This is because we had to write some
> classes to map connectionString and variableList properties (sintax)
> of the processor to the actual protocol, to be able to build then avro
> schema for output flowfile, taking into account variable datatypes,
> etc. We only did this for
> S7 and Modbus. I am sure that there is a better way to do this, so at
> this point you maybe could take a look to find the best solution and
> avoid needing to do this mapping. br/>>>>>>> br/>>>>>>> If you find
> this useful, we could do a ppull request to the main PLC4x repo. Let
> us know what you think. br/>>>>>>>> br/>>>>>>> best regards,
>>>>>>> iñigo
>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>> Iñigo Angulo
>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
> >>>>>> -----------------------------------------
AW: AW: Nifi integration record oriented processor for reading
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Gustavo,
great ... I had something in the back of my mind, but I wasn't sure anymore ... then we're safe on that side ;-)
Chris
-----Ursprüngliche Nachricht-----
Von: Gustavo Fernandez <gu...@zylk.net.INVALID>
Gesendet: Mittwoch, 18. August 2021 12:32
An: dev@plc4x.apache.org
Betreff: Re: AW: Nifi integration record oriented processor for reading
Hi Chris
I sent my ICLA a month ago. The response from ASF was:
[...]
This message acknowledges receipt of your ICLA, which has been filed in the Apache Software Foundation records.
With this message, the PLC4X PMC has been notified that your ICLA has been filed.
[...]
My apache user id is zgus
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
skype: gus.zylk
http://twitter.com/zylknet
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: dev@plc4x.apache.org
> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
> Asunto: AW: Nifi integration record oriented processor for reading
> HI Inigo,
>
> great news ... I hope Otto can spare some cycles to review the PR from
> the NiFi side ... Happy to assist with merging things back.
>
> As it's been quite some time ... I am pretty sure we have an ICLA on
> file from your side, correct?
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
> Gesendet: Mittwoch, 18. August 2021 10:46
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: Nifi integration record oriented processor for reading
>
> Hi Otto, Chris,
>
> We have gone over the comments on the Nifi PR and made a commit
> including revisions for all (or most) topics.
>
> the last changes we did were related to reforcing the methods for
> datatype conversion (for the record schema construction), and
> extending the tests. About this last one, we changed the
> org.apache.plc4x.nifi.BasePlc4xProcessor class to protected. This led
> to renaming the test classes package to match BasePlc4xProcessor's one
> ("org.apache.plc4x.processors.plc4x4nifi" package was renamed to
> "org.apache.plc4x.nifi"). This last commit caused a conflict on the PR, but we think it can be easily corrected when doing the merge.
>
> At the moment, we think this nifi feature is pretty much completed. Of
> course there are things that can be extended yet, which we will continue working on.
> So please, if you could take a look and check if you miss something
> would be great. Hopefully this could be soon tested by users in the
> community too and get some feedback :)
>
> thank you
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Lunes, 5 de Julio 2021 12:38:14
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> thank you for the review of the PR and your comments.
>
> We have answered them, and will review the sections you suggested. I
> would like to know if there are any things you want us to priorize.
> Maybe you have some deadlines for the next release or something, and
> we could try to get things done by this time. If you miss some task we
> could do here before doing the merge, please let us know.
>
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 24 de Junio 2021 14:40:12
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Great,
> I’m doing the review, so you should mark it ready :)
>
>
>
> From: Iñigo Angulo <ia...@zylk.net.invalid>
> <ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org>
> <de...@plc4x.apache.org>
> Date: June 24, 2021 at 04:52:19
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> I have opened the new pull request, from a dedicated branch in our
> fork, and a commit per contributor, the team members that have been working on this.
>
> Please take a look, and let me know if you want me to mark it as ready
> to review.
>
> iñigo
>
> ----------------------------------------- br/>Iñigo Anngulo br/> < ZYLK.net ::
> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
> br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
> br/>----------------------------------------- <
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 17 de Junio 2021 16:38:13
> Asunto: Re: Nifi integration record oriented processor for reading
>
> That sounds great.
>
>> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
>> br/>> Hi Otto, <
>> br/>> So we did this workaround, we re-forked the reepo and added the
> changes of the nifi-record feature on a dedicated branch. I closed the
> previous PR, will upload the new one on the following days (we are
> doing some last test just in case we forgot to copy something...)
>> br/>> I will keep you informed. <
>> br/>> regards, <
>> iñigo
>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>> br/>>
> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
> Ribera de Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950
> Erandio
> (Bizkaia) br/>> ----------------------------------------- <
>> br/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>> Yes, that sounds great. < Thank you br/>>> On Jun 10, 2021, at
>> 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>>> Hi Otto, <
>>> br/>>> I have tried to do the rebase of our ccommits, but I am
>>> having
> difficulties at it... br/>>> br/>>> The issue is: we forked the repo
> on september 2020, and started making tests and commits to our fork (on 'develop'
> branch). Now I am trying to do a rebase (using git rebase -i ID)
> specifying the ID of our first commit. But when the file is open for
> interactive mode, it gets
> ~900 commits on the develop branch (belonging to members of the PLC4X
> community). I think this happens because before opening the
> PullRequest, I did a 'merge upstream' with the actual PLC4X repo, to get the updates of the code.
> So, I understand that in the interactive mode file, I have to leave
> all commits to 'pick' (by default), and change our commits of the nifi feature to 'squash'
> except from the first one (which also remains as 'pick'). However,
> when I tried this, many conflicts appear, almost one per each commit (comunity members'
> commits)...
>>> I may be doing something wrong (never did a rebase before...) and I
> prefered just to ask, as I dont want to break or cause any conflict to
> the repo code.. If you see anything Im missing please let me know.
>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
>>> the
> fork of the PLC4X repo, and add the changes to the code on a dedicated
> 'feature-nifi-record' branch. Maybe this could make things clearer...
>>> What do you think?
>>> br/>>> thank you, <
>>> iñigo
>>> br/>>> ------------------------------------------ br/>>> Iñigo
>>> Angulo
> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.:
> 747412337 br/>>> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203
> br/>>> 48950 Erandio
> (Bizkaia) br/>>> ----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>> Awesome. < br/>>> If you can, can I ask you to: < br/>>> 1.
>>> Mark the PR as ready to review in github 2. rebase or squash it to a
>>> single commit and force push to your branch
> to clean it up
>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
> <ii...@zylk.net.INVALID> wrote:
>>>> br/>>>> Hi Otto, Chris <
>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
>>>> to
> the Pull Request. The changes we have done are the following:
>>>> - deducing avro datatypes from PlcResponse. Here we may check the
> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in
> order to see if it is the best way to do it.
>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was
> rising errors during compilation.
>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
> (comparing to BaseProcessor), as we have included the posibility to
> have an input connection into the processor, and indicate the target
> addressMap through flowfile attributes. The addressMap is now created in the onTrigger method.
>>>> - we have tested the performance with S7 and Modbus protocols
>>>> (using a
> Siemens S7-1200 and Schneider M221). We will upload an updated nifi
> template for both protocols, but regarding this, do you have any
> testing environment to simulate PLCs? If that is the case, we could
> prepare the Processors configuration to match these ones (connection Strings and addressMaps).
>>>> br/>>>> Please take a look at the code,, any suggestion will be
>>>> very
> welcome.
>>>> br/>>>> iñigo <
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
> 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
> 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>>
> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>>>> prrocessor to include the logic of
> getting the values for variables from the PLC response. We tested it
> with the
> S7-1200 and seems to work fine, however we would like to make some
> further tests before commiting it.
>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>>>> the
> response object, we did it in the following way:
>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>> PlcValue> responseDataStructure =
> readResponse.getAsPlcValue().getStruct();
>>>> for (Map.Entry<String, ? extends PlcValue> entry :
> responseDataStructure.entrySet()) {
>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>>
>>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().a
>>>> n d().intType().endUnion().noDefault();
>
>>>> }else if (entry.getValue() instanceof PlcREAL) {
>>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().an
>>>> d ().doubleType().endUnion().noDefault();
>
>>>> } ... and so on for the rest of the classes on the package
> (org.apache.plc4x.java.spi.values.*)
>>>> br/>>>> //the builder object is used tto build avro schema, with
> desired datatypes (for example intType())
>>>> }
>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
> there is a better way to access PlcValues, please let us know and we
> will update it.
>>>> br/>>>> We will upload the code soon soo that you can take a deeper
> look.
>>>> br/>>>> thank you!!
>>>> iñigo
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.:
> 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo
> 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>>
> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Sounds like a good plan!!
>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
>>>>> and
> started to think about the approach of extracting values from response directly.
> During next week, we will work on this and make some updates to the
> code with the suggestions you made. We keep you informed
>>>>> br/>>>>> thank you, <
>>>>> br/>>>>> iñigo <
>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo
> 201-203 br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
> ----------------------------------------- <
>>>>> br/>>>>> ----- Mensaje originall -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>>>> reading br/>>>>> Hi Inigo, < br/>>>>> especially if you havee a
>>>>> look at the KNX protocol. This
> doesn't define the usual IEC datatypes we tried to use for all normal
> PLC drivers.
>>>>> Here we have hundreds of datatypes that don't match any other
> protocol. I think the PLCValue approach would be the simplest.
>>>>> The one thing you have to keep in mind, is that you should check a
> PLCValue, if it's a list (Array type) or a Structure (Which sort of
> relates to komplex types with a more sophisticated structure).
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
> FFreitag, 23. April 2021 15:34
>>>>> An: dev <de...@plc4x.apache.org>
>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>>>>> now,
> we are generating the schema ourselves. We have a record writer who is
> in charge of reading PLC values. Schema is defined previously to
> reading the values. We build this schema getting the protocol from the
> 'connectionString' (S7, Modbus) and the specified variable type from the 'PLC resource address String'
> containing the list of variable to read. From this we deduce the
> expected Avro datatype when reading, for instance, a word in S7 or a coil in Modbus.
> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach
> will be much clearer and useful. Ideally, getting the actual datatype
> from PLCValue when getting the response.
> Regarding this, we tried to keep the previously described 'mapping'
> separated from the rest of the code, so that hopefully it can be
> easily replaced..
>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>>>>> at
> the code and let us know what you think. We will fill the ICLA document too.
>>>>> br/>>>>> thank you <
>>>>> iñigo
>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>> ----- Mensaje original -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>> Hi all, < br/>>>>> Well, you get PlcValuees from the
>>>>> response that wrap the
> different datatypes. So generally you shouldn't care about the detail type.
>>>>> br/>>>>> However, you can call ggetObject() which returns the core
> value the plc-value has ... so if it's the PLCValue for a Short,
> getObject will return a short value.
>>>>> br/>>>>> Does that help??
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>> reading br/>>>>> So, you are generating the schema yourself, such
>>>>> that
> downstream if they inherit schema they will just get what you
> generate? And you are trying to do that by the connection string? If
> so, a different way I could imagine doing would be to get the ’types’
> of the data from the responses themselves. This would be more generic.
> The flow I could imagine ( in OnTrigger
> ):
>>>>> br/>>>>> DO READ <
>>>>> IF NOT HAS SCHEMA
>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH
>>>>> SCHEMA br/>>>>> Maybe Chris can speak tto how to get the types
>>>>> from the
> responses.
>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>>>>>> give
> an overview. In Nifi, information packages are called Flowfiles, and
> these are the actual units of information that are exchanged between
> Procesors, all along the dataflow. Flowfiles have two sections where
> we can manage
> data: Attributes and Content. In the "traditional" Nifi approach, you
> work with both sections, extracting information from the Content to
> the Attributes and viceversa to perform operations. This approach
> could have one limitation when you are processing batch data (lines
> from a CSV file for instance), where you need to split each of the
> lines into different Flowfiles. Thus, a 1000 line CSV file leads to
> 1000 Flowfiles to process, each of them containing a single record.
>>>>>> br/>>>>>> On later versioons of the product, they introduced the
> Record oriented approach. This approach allows you to manage multiple
> records on a single Flowfile's Content, as long as these records have all the same schema.
> This means that the operations defined by the Processors are applied
> simultaneously to the whole content at once. Following with the
> previous example, a 1000 line CSV file could produce a single Flowfile
> with a content of
> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to
> serialize thee Flowfile's Content. Then, the Record Oriented
> Processors use Writers and Readers to present this information in the
> desired format (such as Avro, Json, CSV, etc). Basically, with the
> record oriented approach, Nifi introduced multiple new Processors, and
> also included the Record version of many of the "old" ones. Using this
> Record approach, Nifi perfomance enhances notably, specially when working with large structured information.
>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>> Processor,
> based on the previously existing one Plc4xSourceProcessor, to read
> values from the devices. We have also included a README on the
> plc4x/plc4j/integrations/apache-nifi module explaining the Processor
> configuration and giving an example. Moreover, we put a nifi template
> with a dataflow for testing these processors, if useful.
>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>>>>> that
> is right. We added the writer capability to the existing
> PLC4XSourceProcessor, so that it formats the output to the desired configuration in a record manner.
> At the actual implementation, we did this "protocol adaptation" from
> the sintax of the particular properties on Processor's configuration.
> For example, from connection string 's7://IP:PORT', we extract the S7
> idenifier and map variable datatypes to the actual Avro datatypes for build the record output schema.
> However, here we dont have vast experience with PLC4X libraries, and
> for sure there will be better ways for doing this.
>>>>>> Also about the Base Processor, we were thinking that maybe the
>>>>>> best
> approach could be to have this Base Processor, and then implement
> readers for particular protocols as Controller Services. But here
> also, it could be very helpful to have your opinion.
>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
> defined some naming br/>>>>>> conventions, or expected structure to
> faciliitate later work. At the br/>>>>>> present, we have a fork of
> the project where we have been working on br/>>>>&ggt;> these Nifi
> changes. We updated the content of our fork (fetch/merge
>>>>>> upstream) about 2 weeks ago, and commited our changes to the
> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our commits?
> how do you br/>>>&>>> prefer to receive the code? (we are not very
> experts on git, just in br/>>>>>> case we could cause some
> problemss...)
>>>>>> br/>>>>>> thank you in addvance
>>>>>> br/>>>>>> iñigo <
>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>> Asunto: AW: Nifi integration record oriented processor for
>>>>>> reading br/>>>>>> The more I thinnk of it, br/>>>>>> Perhaps we
>>>>>> shouuld also think of potentially providing some
> information on supported configuration options.
>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> Additionally, the transports could too say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
> like that? Right?
>>>>>> br/>>>>>> Chris <
>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>> reading br/>>>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on
>>>>>> Apache Nifi as well as PLC4X, I would
> be happy to review your processor.
>>>>>> If I understand what you are saying correctly, you have a single
> processor which supports record writing output?
>>>>>> br/>>>>>> plc4x -> reccords
>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
> processor created support on a per protocol basis for configuration
> and validation?
>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>>>>> etc,
> it may be better to have a base processor, and derived processors per
> protocol to handle those differences.
>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>>> Hi all,,
>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>>>>>>> Nifi
> integration part of the project. We have created a Record oriented
> processor for reading PLC data. It is based on the previous existing
> SourceProcessor, but works with records, using a Nifi Writer (such as
> Avro, Json, and so on) to write data on flowfiles content. br/>>>>>>>
> br/>>>>>>> We updated the code on our fork with thee actual PLC4X git
> repo about 2 weeks ago, and tested it reading values with S7 from a
> S7-1200 CPU from Nifi. Also, one of our customers has recently started
> to use it for validation. br/>>>>>>> br/>>>>>>> Currently, it works
> with S7 and Modbus oover TCP. This is because we had to write some
> classes to map connectionString and variableList properties (sintax)
> of the processor to the actual protocol, to be able to build then avro
> schema for output flowfile, taking into account variable datatypes,
> etc. We only did this for
> S7 and Modbus. I am sure that there is a better way to do this, so at
> this point you maybe could take a look to find the best solution and
> avoid needing to do this mapping. br/>>>>>>> br/>>>>>>> If you find
> this useful, we could do a ppull request to the main PLC4x repo. Let
> us know what you think. br/>>>>>>>> br/>>>>>>> best regards,
>>>>>>> iñigo
>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>> Iñigo Angulo
>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
> >>>>>> -----------------------------------------
Re: AW: Nifi integration record oriented processor for reading
Posted by Gustavo Fernandez <gu...@zylk.net.INVALID>.
Hi Chris
I sent my ICLA a month ago. The response from ASF was:
[...]
This message acknowledges receipt of your ICLA, which has been filed in the Apache Software Foundation records.
With this message, the PLC4X PMC has been notified that your ICLA has been filed.
[...]
My apache user id is zgus
un saludo
--gustavo
-----------------------------------------
Gustavo Fernández
ZYLK.net :: consultoría.openSource
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
movil: +34 637546184
ofic.: +34 944272119
email: gus at zylk.net
skype: gus.zylk
http://twitter.com/zylknet
http://www.zylk.net/web/guest/web-2-0/blog
-----------------------------------------
----- Mensaje original -----
> De: "Christofer Dutz" <ch...@c-ware.de>
> Para: dev@plc4x.apache.org
> Enviados: Miércoles, 18 de Agosto 2021 10:49:43
> Asunto: AW: Nifi integration record oriented processor for reading
> HI Inigo,
>
> great news ... I hope Otto can spare some cycles to review the PR from the NiFi
> side ... Happy to assist with merging things back.
>
> As it's been quite some time ... I am pretty sure we have an ICLA on file from
> your side, correct?
>
> Chris
>
>
> -----Ursprüngliche Nachricht-----
> Von: Iñigo Angulo <ia...@zylk.net.INVALID>
> Gesendet: Mittwoch, 18. August 2021 10:46
> An: dev <de...@plc4x.apache.org>
> Betreff: Re: Nifi integration record oriented processor for reading
>
> Hi Otto, Chris,
>
> We have gone over the comments on the Nifi PR and made a commit including
> revisions for all (or most) topics.
>
> the last changes we did were related to reforcing the methods for datatype
> conversion (for the record schema construction), and extending the tests. About
> this last one, we changed the org.apache.plc4x.nifi.BasePlc4xProcessor class to
> protected. This led to renaming the test classes package to match
> BasePlc4xProcessor's one ("org.apache.plc4x.processors.plc4x4nifi" package was
> renamed to "org.apache.plc4x.nifi"). This last commit caused a conflict on the
> PR, but we think it can be easily corrected when doing the merge.
>
> At the moment, we think this nifi feature is pretty much completed. Of course
> there are things that can be extended yet, which we will continue working on.
> So please, if you could take a look and check if you miss something would be
> great. Hopefully this could be soon tested by users in the community too and
> get some feedback :)
>
> thank you
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Lunes, 5 de Julio 2021 12:38:14
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> thank you for the review of the PR and your comments.
>
> We have answered them, and will review the sections you suggested. I would like
> to know if there are any things you want us to priorize. Maybe you have some
> deadlines for the next release or something, and we could try to get things
> done by this time. If you miss some task we could do here before doing the
> merge, please let us know.
>
> iñigo
>
> -----------------------------------------
> Iñigo Angulo
>
> ZYLK.net :: consultoría.openSource
> telf.: 747412337
> Ribera de Axpe, 11
> Edificio A, modulo 201-203
> 48950 Erandio (Bizkaia)
> -----------------------------------------
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 24 de Junio 2021 14:40:12
> Asunto: Re: Nifi integration record oriented processor for reading
>
> Great,
> I’m doing the review, so you should mark it ready :)
>
>
>
> From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
> Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Date: June 24, 2021 at 04:52:19
> To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
> Subject: Re: Nifi integration record oriented processor for reading
>
> Hi Otto,
>
> I have opened the new pull request, from a dedicated branch in our fork, and a
> commit per contributor, the team members that have been working on this.
>
> Please take a look, and let me know if you want me to mark it as ready to
> review.
>
> iñigo
>
> ----------------------------------------- br/>Iñigo Anngulo br/> < ZYLK.net ::
> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11
> br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
> br/>----------------------------------------- <
>
> ----- Mensaje original -----
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 17 de Junio 2021 16:38:13
> Asunto: Re: Nifi integration record oriented processor for reading
>
> That sounds great.
>
>> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
>> br/>> Hi Otto, <
>> br/>> So we did this workaround, we re-forked the reepo and added the
> changes of the nifi-record feature on a dedicated branch. I closed the previous
> PR, will upload the new one on the following days (we are doing some last test
> just in case we forgot to copy something...)
>> br/>> I will keep you informed. <
>> br/>> regards, <
>> iñigo
>> br/>> ----------------------------------------- br/>> Iñigo Angulo
>> br/>>
> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>> Ribera de
> Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950 Erandio
> (Bizkaia) br/>> ----------------------------------------- <
>> br/>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 10 de Junio 2021 16:14:54
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>> Yes, that sounds great. < Thank you br/>>> On Jun 10, 2021, at
>> 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
> wrote:
>>> br/>>> Hi Otto, <
>>> br/>>> I have tried to do the rebase of our ccommits, but I am having
> difficulties at it... br/>>> br/>>> The issue is: we forked the repo on
> september 2020, and started making tests and commits to our fork (on 'develop'
> branch). Now I am trying to do a rebase (using git rebase -i ID) specifying the
> ID of our first commit. But when the file is open for interactive mode, it gets
> ~900 commits on the develop branch (belonging to members of the PLC4X
> community). I think this happens because before opening the PullRequest, I did
> a 'merge upstream' with the actual PLC4X repo, to get the updates of the code.
> So, I understand that in the interactive mode file, I have to leave all commits
> to 'pick' (by default), and change our commits of the nifi feature to 'squash'
> except from the first one (which also remains as 'pick'). However, when I tried
> this, many conflicts appear, almost one per each commit (comunity members'
> commits)...
>>> I may be doing something wrong (never did a rebase before...) and I
> prefered just to ask, as I dont want to break or cause any conflict to the repo
> code.. If you see anything Im missing please let me know.
>>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
>>> the
> fork of the PLC4X repo, and add the changes to the code on a dedicated
> 'feature-nifi-record' branch. Maybe this could make things clearer...
>>> What do you think?
>>> br/>>> thank you, <
>>> iñigo
>>> br/>>> ------------------------------------------ br/>>> Iñigo Angulo
> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: 747412337 br/>>>
> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203 br/>>> 48950 Erandio
> (Bizkaia) br/>>> ----------------------------------------- <
>>> br/>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>> Awesome. < br/>>> If you can, can I ask you to: < br/>>> 1.
>>> Mark the PR as ready to review in github 2. rebase or squash it to a
>>> single commit and force push to your branch
> to clean it up
>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
> <ii...@zylk.net.INVALID> wrote:
>>>> br/>>>> Hi Otto, Chris <
>>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
>>>> to
> the Pull Request. The changes we have done are the following:
>>>> - deducing avro datatypes from PlcResponse. Here we may check the
> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in order to see
> if it is the best way to do it.
>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was rising
> errors during compilation.
>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
> (comparing to BaseProcessor), as we have included the posibility to have an
> input connection into the processor, and indicate the target addressMap through
> flowfile attributes. The addressMap is now created in the onTrigger method.
>>>> - we have tested the performance with S7 and Modbus protocols (using
>>>> a
> Siemens S7-1200 and Schneider M221). We will upload an updated nifi template for
> both protocols, but regarding this, do you have any testing environment to
> simulate PLCs? If that is the case, we could prepare the Processors
> configuration to match these ones (connection Strings and addressMaps).
>>>> br/>>>> Please take a look at the code,, any suggestion will be very
> welcome.
>>>> br/>>>> iñigo <
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
> br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
> Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>>>> prrocessor to include the logic of
> getting the values for variables from the PLC response. We tested it with the
> S7-1200 and seems to work fine, however we would like to make some further
> tests before commiting it.
>>>> br/>>>> Regarding the actual method whiich takes the datatype from
>>>> the
> response object, we did it in the following way:
>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>>> PlcValue> responseDataStructure =
> readResponse.getAsPlcValue().getStruct();
>>>> for (Map.Entry<String, ? extends PlcValue> entry :
> responseDataStructure.entrySet()) {
>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>>
>>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().an
>>>> d().intType().endUnion().noDefault();
>
>>>> }else if (entry.getValue() instanceof PlcREAL) {
>>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().and
>>>> ().doubleType().endUnion().noDefault();
>
>>>> } ... and so on for the rest of the classes on the package
> (org.apache.plc4x.java.spi.values.*)
>>>> br/>>>> //the builder object is used tto build avro schema, with
> desired datatypes (for example intType())
>>>> }
>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
> there is a better way to access PlcValues, please let us know and we will update
> it.
>>>> br/>>>> We will upload the code soon soo that you can take a deeper
> look.
>>>> br/>>>> thank you!!
>>>> iñigo
>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>>> Angulo
> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
> br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
> Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>>> br/>>>> ----- Mensaje original ----- <
>>>> De: "ottobackwards" <ot...@gmail.com>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>>> Asunto: Re: Nifi integration record oriented processor for reading
>>>> br/>>>> Sounds like a good plan!!
>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> we have been reviewingg the comments on the pull request,
>>>>> and
> started to think about the approach of extracting values from response directly.
> During next week, we will work on this and make some updates to the code with
> the suggestions you made. We keep you informed
>>>>> br/>>>>> thank you, <
>>>>> br/>>>>> iñigo <
>>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo 201-203
> br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
> ----------------------------------------- <
>>>>> br/>>>>> ----- Mensaje originall -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>>>> reading br/>>>>> Hi Inigo, < br/>>>>> especially if you havee a
>>>>> look at the KNX protocol. This
> doesn't define the usual IEC datatypes we tried to use for all normal PLC
> drivers.
>>>>> Here we have hundreds of datatypes that don't match any other
> protocol. I think the PLCValue approach would be the simplest.
>>>>> The one thing you have to keep in mind, is that you should check a
> PLCValue, if it's a list (Array type) or a Structure (Which sort of relates to
> komplex types with a more sophisticated structure).
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
> FFreitag, 23. April 2021 15:34
>>>>> An: dev <de...@plc4x.apache.org>
>>>>> Betreff: Re: AW: Nifi integration record oriented processor for
> reading
>>>>> br/>>>>> Hi Otto, Chris, <
>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>>>>> now,
> we are generating the schema ourselves. We have a record writer who is in charge
> of reading PLC values. Schema is defined previously to reading the values. We
> build this schema getting the protocol from the 'connectionString' (S7, Modbus)
> and the specified variable type from the 'PLC resource address String'
> containing the list of variable to read. From this we deduce the expected Avro
> datatype when reading, for instance, a word in S7 or a coil in Modbus.
> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach will be
> much clearer and useful. Ideally, getting the actual datatype from PLCValue
> when getting the response.
> Regarding this, we tried to keep the previously described 'mapping'
> separated from the rest of the code, so that hopefully it can be easily
> replaced..
>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>>>>> at
> the code and let us know what you think. We will fill the ICLA document too.
>>>>> br/>>>>> thank you <
>>>>> iñigo
>>>>> br/>>>>> br/>>>>> br/>>>>>
>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>> ----- Mensaje original -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>> Hi all, < br/>>>>> Well, you get PlcValuees from the
>>>>> response that wrap the
> different datatypes. So generally you shouldn't care about the detail type.
>>>>> br/>>>>> However, you can call ggetObject() which returns the core
> value the plc-value has ... so if it's the PLCValue for a Short, getObject will
> return a short value.
>>>>> br/>>>>> Does that help??
>>>>> br/>>>>> Chris <
>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>>> br/>>>>> So, you are generating the schema yourself, such that
> downstream if they inherit schema they will just get what you generate? And you
> are trying to do that by the connection string? If so, a different way I could
> imagine doing would be to get the ’types’ of the data from the responses
> themselves. This would be more generic. The flow I could imagine ( in OnTrigger
> ):
>>>>> br/>>>>> DO READ <
>>>>> IF NOT HAS SCHEMA
>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH SCHEMA
>>>>> br/>>>>> Maybe Chris can speak tto how to get the types from the
> responses.
>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>>>>>> give
> an overview. In Nifi, information packages are called Flowfiles, and these are
> the actual units of information that are exchanged between Procesors, all along
> the dataflow. Flowfiles have two sections where we can manage
> data: Attributes and Content. In the "traditional" Nifi approach, you work with
> both sections, extracting information from the Content to the Attributes and
> viceversa to perform operations. This approach could have one limitation when
> you are processing batch data (lines from a CSV file for instance), where you
> need to split each of the lines into different Flowfiles. Thus, a 1000 line CSV
> file leads to 1000 Flowfiles to process, each of them containing a single
> record.
>>>>>> br/>>>>>> On later versioons of the product, they introduced the
> Record oriented approach. This approach allows you to manage multiple records on
> a single Flowfile's Content, as long as these records have all the same schema.
> This means that the operations defined by the Processors are applied
> simultaneously to the whole content at once. Following with the previous
> example, a 1000 line CSV file could produce a single Flowfile with a content of
> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to serialize thee
> Flowfile's Content. Then, the Record Oriented Processors use Writers and
> Readers to present this information in the desired format (such as Avro, Json,
> CSV, etc). Basically, with the record oriented approach, Nifi introduced
> multiple new Processors, and also included the Record version of many of the
> "old" ones. Using this Record approach, Nifi perfomance enhances notably,
> specially when working with large structured information.
>>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>>> Processor,
> based on the previously existing one Plc4xSourceProcessor, to read values from
> the devices. We have also included a README on the
> plc4x/plc4j/integrations/apache-nifi module explaining the Processor
> configuration and giving an example. Moreover, we put a nifi template with a
> dataflow for testing these processors, if useful.
>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>>>>> that
> is right. We added the writer capability to the existing PLC4XSourceProcessor,
> so that it formats the output to the desired configuration in a record manner.
> At the actual implementation, we did this "protocol adaptation" from the sintax
> of the particular properties on Processor's configuration. For example, from
> connection string 's7://IP:PORT', we extract the S7 idenifier and map variable
> datatypes to the actual Avro datatypes for build the record output schema.
> However, here we dont have vast experience with PLC4X libraries, and for sure
> there will be better ways for doing this.
>>>>>> Also about the Base Processor, we were thinking that maybe the
>>>>>> best
> approach could be to have this Base Processor, and then implement readers for
> particular protocols as Controller Services. But here also, it could be very
> helpful to have your opinion.
>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have defined
> some naming br/>>>>>> conventions, or expected structure to faciliitate later
> work. At the br/>>>>>> present, we have a fork of the project where we have
> been working on br/>>>>&ggt;> these Nifi changes. We updated the content of our
> fork (fetch/merge
>>>>>> upstream) about 2 weeks ago, and commited our changes to the
> 'develop' br/>>>>>> branch. Do we bettter create a new branch with our commits?
> how do you br/>>>&>>> prefer to receive the code? (we are not very experts on
> git, just in br/>>>>>> case we could cause some problemss...)
>>>>>> br/>>>>>> thank you in addvance
>>>>>> br/>>>>>> iñigo <
>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
>>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>>> br/>>>>>> The more I thinnk of it, br/>>>>>> Perhaps we shouuld
>>>>>> also think of potentially providing some
> information on supported configuration options.
>>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> Additionally, the transports could too say: "I generally have
>>>>>> these
> options and they have these datatypes and mean this"
>>>>>> br/>>>>>> I would be our StreamPipes friends would love something
> like that? Right?
>>>>>> br/>>>>>> Chris <
>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>>> reading br/>>>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on Apache
>>>>>> Nifi as well as PLC4X, I would
> be happy to review your processor.
>>>>>> If I understand what you are saying correctly, you have a single
> processor which supports record writing output?
>>>>>> br/>>>>>> plc4x -> reccords
>>>>>> br/>>>>>> And that you haave, for configuration purposes for that
> processor created support on a per protocol basis for configuration and
> validation?
>>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>>>>> etc,
> it may be better to have a base processor, and derived processors per protocol
> to handle those differences.
>>>>>> br/>>>>>> I look forward to seeing the code.
>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
> <ia...@zylk.net.INVALID> wrote:
>>>>>>> br/>>>>>>> Hi all,,
>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>>>>>>> Nifi
> integration part of the project. We have created a Record oriented processor for
> reading PLC data. It is based on the previous existing SourceProcessor, but
> works with records, using a Nifi Writer (such as Avro, Json, and so on) to
> write data on flowfiles content. br/>>>>>>> br/>>>>>>> We updated the code on
> our fork with thee actual PLC4X git repo about 2 weeks ago, and tested it
> reading values with S7 from a S7-1200 CPU from Nifi. Also, one of our customers
> has recently started to use it for validation. br/>>>>>>> br/>>>>>>> Currently,
> it works with S7 and Modbus oover TCP. This is because we had to write some
> classes to map connectionString and variableList properties (sintax) of the
> processor to the actual protocol, to be able to build then avro schema for
> output flowfile, taking into account variable datatypes, etc. We only did this
> for
> S7 and Modbus. I am sure that there is a better way to do this, so at this point
> you maybe could take a look to find the best solution and avoid needing to do
> this mapping. br/>>>>>>> br/>>>>>>> If you find this useful, we could do a
> ppull request to the main PLC4x repo. Let us know what you think. br/>>>>>>>>
> br/>>>>>>> best regards,
>>>>>>> iñigo
>>>>>>> br/>>>>>>> ------------------------------------------
>>>>>>> Iñigo Angulo
>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>>> telf.: 747412337
>>>>>>> Ribera de Axpe, 11
>>>>>>> Edificio A, modulo 201-203
>>>>>>> 48950 Erandio (Bizkaia)
> >>>>>> -----------------------------------------
AW: Nifi integration record oriented processor for reading
Posted by Christofer Dutz <ch...@c-ware.de>.
HI Inigo,
great news ... I hope Otto can spare some cycles to review the PR from the NiFi side ... Happy to assist with merging things back.
As it's been quite some time ... I am pretty sure we have an ICLA on file from your side, correct?
Chris
-----Ursprüngliche Nachricht-----
Von: Iñigo Angulo <ia...@zylk.net.INVALID>
Gesendet: Mittwoch, 18. August 2021 10:46
An: dev <de...@plc4x.apache.org>
Betreff: Re: Nifi integration record oriented processor for reading
Hi Otto, Chris,
We have gone over the comments on the Nifi PR and made a commit including revisions for all (or most) topics.
the last changes we did were related to reforcing the methods for datatype conversion (for the record schema construction), and extending the tests. About this last one, we changed the org.apache.plc4x.nifi.BasePlc4xProcessor class to protected. This led to renaming the test classes package to match BasePlc4xProcessor's one ("org.apache.plc4x.processors.plc4x4nifi" package was renamed to "org.apache.plc4x.nifi"). This last commit caused a conflict on the PR, but we think it can be easily corrected when doing the merge.
At the moment, we think this nifi feature is pretty much completed. Of course there are things that can be extended yet, which we will continue working on.
So please, if you could take a look and check if you miss something would be great. Hopefully this could be soon tested by users in the community too and get some feedback :)
thank you
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Lunes, 5 de Julio 2021 12:38:14
Asunto: Re: Nifi integration record oriented processor for reading
Hi Otto,
thank you for the review of the PR and your comments.
We have answered them, and will review the sections you suggested. I would like to know if there are any things you want us to priorize. Maybe you have some deadlines for the next release or something, and we could try to get things done by this time. If you miss some task we could do here before doing the merge, please let us know.
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 24 de Junio 2021 14:40:12
Asunto: Re: Nifi integration record oriented processor for reading
Great,
I’m doing the review, so you should mark it ready :)
From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: June 24, 2021 at 04:52:19
To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: Nifi integration record oriented processor for reading
Hi Otto,
I have opened the new pull request, from a dedicated branch in our fork, and a commit per contributor, the team members that have been working on this.
Please take a look, and let me know if you want me to mark it as ready to review.
iñigo
----------------------------------------- br/>Iñigo Anngulo br/> < ZYLK.net :: consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11 br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
br/>----------------------------------------- <
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 17 de Junio 2021 16:38:13
Asunto: Re: Nifi integration record oriented processor for reading
That sounds great.
> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
> br/>> Hi Otto, <
> br/>> So we did this workaround, we re-forked the reepo and added the
changes of the nifi-record feature on a dedicated branch. I closed the previous PR, will upload the new one on the following days (we are doing some last test just in case we forgot to copy something...)
> br/>> I will keep you informed. <
> br/>> regards, <
> iñigo
> br/>> ----------------------------------------- br/>> Iñigo Angulo
> br/>>
br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>> Ribera de Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950 Erandio
(Bizkaia) br/>> ----------------------------------------- <
> br/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 10 de Junio 2021 16:14:54
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> Yes, that sounds great. < Thank you br/>>> On Jun 10, 2021, at
> 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
wrote:
>> br/>>> Hi Otto, <
>> br/>>> I have tried to do the rebase of our ccommits, but I am having
difficulties at it... br/>>> br/>>> The issue is: we forked the repo on september 2020, and started making tests and commits to our fork (on 'develop' branch). Now I am trying to do a rebase (using git rebase -i ID) specifying the ID of our first commit. But when the file is open for interactive mode, it gets ~900 commits on the develop branch (belonging to members of the PLC4X community). I think this happens because before opening the PullRequest, I did a 'merge upstream' with the actual PLC4X repo, to get the updates of the code. So, I understand that in the interactive mode file, I have to leave all commits to 'pick' (by default), and change our commits of the nifi feature to 'squash' except from the first one (which also remains as 'pick'). However, when I tried this, many conflicts appear, almost one per each commit (comunity members' commits)...
>> I may be doing something wrong (never did a rebase before...) and I
prefered just to ask, as I dont want to break or cause any conflict to the repo code.. If you see anything Im missing please let me know.
>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do
>> the
fork of the PLC4X repo, and add the changes to the code on a dedicated 'feature-nifi-record' branch. Maybe this could make things clearer...
>> What do you think?
>> br/>>> thank you, <
>> iñigo
>> br/>>> ------------------------------------------ br/>>> Iñigo Angulo
br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: 747412337 br/>>> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203 br/>>> 48950 Erandio (Bizkaia) br/>>> ----------------------------------------- <
>> br/>>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>>> Awesome. < br/>>> If you can, can I ask you to: < br/>>> 1.
>> Mark the PR as ready to review in github 2. rebase or squash it to a
>> single commit and force push to your branch
to clean it up
>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
<ii...@zylk.net.INVALID> wrote:
>>> br/>>>> Hi Otto, Chris <
>>> br/>>>> we have finally commited the uupdates on the Nifi Processor
>>> to
the Pull Request. The changes we have done are the following:
>>> - deducing avro datatypes from PlcResponse. Here we may check the
method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in order to see if it is the best way to do it.
>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
org.apache.nifi:nifi-standard-nar, as it is used on runtime and was rising errors during compilation.
>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
(comparing to BaseProcessor), as we have included the posibility to have an input connection into the processor, and indicate the target addressMap through flowfile attributes. The addressMap is now created in the onTrigger method.
>>> - we have tested the performance with S7 and Modbus protocols (using
>>> a
Siemens S7-1200 and Schneider M221). We will upload an updated nifi template for both protocols, but regarding this, do you have any testing environment to simulate PLCs? If that is the case, we could prepare the Processors configuration to match these ones (connection Strings and addressMaps).
>>> br/>>>> Please take a look at the code,, any suggestion will be very
welcome.
>>> br/>>>> iñigo <
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the
>>> prrocessor to include the logic of
getting the values for variables from the PLC response. We tested it with the S7-1200 and seems to work fine, however we would like to make some further tests before commiting it.
>>> br/>>>> Regarding the actual method whiich takes the datatype from
>>> the
response object, we did it in the following way:
>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends
>>> PlcValue> responseDataStructure =
readResponse.getAsPlcValue().getStruct();
>>> for (Map.Entry<String, ? extends PlcValue> entry :
responseDataStructure.entrySet()) {
>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>>
>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().an
>>> d().intType().endUnion().noDefault();
>>> }else if (entry.getValue() instanceof PlcREAL) {
>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().and
>>> ().doubleType().endUnion().noDefault();
>>> } ... and so on for the rest of the classes on the package
(org.apache.plc4x.java.spi.values.*)
>>> br/>>>> //the builder object is used tto build avro schema, with
desired datatypes (for example intType())
>>> }
>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
there is a better way to access PlcValues, please let us know and we will update it.
>>> br/>>>> We will upload the code soon soo that you can take a deeper
look.
>>> br/>>>> thank you!!
>>> iñigo
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo
>>> Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337 br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950 Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Sounds like a good plan!!
>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> we have been reviewingg the comments on the pull request,
>>>> and
started to think about the approach of extracting values from response directly. During next week, we will work on this and make some updates to the code with the suggestions you made. We keep you informed
>>>> br/>>>>> thank you, <
>>>> br/>>>>> iñigo <
>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo 201-203 br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
----------------------------------------- <
>>>> br/>>>>> ----- Mensaje originall -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>> Asunto: AW: AW: Nifi integration record oriented processor for
>>>> reading br/>>>>> Hi Inigo, < br/>>>>> especially if you havee a
>>>> look at the KNX protocol. This
doesn't define the usual IEC datatypes we tried to use for all normal PLC drivers.
>>>> Here we have hundreds of datatypes that don't match any other
protocol. I think the PLCValue approach would be the simplest.
>>>> The one thing you have to keep in mind, is that you should check a
PLCValue, if it's a list (Array type) or a Structure (Which sort of relates to komplex types with a more sophisticated structure).
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
FFreitag, 23. April 2021 15:34
>>>> An: dev <de...@plc4x.apache.org>
>>>> Betreff: Re: AW: Nifi integration record oriented processor for
reading
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> Yes, I think the approoach you propose will be best. By
>>>> now,
we are generating the schema ourselves. We have a record writer who is in charge of reading PLC values. Schema is defined previously to reading the values. We build this schema getting the protocol from the 'connectionString' (S7, Modbus) and the specified variable type from the 'PLC resource address String' containing the list of variable to read. From this we deduce the expected Avro datatype when reading, for instance, a word in S7 or a coil in Modbus. br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach will be much clearer and useful. Ideally, getting the actual datatype from PLCValue when getting the response.
Regarding this, we tried to keep the previously described 'mapping'
separated from the rest of the code, so that hopefully it can be easily replaced..
>>>> br/>>>>> We have done the pull rrequest, hope you can take a look
>>>> at
the code and let us know what you think. We will fill the ICLA document too.
>>>> br/>>>>> thank you <
>>>> iñigo
>>>> br/>>>>> br/>>>>> br/>>>>>
>>>> ----------------------------------------- < Iñigo Angulo br/>>>>>
>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> br/>>>>> ----- Mensaje original -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>> br/>>>>> Hi all, < br/>>>>> Well, you get PlcValuees from the
>>>> response that wrap the
different datatypes. So generally you shouldn't care about the detail type.
>>>> br/>>>>> However, you can call ggetObject() which returns the core
value the plc-value has ... so if it's the PLCValue for a Short, getObject will return a short value.
>>>> br/>>>>> Does that help??
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Otto Fowler <ot...@gmail.com>
>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>> br/>>>>> So, you are generating the schema yourself, such that
downstream if they inherit schema they will just get what you generate? And you are trying to do that by the connection string? If so, a different way I could imagine doing would be to get the ’types’ of the data from the responses themselves. This would be more generic. The flow I could imagine ( in OnTrigger ):
>>>> br/>>>>> DO READ <
>>>> IF NOT HAS SCHEMA
>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH SCHEMA
>>>> br/>>>>> Maybe Chris can speak tto how to get the types from the
responses.
>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
<ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to
>>>>> give
an overview. In Nifi, information packages are called Flowfiles, and these are the actual units of information that are exchanged between Procesors, all along the dataflow. Flowfiles have two sections where we can manage
data: Attributes and Content. In the "traditional" Nifi approach, you work with both sections, extracting information from the Content to the Attributes and viceversa to perform operations. This approach could have one limitation when you are processing batch data (lines from a CSV file for instance), where you need to split each of the lines into different Flowfiles. Thus, a 1000 line CSV file leads to 1000 Flowfiles to process, each of them containing a single record.
>>>>> br/>>>>>> On later versioons of the product, they introduced the
Record oriented approach. This approach allows you to manage multiple records on a single Flowfile's Content, as long as these records have all the same schema. This means that the operations defined by the Processors are applied simultaneously to the whole content at once. Following with the previous example, a 1000 line CSV file could produce a single Flowfile with a content of 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to serialize thee Flowfile's Content. Then, the Record Oriented Processors use Writers and Readers to present this information in the desired format (such as Avro, Json, CSV, etc). Basically, with the record oriented approach, Nifi introduced multiple new Processors, and also included the Record version of many of the "old" ones. Using this Record approach, Nifi perfomance enhances notably, specially when working with large structured information.
>>>>> br/>>>>>> The work we didd was creating a Record Oriented
>>>>> Processor,
based on the previously existing one Plc4xSourceProcessor, to read values from the devices. We have also included a README on the plc4x/plc4j/integrations/apache-nifi module explaining the Processor configuration and giving an example. Moreover, we put a nifi template with a dataflow for testing these processors, if useful.
>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor,
>>>>> that
is right. We added the writer capability to the existing PLC4XSourceProcessor, so that it formats the output to the desired configuration in a record manner. At the actual implementation, we did this "protocol adaptation" from the sintax of the particular properties on Processor's configuration. For example, from connection string 's7://IP:PORT', we extract the S7 idenifier and map variable datatypes to the actual Avro datatypes for build the record output schema. However, here we dont have vast experience with PLC4X libraries, and for sure there will be better ways for doing this.
>>>>> Also about the Base Processor, we were thinking that maybe the
>>>>> best
approach could be to have this Base Processor, and then implement readers for particular protocols as Controller Services. But here also, it could be very helpful to have your opinion.
>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have defined some naming br/>>>>>> conventions, or expected structure to faciliitate later work. At the br/>>>>>> present, we have a fork of the project where we have been working on br/>>>>&ggt;> these Nifi changes. We updated the content of our fork (fetch/merge
>>>>> upstream) about 2 weeks ago, and commited our changes to the
'develop' br/>>>>>> branch. Do we bettter create a new branch with our commits? how do you br/>>>&>>> prefer to receive the code? (we are not very experts on git, just in br/>>>>>> case we could cause some problemss...)
>>>>> br/>>>>>> thank you in addvance
>>>>> br/>>>>>> iñigo <
>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>>> The more I thinnk of it, br/>>>>>> Perhaps we shouuld
>>>>> also think of potentially providing some
information on supported configuration options.
>>>>> Wouldn't it be cool if the driver could say: "I generally have
>>>>> these
options and they have these datatypes and mean this"
>>>>> Additionally, the transports could too say: "I generally have
>>>>> these
options and they have these datatypes and mean this"
>>>>> br/>>>>>> I would be our StreamPipes friends would love something
like that? Right?
>>>>> br/>>>>>> Chris <
>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for
>>>>> reading br/>>>>>> Hi Inigo, < br/>>>>>> I’m a coommitter on Apache
>>>>> Nifi as well as PLC4X, I would
be happy to review your processor.
>>>>> If I understand what you are saying correctly, you have a single
processor which supports record writing output?
>>>>> br/>>>>>> plc4x -> reccords
>>>>> br/>>>>>> And that you haave, for configuration purposes for that
processor created support on a per protocol basis for configuration and validation?
>>>>> br/>>>>>> If there is perr protocol configuration / validation
>>>>> etc,
it may be better to have a base processor, and derived processors per protocol to handle those differences.
>>>>> br/>>>>>> I look forward to seeing the code.
>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>>> Hi all,,
>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache
>>>>>> Nifi
integration part of the project. We have created a Record oriented processor for reading PLC data. It is based on the previous existing SourceProcessor, but works with records, using a Nifi Writer (such as Avro, Json, and so on) to write data on flowfiles content. br/>>>>>>> br/>>>>>>> We updated the code on our fork with thee actual PLC4X git repo about 2 weeks ago, and tested it reading values with S7 from a S7-1200 CPU from Nifi. Also, one of our customers has recently started to use it for validation. br/>>>>>>> br/>>>>>>> Currently, it works with S7 and Modbus oover TCP. This is because we had to write some classes to map connectionString and variableList properties (sintax) of the processor to the actual protocol, to be able to build then avro schema for output flowfile, taking into account variable datatypes, etc. We only did this for
S7 and Modbus. I am sure that there is a better way to do this, so at this point you maybe could take a look to find the best solution and avoid needing to do this mapping. br/>>>>>>> br/>>>>>>> If you find this useful, we could do a ppull request to the main PLC4x repo. Let us know what you think. br/>>>>>>>> br/>>>>>>> best regards,
>>>>>> iñigo
>>>>>> br/>>>>>>> ------------------------------------------
>>>>>> Iñigo Angulo
>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------
Re: Nifi integration record oriented processor for reading
Posted by Iñigo Angulo <ia...@zylk.net.INVALID>.
Hi Otto, Chris,
We have gone over the comments on the Nifi PR and made a commit including revisions for all (or most) topics.
the last changes we did were related to reforcing the methods for datatype conversion (for the record schema construction), and extending the tests. About this last one, we changed the org.apache.plc4x.nifi.BasePlc4xProcessor class to protected. This led to renaming the test classes package to match BasePlc4xProcessor's one ("org.apache.plc4x.processors.plc4x4nifi" package was renamed to "org.apache.plc4x.nifi"). This last commit caused a conflict on the PR, but we think it can be easily corrected when doing the merge.
At the moment, we think this nifi feature is pretty much completed. Of course there are things that can be extended yet, which we will continue working on.
So please, if you could take a look and check if you miss something would be great. Hopefully this could be soon tested by users in the community too and get some feedback :)
thank you
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Lunes, 5 de Julio 2021 12:38:14
Asunto: Re: Nifi integration record oriented processor for reading
Hi Otto,
thank you for the review of the PR and your comments.
We have answered them, and will review the sections you suggested. I would like to know if there are any things you want us to priorize. Maybe you have some deadlines for the next release or something, and we could try to get things done by this time. If you miss some task we could do here before doing the merge, please let us know.
iñigo
-----------------------------------------
Iñigo Angulo
ZYLK.net :: consultoría.openSource
telf.: 747412337
Ribera de Axpe, 11
Edificio A, modulo 201-203
48950 Erandio (Bizkaia)
-----------------------------------------
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 24 de Junio 2021 14:40:12
Asunto: Re: Nifi integration record oriented processor for reading
Great,
I’m doing the review, so you should mark it ready :)
From: Iñigo Angulo <ia...@zylk.net.invalid> <ia...@zylk.net.invalid>
Reply: dev@plc4x.apache.org <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Date: June 24, 2021 at 04:52:19
To: dev <de...@plc4x.apache.org> <de...@plc4x.apache.org>
Subject: Re: Nifi integration record oriented processor for reading
Hi Otto,
I have opened the new pull request, from a dedicated branch in our fork,
and a commit per contributor, the team members that have been working on
this.
Please take a look, and let me know if you want me to mark it as ready to
review.
iñigo
----------------------------------------- br/>Iñigo Anngulo br/> <
ZYLK.net :: consultoría.openSource br/>telf.: 7474123337 br/>Ribera de
Axpe, 11 br/>Edificio A, modulo 201-203 br/>48950 Erandio (Bizkaia)
br/>----------------------------------------- <
----- Mensaje original -----
De: "ottobackwards" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>
Enviados: Jueves, 17 de Junio 2021 16:38:13
Asunto: Re: Nifi integration record oriented processor for reading
That sounds great.
> On Jun 17, 2021, at 10:14, Iñigo Angulo <ia...@zylk.net.INVALID> wrote:
> br/>> Hi Otto, <
> br/>> So we did this workaround, we re-forked the reepo and added the
changes of the nifi-record feature on a dedicated branch. I closed the
previous PR, will upload the new one on the following days (we are doing
some last test just in case we forgot to copy something...)
> br/>> I will keep you informed. <
> br/>> regards, <
> iñigo
> br/>> ----------------------------------------- br/>> Iñigo Angulo br/>>
br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>>
Ribera de Axpe, 11 br/>> Edificio A, modulo 201-203 br/>> 48950 Erandio
(Bizkaia) br/>> ----------------------------------------- <
> br/>> ----- Mensaje original ----- <
> De: "ottobackwards" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>
> Enviados: Jueves, 10 de Junio 2021 16:14:54
> Asunto: Re: Nifi integration record oriented processor for reading
> br/>> Yes, that sounds great. <
> Thank you
> br/>>> On Jun 10, 2021, at 03:43, Iñigo Anggulo <ia...@zylk.net.INVALID>
wrote:
>> br/>>> Hi Otto, <
>> br/>>> I have tried to do the rebase of our ccommits, but I am having
difficulties at it... br/>>> br/>>> The issue is: we forked the repo on
september 2020, and started making tests and commits to our fork (on
'develop' branch). Now I am trying to do a rebase (using git rebase -i ID)
specifying the ID of our first commit. But when the file is open for
interactive mode, it gets ~900 commits on the develop branch (belonging to
members of the PLC4X community). I think this happens because before
opening the PullRequest, I did a 'merge upstream' with the actual PLC4X
repo, to get the updates of the code. So, I understand that in the
interactive mode file, I have to leave all commits to 'pick' (by default),
and change our commits of the nifi feature to 'squash' except from the
first one (which also remains as 'pick'). However, when I tried this, many
conflicts appear, almost one per each commit (comunity members' commits)...
>> I may be doing something wrong (never did a rebase before...) and I
prefered just to ask, as I dont want to break or cause any conflict to the
repo code.. If you see anything Im missing please let me know.
>> br/>>> As a workaround, I was thinking we coulld close the PR, re-do the
fork of the PLC4X repo, and add the changes to the code on a dedicated
'feature-nifi-record' branch. Maybe this could make things clearer...
>> What do you think?
>> br/>>> thank you, <
>> iñigo
>> br/>>> ------------------------------------------ br/>>> Iñigo Angulo
br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: 747412337
br/>>> Ribera de Axpe, 11 br/>>> Edificio A, modulo 201-203 br/>>> 48950
Erandio (Bizkaia) br/>>> ----------------------------------------- <
>> br/>>> ----- Mensaje original ----- <
>> De: "ottobackwards" <ot...@gmail.com>
>> Para: "dev" <de...@plc4x.apache.org>
>> Enviados: Jueves, 27 de Mayo 2021 16:10:19
>> Asunto: Re: Nifi integration record oriented processor for reading
>> br/>>> Awesome. <
>> br/>>> If you can, can I ask you to: <
>> br/>>> 1. Mark the PR as ready to review in github
>> 2. rebase or squash it to a single commit and force push to your branch
to clean it up
>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo
<ii...@zylk.net.INVALID> wrote:
>>> br/>>>> Hi Otto, Chris <
>>> br/>>>> we have finally commited the uupdates on the Nifi Processor to
the Pull Request. The changes we have done are the following:
>>> - deducing avro datatypes from PlcResponse. Here we may check the
method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in order to
see if it is the best way to do it.
>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for
org.apache.nifi:nifi-standard-nar, as it is used on runtime and was rising
errors during compilation.
>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor
(comparing to BaseProcessor), as we have included the posibility to have an
input connection into the processor, and indicate the target addressMap
through flowfile attributes. The addressMap is now created in the onTrigger
method.
>>> - we have tested the performance with S7 and Modbus protocols (using a
Siemens S7-1200 and Schneider M221). We will upload an updated nifi
template for both protocols, but regarding this, do you have any testing
environment to simulate PLCs? If that is the case, we could prepare the
Processors configuration to match these ones (connection Strings and
addressMaps).
>>> br/>>>> Please take a look at the code,, any suggestion will be very
welcome.
>>> br/>>>> iñigo <
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "Iñigo Angulo" <ia...@zylk.net.INVALID>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Hi Otto, Chris, <
>>> br/>>>> we have been working on the prrocessor to include the logic of
getting the values for variables from the PLC response. We tested it with
the S7-1200 and seems to work fine, however we would like to make some
further tests before commiting it.
>>> br/>>>> Regarding the actual method whiich takes the datatype from the
response object, we did it in the following way:
>>> br/>>>> //PlcReadResponse readResponse
>>> Map<String, ? extends PlcValue> responseDataStructure =
readResponse.getAsPlcValue().getStruct();
>>> for (Map.Entry<String, ? extends PlcValue> entry :
responseDataStructure.entrySet()) {
>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> builder.name(fielldName).type().unionOf().nullBuilder().endNull().and().intType().endUnion().noDefault();
>>> }else if (entry.getValue() instanceof PlcREAL) {
>>> builder.name(fieldName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();
>>> } ... and so on for the rest of the classes on the package
(org.apache.plc4x.java.spi.values.*)
>>> br/>>>> //the builder object is used tto build avro schema, with
desired datatypes (for example intType())
>>> }
>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you think
there is a better way to access PlcValues, please let us know and we will
update it.
>>> br/>>>> We will upload the code soon soo that you can take a deeper
look.
>>> br/>>>> thank you!!
>>> iñigo
>>> br/>>>> ------------------------------------------ br/>>>> Iñigo Angulo
br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: 747412337
br/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo 201-203 br/>>>> 48950
Erandio (Bizkaia) br/>>>> ----------------------------------------- <
>>> br/>>>> ----- Mensaje original ----- <
>>> De: "ottobackwards" <ot...@gmail.com>
>>> Para: "dev" <de...@plc4x.apache.org>
>>> Enviados: Viernes, 30 de Abril 2021 14:50:11
>>> Asunto: Re: Nifi integration record oriented processor for reading
>>> br/>>>> Sounds like a good plan!!
>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> we have been reviewingg the comments on the pull request, and
started to think about the approach of extracting values from response
directly. During next week, we will work on this and make some updates to
the code with the suggestions you made. We keep you informed
>>>> br/>>>>> thank you, <
>>>> br/>>>>> iñigo <
>>>> br/>>>>> ------------------------------------------ br/>>>>> Iñigo
Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> telf.:
747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo 201-203
br/>>>>> 48950 Erandio (Bizkaia) br/>>>>>
----------------------------------------- <
>>>> br/>>>>> ----- Mensaje originall -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35
>>>> Asunto: AW: AW: Nifi integration record oriented processor for reading
>>>> br/>>>>> Hi Inigo, <
>>>> br/>>>>> especially if you havee a look at the KNX protocol. This
doesn't define the usual IEC datatypes we tried to use for all normal PLC
drivers.
>>>> Here we have hundreds of datatypes that don't match any other
protocol. I think the PLCValue approach would be the simplest.
>>>> The one thing you have to keep in mind, is that you should check a
PLCValue, if it's a list (Array type) or a Structure (Which sort of relates
to komplex types with a more sophisticated structure).
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Iñigo Angulo <ia...@zylk.net.INVALID> br/>>>>> Gesendet:
FFreitag, 23. April 2021 15:34
>>>> An: dev <de...@plc4x.apache.org>
>>>> Betreff: Re: AW: Nifi integration record oriented processor for
reading
>>>> br/>>>>> Hi Otto, Chris, <
>>>> br/>>>>> Yes, I think the approoach you propose will be best. By now,
we are generating the schema ourselves. We have a record writer who is in
charge of reading PLC values. Schema is defined previously to reading the
values. We build this schema getting the protocol from the
'connectionString' (S7, Modbus) and the specified variable type from the
'PLC resource address String' containing the list of variable to read. From
this we deduce the expected Avro datatype when reading, for instance, a
word in S7 or a coil in Modbus. br/>&ggt;>>> br/>>>>> However, as you
mentioned, the oother approach will be much clearer and useful. Ideally,
getting the actual datatype from PLCValue when getting the response.
Regarding this, we tried to keep the previously described 'mapping'
separated from the rest of the code, so that hopefully it can be easily
replaced..
>>>> br/>>>>> We have done the pull rrequest, hope you can take a look at
the code and let us know what you think. We will fill the ICLA document
too.
>>>> br/>>>>> thank you <
>>>> iñigo
>>>> br/>>>>> br/>>>>> br/>>>>> ----------------------------------------- <
>>>> Iñigo Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource <
>>>> telf.: 747412337
>>>> Ribera de Axpe, 11
>>>> Edificio A, modulo 201-203
>>>> 48950 Erandio (Bizkaia)
>>>> -----------------------------------------
>>>> br/>>>>> ----- Mensaje original -----
>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>> Para: "dev" <de...@plc4x.apache.org>
>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49
>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>> br/>>>>> Hi all, <
>>>> br/>>>>> Well, you get PlcValuees from the response that wrap the
different datatypes. So generally you shouldn't care about the detail type.
>>>> br/>>>>> However, you can call ggetObject() which returns the core
value the plc-value has ... so if it's the PLCValue for a Short, getObject
will return a short value.
>>>> br/>>>>> Does that help??
>>>> br/>>>>> Chris <
>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- <
>>>> Von: Otto Fowler <ot...@gmail.com>
>>>> Gesendet: Donnerstag, 22. April 2021 15:21
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>> br/>>>>> So, you are generating the schema yourself, such that
downstream if they inherit schema they will just get what you generate? And
you are trying to do that by the connection string? If so, a different way
I could imagine doing would be to get the ’types’ of the data from the
responses themselves. This would be more generic. The flow I could imagine
( in OnTrigger ):
>>>> br/>>>>> DO READ <
>>>> IF NOT HAS SCHEMA
>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH SCHEMA
>>>> br/>>>>> Maybe Chris can speak tto how to get the types from the
responses.
>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo
<ia...@zylk.net.INVALID> wrote:
>>>>> br/>>>>>> Hi Chris, Otto,,
>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to give
an overview. In Nifi, information packages are called Flowfiles, and these
are the actual units of information that are exchanged between Procesors,
all along the dataflow. Flowfiles have two sections where we can manage
data: Attributes and Content. In the "traditional" Nifi approach, you work
with both sections, extracting information from the Content to the
Attributes and viceversa to perform operations. This approach could have
one limitation when you are processing batch data (lines from a CSV file
for instance), where you need to split each of the lines into different
Flowfiles. Thus, a 1000 line CSV file leads to 1000 Flowfiles to process,
each of them containing a single record.
>>>>> br/>>>>>> On later versioons of the product, they introduced the
Record oriented approach. This approach allows you to manage multiple
records on a single Flowfile's Content, as long as these records have all
the same schema. This means that the operations defined by the Processors
are applied simultaneously to the whole content at once. Following with the
previous example, a 1000 line CSV file could produce a single Flowfile with
a content of 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro,
to serialize thee Flowfile's Content. Then, the Record Oriented Processors
use Writers and Readers to present this information in the desired format
(such as Avro, Json, CSV, etc). Basically, with the record oriented
approach, Nifi introduced multiple new Processors, and also included the
Record version of many of the "old" ones. Using this Record approach, Nifi
perfomance enhances notably, specially when working with large structured
information.
>>>>> br/>>>>>> The work we didd was creating a Record Oriented Processor,
based on the previously existing one Plc4xSourceProcessor, to read values
from the devices. We have also included a README on the
plc4x/plc4j/integrations/apache-nifi module explaining the Processor
configuration and giving an example. Moreover, we put a nifi template with
a dataflow for testing these processors, if useful.
>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor, that
is right. We added the writer capability to the existing
PLC4XSourceProcessor, so that it formats the output to the desired
configuration in a record manner. At the actual implementation, we did this
"protocol adaptation" from the sintax of the particular properties on
Processor's configuration. For example, from connection string
's7://IP:PORT', we extract the S7 idenifier and map variable datatypes to
the actual Avro datatypes for build the record output schema. However, here
we dont have vast experience with PLC4X libraries, and for sure there will
be better ways for doing this.
>>>>> Also about the Base Processor, we were thinking that maybe the best
approach could be to have this Base Processor, and then implement readers
for particular protocols as Controller Services. But here also, it could be
very helpful to have your opinion.
>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any
documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have
defined some naming br/>>>>>> conventions, or expected structure to
faciliitate later work. At the br/>>>>>> present, we have a fork of the
project where we have been working on br/>>>>&ggt;> these Nifi changes. We
updated the content of our fork (fetch/merge
>>>>> upstream) about 2 weeks ago, and commited our changes to the
'develop' br/>>>>>> branch. Do we bettter create a new branch with our
commits? how do you br/>>>&>>> prefer to receive the code? (we are not very
experts on git, just in br/>>>>>> case we could cause some problemss...)
>>>>> br/>>>>>> thank you in addvance
>>>>> br/>>>>>> iñigo <
>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- <
>>>>> Iñigo Angulo
>>>>> br/>>>>>> ZYLK.net :: connsultoría.openSource
>>>>> telf.: 747412337
>>>>> Ribera de Axpe, 11
>>>>> Edificio A, modulo 201-203
>>>>> 48950 Erandio (Bizkaia)
>>>>> -----------------------------------------
>>>>> br/>>>>>> ----- Mensaje ooriginal -----
>>>>> De: "Christofer Dutz" <ch...@c-ware.de>
>>>>> Para: "dev" <de...@plc4x.apache.org>
>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15
>>>>> Asunto: AW: Nifi integration record oriented processor for reading
>>>>> br/>>>>>> The more I thinnk of it,
>>>>> br/>>>>>> Perhaps we shouuld also think of potentially providing some
information on supported configuration options.
>>>>> Wouldn't it be cool if the driver could say: "I generally have these
options and they have these datatypes and mean this"
>>>>> Additionally, the transports could too say: "I generally have these
options and they have these datatypes and mean this"
>>>>> br/>>>>>> I would be our StreamPipes friends would love something
like that? Right?
>>>>> br/>>>>>> Chris <
>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- <
>>>>> Von: Otto Fowler <ot...@gmail.com>
>>>>> Gesendet: Mittwoch, 21. April 2021 17:46
>>>>> An: dev@plc4x.apache.org
>>>>> Betreff: Re: Nifi integration record oriented processor for reading
>>>>> br/>>>>>> Hi Inigo, <
>>>>> br/>>>>>> I’m a coommitter on Apache Nifi as well as PLC4X, I would
be happy to review your processor.
>>>>> If I understand what you are saying correctly, you have a single
processor which supports record writing output?
>>>>> br/>>>>>> plc4x -> reccords
>>>>> br/>>>>>> And that you haave, for configuration purposes for that
processor created support on a per protocol basis for configuration and
validation?
>>>>> br/>>>>>> If there is perr protocol configuration / validation etc,
it may be better to have a base processor, and derived processors per
protocol to handle those differences.
>>>>> br/>>>>>> I look forward to seeing the code.
>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo
<ia...@zylk.net.INVALID> wrote:
>>>>>> br/>>>>>>> Hi all,,
>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache Nifi
integration part of the project. We have created a Record oriented
processor for reading PLC data. It is based on the previous existing
SourceProcessor, but works with records, using a Nifi Writer (such as Avro,
Json, and so on) to write data on flowfiles content. br/>>>>>>> br/>>>>>>>
We updated the code on our fork with thee actual PLC4X git repo about 2
weeks ago, and tested it reading values with S7 from a S7-1200 CPU from
Nifi. Also, one of our customers has recently started to use it for
validation. br/>>>>>>> br/>>>>>>> Currently, it works with S7 and Modbus
oover TCP. This is because we had to write some classes to map
connectionString and variableList properties (sintax) of the processor to
the actual protocol, to be able to build then avro schema for output
flowfile, taking into account variable datatypes, etc. We only did this for
S7 and Modbus. I am sure that there is a better way to do this, so at this
point you maybe could take a look to find the best solution and avoid
needing to do this mapping. br/>>>>>>> br/>>>>>>> If you find this useful,
we could do a ppull request to the main PLC4x repo. Let us know what you
think. br/>>>>>>>> br/>>>>>>> best regards,
>>>>>> iñigo
>>>>>> br/>>>>>>> ------------------------------------------
>>>>>> Iñigo Angulo
>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource
>>>>>> telf.: 747412337
>>>>>> Ribera de Axpe, 11
>>>>>> Edificio A, modulo 201-203
>>>>>> 48950 Erandio (Bizkaia)
>>>>>> -----------------------------------------