You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Unai Leria <un...@zylk.net.INVALID> on 2023/05/02 15:09:18 UTC
[Opinion] NiFi integration Listener
Hi
I have been taking a look into the possibility of adding a PLC4x Listener to the NiFi integration.
Under my research, and following how most of the ListenXXX processors in the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a queue.
* The processor reads the messages from this queue only running when not empty.
In particular ListenTCP and ListenUDP would be the closest ones to the implementation I describe.
In order to account for inevitable disconnections I suggest including an additional queue prior to the queue that connects to the processor. Its function would be to check if the time to the last event is greater than x amount and reconnect in that case.
As a proof of concept I already have a working version on https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener.
I would appreciate your opinion on this feature.
If you like it I can create a PR.
Thanks you for your time!
Unai Lería
Re: AW: [Opinion] NiFi integration Listener
Posted by Otto Fowler <ot...@gmail.com>.
From a Nifi point of view, having possibly many background threads that are
not in the thread pools and governed by the Nifi scheduler is a bad idea, I
*think*.
I would recommend that you take to the [nifi dev mailing list or slack](
https://nifi.apache.org/mailing_lists.html) for further discussion on the
best implementation and have that straight.
Maybe it will get checked off on as well.
I will see things in either place as I am a committer on Nifi as well.
On May 15, 2023 at 11:05:50 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi again,
I think even if the subscriptions would rely on internal polling a NiFi
processor that gives access to subscription operation would be a nice
addition to the integration.
Furthermore we would be covering all operations except browsing (for which
I don't see a use case in NiFi).
I been testing and improving the proof of concept so the main work is
already done. I will create a PR soon
Thanks for the feedback!
Unai
Re: AW: [Opinion] NiFi integration Listener
Posted by Unai Leria <un...@zylk.net.INVALID>.
Hi again,
I think even if the subscriptions would rely on internal polling a NiFi processor that gives access to subscription operation would be a nice addition to the integration.
Furthermore we would be covering all operations except browsing (for which I don't see a use case in NiFi).
I been testing and improving the proof of concept so the main work is already done. I will create a PR soon
Thanks for the feedback!
Unai
AW: [Opinion] NiFi integration Listener
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Unai,
currently only protocoly actively supporting subscriptions implement the Subscription API.
We were planning on adding a transparent polling emulation of subscriptions for protocols, that don’t natively support them, however. But I guess that’s still going to take quite some time till we implement that.
Chris
Von: Unai Leria <un...@zylk.net.INVALID>
Datum: Donnerstag, 4. Mai 2023 um 09:37
An: Otto Fowler <ot...@gmail.com>, dev <de...@plc4x.apache.org>
Betreff: Re: [Opinion] NiFi integration Listener
The idea is that the processor only allows the use of plc4x subscription requests. It would only work on drivers that support subscription.
I have made the assumption that the all plc4x subscriptions implementations are optimized listeners and are not based only on continuous polling.
De: "Otto Fowler" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>, "unai leria" <un...@zylk.net>
Enviados: Miércoles, 3 de Mayo 2023 16:38:08
Asunto: Re: [Opinion] NiFi integration Listener
So this would be a plc4x subscription listener not something polling? I don’t think that is a bad idea overall, as long as it is true
Unsolicited / subscription / Exception based processing and not polling in a background thread.
On May 3, 2023 at 4:36:03 AM, Unai Leria ( [ mailto:unai.leria@zylk.net.invalid | unai.leria@zylk.net.invalid ] ) wrote:
Hi Otto!
The idea was to use the processor to support plc4x subscriptions. Maybe the "listener" name was not the most appropriate one but I was trying to follow the NiFi naming convention.
Sorry for the misunderstanding
Unai
----- Mensaje original -----
De: "Otto Fowler" < [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] >
Para: "dev" < [ mailto:dev@plc4x.apache.org | dev@plc4x.apache.org ] >, "unai leria" < [ mailto:unai.leria@zylk.net | unai.leria@zylk.net ] >
Enviados: Miércoles, 3 de Mayo 2023 3:53:17
Asunto: Re: [Opinion] NiFi integration Listener
On May 2, 2023 at 9:52:08 PM, Otto Fowler ( [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] ) wrote:
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
[ https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener | https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener ] .
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>
Re: [Opinion] NiFi integration Listener
Posted by Łukasz Dywicki <lu...@code-house.org>.
Hello Unai and Otto,
There are certain traps with this approach as not all protocols are
equal in this regard. Currently
PlcConnection.getMetadata().canSubscribe() determines if driver/protocol
itself supports subscriptions, however it does not indicate that
connection as a whole will handle it properly.
Given recent example of S7 updates - S7-1200 does not support
subscriptions or supports it in very limited. S7-1500 will handle wider
pool of areas where you can subscribe. Luckily we can attempt to guess
what sits on other side of connection to dynamically determine if
subscription is possible.
Pulling example further - with BACnet devices (such driver does not
exist yet for Java) you have a single "logical connection" but multiple
devices behind it. Each device behind that connection may or may not
support subscription (change of values).
ADS on other hand I think have no limitations on what to subscribe, thus
any field will work.
There are specific relations between driver, protocol and device and our
ability to tell exactly if plc4x subscriptions will work. Bear in mind
that while you will be working on a listener.
Cheers,
Łukasz
On 4.05.2023 09:37, Unai Leria wrote:
> The idea is that the processor only allows the use of plc4x subscription requests. It would only work on drivers that support subscription.
> I have made the assumption that the all plc4x subscriptions implementations are optimized listeners and are not based only on continuous polling.
>
>
> De: "Otto Fowler" <ot...@gmail.com>
> Para: "dev" <de...@plc4x.apache.org>, "unai leria" <un...@zylk.net>
> Enviados: Miércoles, 3 de Mayo 2023 16:38:08
> Asunto: Re: [Opinion] NiFi integration Listener
>
> So this would be a plc4x subscription listener not something polling? I don’t think that is a bad idea overall, as long as it is true
> Unsolicited / subscription / Exception based processing and not polling in a background thread.
>
>
>
>
>
>
> On May 3, 2023 at 4:36:03 AM, Unai Leria ( [ mailto:unai.leria@zylk.net.invalid | unai.leria@zylk.net.invalid ] ) wrote:
>
>
> Hi Otto!
>
> The idea was to use the processor to support plc4x subscriptions. Maybe the "listener" name was not the most appropriate one but I was trying to follow the NiFi naming convention.
>
> Sorry for the misunderstanding
>
> Unai
>
> ----- Mensaje original -----
> De: "Otto Fowler" < [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] >
> Para: "dev" < [ mailto:dev@plc4x.apache.org | dev@plc4x.apache.org ] >, "unai leria" < [ mailto:unai.leria@zylk.net | unai.leria@zylk.net ] >
> Enviados: Miércoles, 3 de Mayo 2023 3:53:17
> Asunto: Re: [Opinion] NiFi integration Listener
>
> On May 2, 2023 at 9:52:08 PM, Otto Fowler ( [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] ) wrote:
>
> Hi Unai!
>
> So, this listener would listen for exception based or unsolicited messages?
> If it is just polling on a background thread, I don’t think how you want to
> do something in nifi, and we already have “polling” processors.
>
>
>
> On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
> wrote:
>
> Hi br/> <
> I have been taking a look into the possibility of adding a PLC4x Listener
> to the NiFi integration. br/> <
> Under my research, and following how most of the ListenXXX processors in
> the standard-bundle of NiFi are, a good solution would be:
>
>
> * A processor that starts a thread when it is scheduled.
> * That thread connects to the plc and appends the messages received into a
> queue. br/> * The processor reads the messages from thiis queue only
> running when not empty. br/> <
> In particular ListenTCP and ListenUDP would be the closest ones to the
> implementation I describe. br/> <
> In order to account for inevitable disconnections I suggest including an
> additional queue prior to the queue that connects to the processor. Its
> function would be to check if the time to the last event is greater than x
> amount and reconnect in that case. br/> <
> As a proof of concept I already have a working version on
> [ https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener | https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener ] .
> br/> <
> I would appreciate your opinion on this feature. br/>If you like it I can
> create a PR. br/> <
> Thanks you for your time! br/> <
> Unai Lería br/>
>
>
>
>
Re: [Opinion] NiFi integration Listener
Posted by Unai Leria <un...@zylk.net.INVALID>.
The idea is that the processor only allows the use of plc4x subscription requests. It would only work on drivers that support subscription.
I have made the assumption that the all plc4x subscriptions implementations are optimized listeners and are not based only on continuous polling.
De: "Otto Fowler" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>, "unai leria" <un...@zylk.net>
Enviados: Miércoles, 3 de Mayo 2023 16:38:08
Asunto: Re: [Opinion] NiFi integration Listener
So this would be a plc4x subscription listener not something polling? I don’t think that is a bad idea overall, as long as it is true
Unsolicited / subscription / Exception based processing and not polling in a background thread.
On May 3, 2023 at 4:36:03 AM, Unai Leria ( [ mailto:unai.leria@zylk.net.invalid | unai.leria@zylk.net.invalid ] ) wrote:
Hi Otto!
The idea was to use the processor to support plc4x subscriptions. Maybe the "listener" name was not the most appropriate one but I was trying to follow the NiFi naming convention.
Sorry for the misunderstanding
Unai
----- Mensaje original -----
De: "Otto Fowler" < [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] >
Para: "dev" < [ mailto:dev@plc4x.apache.org | dev@plc4x.apache.org ] >, "unai leria" < [ mailto:unai.leria@zylk.net | unai.leria@zylk.net ] >
Enviados: Miércoles, 3 de Mayo 2023 3:53:17
Asunto: Re: [Opinion] NiFi integration Listener
On May 2, 2023 at 9:52:08 PM, Otto Fowler ( [ mailto:ottobackwards@gmail.com | ottobackwards@gmail.com ] ) wrote:
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
[ https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener | https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener ] .
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>
Re: [Opinion] NiFi integration Listener
Posted by Otto Fowler <ot...@gmail.com>.
So this would be a plc4x subscription listener not something polling? I
don’t think that is a bad idea overall, as long as it is true
Unsolicited / subscription / Exception based processing and not polling in
a background thread.
On May 3, 2023 at 4:36:03 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi Otto!
The idea was to use the processor to support plc4x subscriptions. Maybe the
"listener" name was not the most appropriate one but I was trying to follow
the NiFi naming convention.
Sorry for the misunderstanding
Unai
----- Mensaje original -----
De: "Otto Fowler" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>, "unai leria" <un...@zylk.net>
Enviados: Miércoles, 3 de Mayo 2023 3:53:17
Asunto: Re: [Opinion] NiFi integration Listener
On May 2, 2023 at 9:52:08 PM, Otto Fowler (ottobackwards@gmail.com) wrote:
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener.
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>
Re: [Opinion] NiFi integration Listener
Posted by Unai Leria <un...@zylk.net.INVALID>.
Hi Otto!
The idea was to use the processor to support plc4x subscriptions. Maybe the "listener" name was not the most appropriate one but I was trying to follow the NiFi naming convention.
Sorry for the misunderstanding
Unai
----- Mensaje original -----
De: "Otto Fowler" <ot...@gmail.com>
Para: "dev" <de...@plc4x.apache.org>, "unai leria" <un...@zylk.net>
Enviados: Miércoles, 3 de Mayo 2023 3:53:17
Asunto: Re: [Opinion] NiFi integration Listener
On May 2, 2023 at 9:52:08 PM, Otto Fowler (ottobackwards@gmail.com) wrote:
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener.
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>
Re: [Opinion] NiFi integration Listener
Posted by Otto Fowler <ot...@gmail.com>.
On May 2, 2023 at 9:52:08 PM, Otto Fowler (ottobackwards@gmail.com) wrote:
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener.
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>
Re: [Opinion] NiFi integration Listener
Posted by Otto Fowler <ot...@gmail.com>.
Hi Unai!
So, this listener would listen for exception based or unsolicited messages?
If it is just polling on a background thread, I don’t think how you want to
do something in nifi, and we already have “polling” processors.
On May 2, 2023 at 11:09:33 AM, Unai Leria (unai.leria@zylk.net.invalid)
wrote:
Hi br/> <
I have been taking a look into the possibility of adding a PLC4x Listener
to the NiFi integration. br/> <
Under my research, and following how most of the ListenXXX processors in
the standard-bundle of NiFi are, a good solution would be:
* A processor that starts a thread when it is scheduled.
* That thread connects to the plc and appends the messages received into a
queue. br/> * The processor reads the messages from thiis queue only
running when not empty. br/> <
In particular ListenTCP and ListenUDP would be the closest ones to the
implementation I describe. br/> <
In order to account for inevitable disconnections I suggest including an
additional queue prior to the queue that connects to the processor. Its
function would be to check if the time to the last event is greater than x
amount and reconnect in that case. br/> <
As a proof of concept I already have a working version on
https://github.com/zylklab/plc4x/tree/feature/nifi-integration-record-listener.
br/> <
I would appreciate your opinion on this feature. br/>If you like it I can
create a PR. br/> <
Thanks you for your time! br/> <
Unai Lería br/>