You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by Alessio Bernesco Làvore <al...@gmail.com> on 2021/01/30 12:55:45 UTC
How to use Frequency Counter processors
Hi everyone,
i'm trying to count how many MQTT messages are landing on a topic.
I just created the MQTT connector, if i pass it directly, for example, to
the Notification Sink everything works fine, i see the variables extracted
from the mqtt message.
If i try to insert in the middle a Frequency Counter then i cannot find a
sink able to process the output.
How i'm supposed to use this processor?
Greetings,
Alessio
Re: How to use Frequency Counter processors
Posted by Alessio Bernesco Làvore <al...@gmail.com>.
Yes, of course, i'll keep you posted.
I think in the next day or two i'll connect a real modbus device to do some
field tests.
Greetings,
Alessio
On Sat, Jan 30, 2021 at 4:34 PM Philipp Zehnder <ze...@apache.org> wrote:
> Hi Alessio,
>
> ok, I am happy to hear that the problem is fixed.
> Your use case sounds very interesting!
>
> We also support PLC4X in our connect adapters, but the maturity of the
> implementations for the different protocols is different.
> The adapter for the S7 PLC is already quite mature, but we also have an
> implementation for modbus.
> However, we did not have any real modbus devices when developing it. So it
> would be very helpful to get feedback on the adapter.
> Maybe you can also try it in your setup and give us feedback if it worked
> or if there are any problems with the current version.
>
> It would especially be interesting for us, if the current configurations
> are understandable from a modbus developers point of view.
>
> Philipp
>
> [1]
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/adapters/plc4x
>
> On 30. Jan 2021, at 15:57, Alessio Bernesco Làvore <
> alessio.bernesco@gmail.com> wrote:
>
> Hello Philipp,
> thanks for your prompt response.
>
> I think the source of my problem was some kind of "overlap" of various
> Streampipes versions (0.66, 0.67, compose, cli, etc...).
> I've done a total prune (images, containers, volumes, etc), and switched
> to the cli version 0.67 full, now everything works fine.
>
> Probably the misleading fact was that i've manually installed the
> Frequency Counter over a minimal compose installation and something was
> missed, because the attribute selector once you insert the processor was
> empty, so subsequently it was not able to push the event/message to the
> next step. Once reinstalled the full version, the processor works as
> expected, providing a list of attributes coming from the previous node (an
> mqtt source). So my fault :)
>
> To answer your question about the process, i'm trying to simulate a stream
> where i send an MQTT message with a "device error" status when a light was
> blinking, and then a "device ready" status when the light goes off. The
> blinking status is coming from a remote I/O (Moxa 1212) that reads the
> light, there's then a python script that reads the ModBus and send an mqtt
> "blink" any time the light change its state.
>
> Now that i've solved the frequency counter problem i've two new tasks to
> complete:
> - remove the python modbus part and read the Moxa using PLC4X
> - understand how to keep the last status sent (device error/device ready)
> cause i've to send the MQTT message with the status just once on change.
> Usually i do this kind of things in Flink saving the information inside the
> context state.
>
> Thank you for your support, any insight or hint you would provide is
> welcome!
>
> Greetings,
> Alessio
>
> On Sat, Jan 30, 2021 at 3:29 PM Philipp Zehnder <ze...@apache.org>
> wrote:
>
>> Hi Alessio,
>>
>> welcome to the mailing list and thank you for writing us and reporting
>> the problem.
>>
>> That is a strange behavior and I will look into it.
>> You should always be able to use the dashboard sink. This is also the one
>> I usually use for testing pipelines.
>>
>> Can you please provide me with some more information?
>> Which StreamPipes version are you using and did you install it the
>> “light” (this is the default) or “full" version?
>> What is the exact name of the processor that is not working?
>>
>> Can you also tell me what is the goal you want to achieve?
>> Do you want to get a total count of events of a specific topic, or are
>> you rather interested in a frequency (e.g. Events / second)”?
>>
>> We really appreciate your feedback, so please do not hesitate to contact
>> us if there are any other problems / questions.
>>
>> Philipp
>>
>>
>>
>> > On 30. Jan 2021, at 13:55, Alessio Bernesco Làvore <
>> alessio.bernesco@gmail.com> wrote:
>> >
>> > Hi everyone,
>> >
>> > i'm trying to count how many MQTT messages are landing on a topic.
>> >
>> > I just created the MQTT connector, if i pass it directly, for example,
>> to the Notification Sink everything works fine, i see the variables
>> extracted from the mqtt message.
>> >
>> > If i try to insert in the middle a Frequency Counter then i cannot find
>> a sink able to process the output.
>> >
>> > How i'm supposed to use this processor?
>> >
>> > Greetings,
>> > Alessio
>>
>>
>
Re: How to use Frequency Counter processors
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Alessio,
ok, I am happy to hear that the problem is fixed.
Your use case sounds very interesting!
We also support PLC4X in our connect adapters, but the maturity of the implementations for the different protocols is different.
The adapter for the S7 PLC is already quite mature, but we also have an implementation for modbus.
However, we did not have any real modbus devices when developing it. So it would be very helpful to get feedback on the adapter.
Maybe you can also try it in your setup and give us feedback if it worked or if there are any problems with the current version.
It would especially be interesting for us, if the current configurations are understandable from a modbus developers point of view.
Philipp
[1] https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/src/main/java/org/apache/streampipes/connect/adapters/plc4x
> On 30. Jan 2021, at 15:57, Alessio Bernesco Làvore <al...@gmail.com> wrote:
>
> Hello Philipp,
> thanks for your prompt response.
>
> I think the source of my problem was some kind of "overlap" of various Streampipes versions (0.66, 0.67, compose, cli, etc...).
> I've done a total prune (images, containers, volumes, etc), and switched to the cli version 0.67 full, now everything works fine.
>
> Probably the misleading fact was that i've manually installed the Frequency Counter over a minimal compose installation and something was missed, because the attribute selector once you insert the processor was empty, so subsequently it was not able to push the event/message to the next step. Once reinstalled the full version, the processor works as expected, providing a list of attributes coming from the previous node (an mqtt source). So my fault :)
>
> To answer your question about the process, i'm trying to simulate a stream where i send an MQTT message with a "device error" status when a light was blinking, and then a "device ready" status when the light goes off. The blinking status is coming from a remote I/O (Moxa 1212) that reads the light, there's then a python script that reads the ModBus and send an mqtt "blink" any time the light change its state.
>
> Now that i've solved the frequency counter problem i've two new tasks to complete:
> - remove the python modbus part and read the Moxa using PLC4X
> - understand how to keep the last status sent (device error/device ready) cause i've to send the MQTT message with the status just once on change. Usually i do this kind of things in Flink saving the information inside the context state.
>
> Thank you for your support, any insight or hint you would provide is welcome!
>
> Greetings,
> Alessio
>
> On Sat, Jan 30, 2021 at 3:29 PM Philipp Zehnder <zehnder@apache.org <ma...@apache.org>> wrote:
> Hi Alessio,
>
> welcome to the mailing list and thank you for writing us and reporting the problem.
>
> That is a strange behavior and I will look into it.
> You should always be able to use the dashboard sink. This is also the one I usually use for testing pipelines.
>
> Can you please provide me with some more information?
> Which StreamPipes version are you using and did you install it the “light” (this is the default) or “full" version?
> What is the exact name of the processor that is not working?
>
> Can you also tell me what is the goal you want to achieve?
> Do you want to get a total count of events of a specific topic, or are you rather interested in a frequency (e.g. Events / second)”?
>
> We really appreciate your feedback, so please do not hesitate to contact us if there are any other problems / questions.
>
> Philipp
>
>
>
> > On 30. Jan 2021, at 13:55, Alessio Bernesco Làvore <alessio.bernesco@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi everyone,
> >
> > i'm trying to count how many MQTT messages are landing on a topic.
> >
> > I just created the MQTT connector, if i pass it directly, for example, to the Notification Sink everything works fine, i see the variables extracted from the mqtt message.
> >
> > If i try to insert in the middle a Frequency Counter then i cannot find a sink able to process the output.
> >
> > How i'm supposed to use this processor?
> >
> > Greetings,
> > Alessio
>
Re: How to use Frequency Counter processors
Posted by Alessio Bernesco Làvore <al...@gmail.com>.
Hello Philipp,
thanks for your prompt response.
I think the source of my problem was some kind of "overlap" of various
Streampipes versions (0.66, 0.67, compose, cli, etc...).
I've done a total prune (images, containers, volumes, etc), and switched to
the cli version 0.67 full, now everything works fine.
Probably the misleading fact was that i've manually installed the Frequency
Counter over a minimal compose installation and something was missed,
because the attribute selector once you insert the processor was empty, so
subsequently it was not able to push the event/message to the next step.
Once reinstalled the full version, the processor works as expected,
providing a list of attributes coming from the previous node (an mqtt
source). So my fault :)
To answer your question about the process, i'm trying to simulate a stream
where i send an MQTT message with a "device error" status when a light was
blinking, and then a "device ready" status when the light goes off. The
blinking status is coming from a remote I/O (Moxa 1212) that reads the
light, there's then a python script that reads the ModBus and send an mqtt
"blink" any time the light change its state.
Now that i've solved the frequency counter problem i've two new tasks to
complete:
- remove the python modbus part and read the Moxa using PLC4X
- understand how to keep the last status sent (device error/device ready)
cause i've to send the MQTT message with the status just once on change.
Usually i do this kind of things in Flink saving the information inside the
context state.
Thank you for your support, any insight or hint you would provide is
welcome!
Greetings,
Alessio
On Sat, Jan 30, 2021 at 3:29 PM Philipp Zehnder <ze...@apache.org> wrote:
> Hi Alessio,
>
> welcome to the mailing list and thank you for writing us and reporting the
> problem.
>
> That is a strange behavior and I will look into it.
> You should always be able to use the dashboard sink. This is also the one
> I usually use for testing pipelines.
>
> Can you please provide me with some more information?
> Which StreamPipes version are you using and did you install it the “light”
> (this is the default) or “full" version?
> What is the exact name of the processor that is not working?
>
> Can you also tell me what is the goal you want to achieve?
> Do you want to get a total count of events of a specific topic, or are you
> rather interested in a frequency (e.g. Events / second)”?
>
> We really appreciate your feedback, so please do not hesitate to contact
> us if there are any other problems / questions.
>
> Philipp
>
>
>
> > On 30. Jan 2021, at 13:55, Alessio Bernesco Làvore <
> alessio.bernesco@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > i'm trying to count how many MQTT messages are landing on a topic.
> >
> > I just created the MQTT connector, if i pass it directly, for example,
> to the Notification Sink everything works fine, i see the variables
> extracted from the mqtt message.
> >
> > If i try to insert in the middle a Frequency Counter then i cannot find
> a sink able to process the output.
> >
> > How i'm supposed to use this processor?
> >
> > Greetings,
> > Alessio
>
>
Re: How to use Frequency Counter processors
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Alessio,
welcome to the mailing list and thank you for writing us and reporting the problem.
That is a strange behavior and I will look into it.
You should always be able to use the dashboard sink. This is also the one I usually use for testing pipelines.
Can you please provide me with some more information?
Which StreamPipes version are you using and did you install it the “light” (this is the default) or “full" version?
What is the exact name of the processor that is not working?
Can you also tell me what is the goal you want to achieve?
Do you want to get a total count of events of a specific topic, or are you rather interested in a frequency (e.g. Events / second)”?
We really appreciate your feedback, so please do not hesitate to contact us if there are any other problems / questions.
Philipp
> On 30. Jan 2021, at 13:55, Alessio Bernesco Làvore <al...@gmail.com> wrote:
>
> Hi everyone,
>
> i'm trying to count how many MQTT messages are landing on a topic.
>
> I just created the MQTT connector, if i pass it directly, for example, to the Notification Sink everything works fine, i see the variables extracted from the mqtt message.
>
> If i try to insert in the middle a Frequency Counter then i cannot find a sink able to process the output.
>
> How i'm supposed to use this processor?
>
> Greetings,
> Alessio