You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aissa Elaffani <ai...@gmail.com> on 2020/05/15 06:21:57 UTC
Flink suggestions;
Hello Guys,
I am a beginner in this field of real-time streaming and i am working with
apache flink, and i ignore a lot of features of it, and actually I am
building an application, in which i receive some sensors data in this
format {"status": "Alerte", "classe": " ", "value": {"temperature": 15.7},
"mode": "ON", "equipementID": 1, "date": "2019-03-20 22:00", "sensorID":
9157}, each sensor is installed on an equipment in a workshop in a factory
somewhere. My goal is :
If one sensor of a factory get down (status="alerte"), I want that the
status of all the factory to be Alerte. But as the Stream does not contain
the factory ID, other Static data set source that contain the data of
factories and the sensors that belongs to each one.
So Please guys i want to know the optimized way to do so, and the
aggregation that i need to do!
Sorry for disturbing you, i wish you all the best! And i hope you share
with me the of your experiences!
Best regards,
Aissa
Re: Flink suggestions;
Posted by Arvid Heise <ar...@ververica.com>.
Hi Aissa,
Your use case is quite unusual. You usually have an alert in a dashboard
(potentially sending an email), if any of the sensors show an error. You
usually want to retain the original error code to be able to quickly
identify the issue.
Your use case would make sense if you want to filter the data for all
sensors. But usually, you do it downstream when you aggregate the events.
For example, you want to determine the average temperature; at that point,
you could also aggregate the #failed sensors. Then you can filter on the
#failed field.
Another common pattern is to just ignore the value of the failed sensor in
aggregation. That's achieved by just filtering the record initially.
On Fri, May 15, 2020 at 10:31 AM Chesnay Schepler <ch...@apache.org>
wrote:
> Am I understanding you correctly in that, if one sensor of on factory
> raises an alert, then you want all sensors in that same factory to raise
> an alert?
>
> How big is this dataset that maps sensors to factories?
>
> Maybe you can just load them into a Map<Sensor, Factory> in say a
> FlatMap, enrich the sensor data stream, keyBy the factory id and then do
> your windowing logic (I assume you have some time window)?
> Although this would only work if you want alerts only for sensors in a
> given window.
>
> If you want to have an alert for literally all sensors in a factory then
> you could instead add the FlatMap after the window, and map the sensor
> -> factory -> all_sensors and duplicate the alerts.
>
> On 15/05/2020 08:21, Aissa Elaffani wrote:
> > Hello Guys,
> > I am a beginner in this field of real-time streaming and i am working
> > with apache flink, and i ignore a lot of features of it, and actually
> > I am building an application, in which i receive some sensors data in
> > this format {"status": "Alerte", "classe": " ", "value":
> > {"temperature": 15.7}, "mode": "ON", "equipementID": 1, "date":
> > "2019-03-20 22:00", "sensorID": 9157}, each sensor is installed on an
> > equipment in a workshop in a factory somewhere. My goal is :
> > If one sensor of a factory get down (status="alerte"), I want that the
> > status of all the factory to be Alerte. But as the Stream does not
> > contain the factory ID, other Static data set source that contain the
> > data of factories and the sensors that belongs to each one.
> > So Please guys i want to know the optimized way to do so, and the
> > aggregation that i need to do!
> > Sorry for disturbing you, i wish you all the best! And i hope you
> > share with me the of your experiences!
> > Best regards,
> > Aissa
>
>
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
Re: Flink suggestions;
Posted by Chesnay Schepler <ch...@apache.org>.
Am I understanding you correctly in that, if one sensor of on factory
raises an alert, then you want all sensors in that same factory to raise
an alert?
How big is this dataset that maps sensors to factories?
Maybe you can just load them into a Map<Sensor, Factory> in say a
FlatMap, enrich the sensor data stream, keyBy the factory id and then do
your windowing logic (I assume you have some time window)?
Although this would only work if you want alerts only for sensors in a
given window.
If you want to have an alert for literally all sensors in a factory then
you could instead add the FlatMap after the window, and map the sensor
-> factory -> all_sensors and duplicate the alerts.
On 15/05/2020 08:21, Aissa Elaffani wrote:
> Hello Guys,
> I am a beginner in this field of real-time streaming and i am working
> with apache flink, and i ignore a lot of features of it, and actually
> I am building an application, in which i receive some sensors data in
> this format {"status": "Alerte", "classe": " ", "value":
> {"temperature": 15.7}, "mode": "ON", "equipementID": 1, "date":
> "2019-03-20 22:00", "sensorID": 9157}, each sensor is installed on an
> equipment in a workshop in a factory somewhere. My goal is :
> If one sensor of a factory get down (status="alerte"), I want that the
> status of all the factory to be Alerte. But as the Stream does not
> contain the factory ID, other Static data set source that contain the
> data of factories and the sensors that belongs to each one.
> So Please guys i want to know the optimized way to do so, and the
> aggregation that i need to do!
> Sorry for disturbing you, i wish you all the best! And i hope you
> share with me the of your experiences!
> Best regards,
> Aissa