Sensor's Data Aggregation Out of order with EventTime.

[Steve <sd...@gmail.com>]

Hi all,
 
We have some sensors that sends data into kafka. Each kafka partition have a
set of deferent sensor writing data in it. We consume the data from flink.
We want to SUM up the values in half an hour intervals in eventTime(extract
from data).
The result is a keyed stream by sensor_id with timeWindow of 30 minutes.

Situation:
Usually when you have to deal with Sensor data you have a priori accept that
your data will be ordered by timestamp for each sensor id. A characteristic
example of data arriving is described below. 

Problem:
The problem is that a watermark generated is closing the windows before all
nodes have finished(7record).


<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n9361/flinkExample.png> 


Question:
Is this scenario possible with EventTime? Or we need to choose another
technique? We thought that it could be done by using one kafka partition for
each sensor, but this would result in thousand partitions in kafka, which
may be inefficient. Could you propose a possible solution for these kind of
data arriving?




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Sensor-s-Data-Aggregation-Out-of-order-with-EventTime-tp9361.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Sensor's Data Aggregation Out of order with EventTime.

[Maximilian Michels <mx...@apache.org>]

This is a common problem in Event Time which is referred to as late
data. You can a) change the Watermark generation code 2) Allow
elements to be late and re-trigger a window execution.

For 2) see https://ci.apache.org/projects/flink/flink-docs-master/dev/windows.html#dealing-with-late-data

-Max


On Thu, Oct 6, 2016 at 10:53 AM, Steve <sd...@gmail.com> wrote:
> Hi all,
>
> We have some sensors that sends data into kafka. Each kafka partition have a
> set of deferent sensor writing data in it. We consume the data from flink.
> We want to SUM up the values in half an hour intervals in eventTime(extract
> from data).
> The result is a keyed stream by sensor_id with timeWindow of 30 minutes.
>
> Situation:
> Usually when you have to deal with Sensor data you have a priori accept that
> your data will be ordered by timestamp for each sensor id. A characteristic
> example of data arriving is described below.
>
> Problem:
> The problem is that a watermark generated is closing the windows before all
> nodes have finished(7record).
>
>
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n9361/flinkExample.png>
>
>
> Question:
> Is this scenario possible with EventTime? Or we need to choose another
> technique? We thought that it could be done by using one kafka partition for
> each sensor, but this would result in thousand partitions in kafka, which
> may be inefficient. Could you propose a possible solution for these kind of
> data arriving?
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Sensor-s-Data-Aggregation-Out-of-order-with-EventTime-tp9361.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.