[Question] Is message order preserved in windows?

["Balogh, György" <bo...@ultinous.com>]

Hi,

I have video analytic data:
- time
- object type (car, person, etc)
- object bounding box (x,y,width,height)
- track id

I'd like to do transformations where I need to process detections belonging
to the same track in time order.

My input data is ordered by time. I do some filtering first (eg.: filter
cars) than apply sliding windows (eg.: 10 sec window size, with 5 sec step)
and group by track id.

Can we assume anything about the order of events in the windows? Is the
order preserved? I need to process them in time order, if the order is not
preserved what would be the best approach? I can of course collect and sort
the data but it has significant performance and memory impact. (I need to
process events on the billion scale).

Thank you,
Gyorgy


-- 

György Balogh
CEO
E gyorgy.balogh@ultinous.com <zs...@ultinous.com>
M +36 30 270 8342 <+36%2030%20270%208342>
A HU, 1117 Budapest, Budafoki út 209.
W www.ultinous.com

Re: [Question] Is message order preserved in windows?

[Evan Galpin <eg...@apache.org>]

Hi Gyorgy,

To my knowledge, ordering of elements is not guaranteed. To order your
elements in each window by time, you would need to sort the iterables
produced by the groupByKey. You might use SortValues[1] to do that.

[1]
https://beam.apache.org/documentation/sdks/java-extensions/#example-usage-of-sortvalues

Thanks,
Evan


On Tue, May 17, 2022 at 04:35 Balogh, György <bo...@ultinous.com> wrote:

> Hi,
>
> I have video analytic data:
> - time
> - object type (car, person, etc)
> - object bounding box (x,y,width,height)
> - track id
>
> I'd like to do transformations where I need to process detections
> belonging to the same track in time order.
>
> My input data is ordered by time. I do some filtering first (eg.: filter
> cars) than apply sliding windows (eg.: 10 sec window size, with 5 sec step)
> and group by track id.
>
> Can we assume anything about the order of events in the windows? Is the
> order preserved? I need to process them in time order, if the order is not
> preserved what would be the best approach? I can of course collect and sort
> the data but it has significant performance and memory impact. (I need to
> process events on the billion scale).
>
> Thank you,
> Gyorgy
>
>
> --
>
> György Balogh
> CEO
> E gyorgy.balogh@ultinous.com <zs...@ultinous.com>
> M +36 30 270 8342 <+36%2030%20270%208342>
> A HU, 1117 Budapest, Budafoki út 209
> <https://www.google.com/maps/search/HU,+1117+Budapest,+Budafoki+%C3%BAt+209?entry=gmail&source=g>
> .
> W www.ultinous.com
>