You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jiahui Jiang <qz...@hotmail.com> on 2020/05/11 15:17:04 UTC
Preserve record orders after WINDOW function
Hello! I'm writing a SQL query with a OVER window function ordered by processing time.
I'm wondering since timestamp is only millisecond granularity.
For a query using over window and sorted on processing time column, for example,
```
SELECT col1,
max(col2) OVER (PARTITION BY col1, ORDER BY _processing_time_column)
FROM table
```
If 2 records have the same processing timestamp (they arrived the operator at the exact same millisecond), is the order guaranteed to be preserved after the window function? If not, what would to the recommended workaround to keep the order?
Thank you!
Jiahui
Re: Preserve record orders after WINDOW function
Posted by Jark Wu <im...@gmail.com>.
Yes, that's right.
On Tue, 12 May 2020 at 10:55, Jiahui Jiang <qz...@hotmail.com> wrote:
> Thank you for confirming!
>
> Just want to make sure my understanding of the internal implementation is
> correct:
>
> When applying an over window and ordered by processing time using SQL, the
> datastream plan it translates into doesn't actually have an order by logic.
> It just sequentially process all the elements that fall into this window in
> their order inside the stream. Is that correct?
>
> Thanks again! 😊
> ------------------------------
> *From:* Jark Wu <im...@gmail.com>
> *Sent:* Monday, May 11, 2020 8:52 PM
> *To:* Jiahui Jiang <qz...@hotmail.com>
> *Cc:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* Re: Preserve record orders after WINDOW function
>
> Hi Jiahui,
>
> Yes, if they arrive at the same millisecond, they are perserved in the
> arriving order.
>
> Best,
> Jark
>
>
>
> On Mon, 11 May 2020 at 23:17, Jiahui Jiang <qz...@hotmail.com>
> wrote:
>
> Hello! I'm writing a SQL query with a OVER window function ordered by
> processing time.
>
> I'm wondering since timestamp is only millisecond granularity.
>
> For a query using over window and sorted on processing time column, for
> example,
> ```
> SELECT col1,
> max(col2) OVER (PARTITION BY col1, ORDER BY _processing_time_column)
> FROM table
> ```
>
> If 2 records have the same processing timestamp (they arrived the operator
> at the exact same millisecond), is the order guaranteed to be preserved
> after the window function? If not, what would to the recommended workaround
> to keep the order?
>
> Thank you!
> Jiahui
>
>
Re: Preserve record orders after WINDOW function
Posted by Jiahui Jiang <qz...@hotmail.com>.
Thank you for confirming!
Just want to make sure my understanding of the internal implementation is correct:
When applying an over window and ordered by processing time using SQL, the datastream plan it translates into doesn't actually have an order by logic. It just sequentially process all the elements that fall into this window in their order inside the stream. Is that correct?
Thanks again! 😊
________________________________
From: Jark Wu <im...@gmail.com>
Sent: Monday, May 11, 2020 8:52 PM
To: Jiahui Jiang <qz...@hotmail.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Preserve record orders after WINDOW function
Hi Jiahui,
Yes, if they arrive at the same millisecond, they are perserved in the arriving order.
Best,
Jark
On Mon, 11 May 2020 at 23:17, Jiahui Jiang <qz...@hotmail.com>> wrote:
Hello! I'm writing a SQL query with a OVER window function ordered by processing time.
I'm wondering since timestamp is only millisecond granularity.
For a query using over window and sorted on processing time column, for example,
```
SELECT col1,
max(col2) OVER (PARTITION BY col1, ORDER BY _processing_time_column)
FROM table
```
If 2 records have the same processing timestamp (they arrived the operator at the exact same millisecond), is the order guaranteed to be preserved after the window function? If not, what would to the recommended workaround to keep the order?
Thank you!
Jiahui
Re: Preserve record orders after WINDOW function
Posted by Jark Wu <im...@gmail.com>.
Hi Jiahui,
Yes, if they arrive at the same millisecond, they are perserved in the
arriving order.
Best,
Jark
On Mon, 11 May 2020 at 23:17, Jiahui Jiang <qz...@hotmail.com> wrote:
> Hello! I'm writing a SQL query with a OVER window function ordered by
> processing time.
>
> I'm wondering since timestamp is only millisecond granularity.
>
> For a query using over window and sorted on processing time column, for
> example,
> ```
> SELECT col1,
> max(col2) OVER (PARTITION BY col1, ORDER BY _processing_time_column)
> FROM table
> ```
>
> If 2 records have the same processing timestamp (they arrived the operator
> at the exact same millisecond), is the order guaranteed to be preserved
> after the window function? If not, what would to the recommended workaround
> to keep the order?
>
> Thank you!
> Jiahui
>