You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Qing Lim <q....@mwam.com> on 2022/11/02 17:32:35 UTC
Does reduce function on keyed window gives any guarantee on the order of elements?
Hi, Flink User Group
I am trying to use Reduce function, I wonder does it guarantee order when its called?
The signature is as follow:
T reduce(T value1, T value2) throws Exception;
Does it guarantee that it will be called in the same order of elements in the stream, where value2 is always 1 element after value1?
Kind regards
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail: q.lim@mwam.com<ma...@mwam.com> | Tel: +44 207 925 4865
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. (“MWNA”), which is registered with the US Securities and Exchange Commission (“SEC”) as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.
RE: Does reduce function on keyed window gives any guarantee on the order of elements?
Posted by Qing Lim <q....@mwam.com>.
That’s my understanding as well, thanks for your confirmation.
From: Yanfei Lei <fr...@gmail.com>
Sent: 04 November 2022 16:03
To: Qing Lim <q....@mwam.com>
Cc: User <us...@flink.apache.org>
Subject: Re: Does reduce function on keyed window gives any guarantee on the order of elements?
Hi Qing,
> am I right to think that there will be 1 reduce function per key, and they will never overlap?
I agree with this, please correct me if I'm wrong. For a certain key, it will be divided into a certain key group range, and thus sent to a certain sub-task(reduce function instance). A key is only processed by one sub-task of an operator, so the subtasks of one key don't overlap. But in turn, a subtask will process multiple keys.
For example: if we have many pairs like: (k1,v11), ... (k2,v2) ... (k1,v12) ...(k3,v3) ... (k1,v13),... v12 is always processed after v11.
Best,
Yanfei
Qing Lim <q....@mwam.com>> 于2022年11月3日周四 16:17写道：
Hi Yanfei
Thanks for the explanation.
If I use reduce in the context of keyed stream with window, am I right to think that there will be 1 reduce function per key, and they will never overlap? Each reduce function instance will only receive elements from the same key in order.
From: Yanfei Lei <fr...@gmail.com>>
Sent: 03 November 2022 03:06
To: Qing Lim <q....@mwam.com>>
Cc: User <us...@flink.apache.org>>
Subject: Re: Does reduce function on keyed window gives any guarantee on the order of elements?
Hi Qing,
> Does it guarantee that it will be called in the same order of elements in the stream, where value2 is always 1 element after value1?
Order is maintained within each parallel stream partition. If the reduce operator only has one sending- sub-task, the answer is YES, but if reduce operator has multiple sending- sub-task, order among elements is only preserved for each pair of sending- and receiving tasks.
The answers under https://stackoverflow.com/questions/38354713/ordering-of-records-in-stream might help.
Best,
Yanfei
Qing Lim <q....@mwam.com>> 于2022年11月3日周四 01:32写道：
Hi, Flink User Group
I am trying to use Reduce function, I wonder does it guarantee order when its called?
The signature is as follow:
T reduce(T value1, T value2) throws Exception;
Does it guarantee that it will be called in the same order of elements in the stream, where value2 is always 1 element after value1?
Kind regards
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail: q.lim@mwam.com<ma...@mwam.com> | Tel: +44 207 925 4865
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.
--
Best,
Yanfei
Re: Does reduce function on keyed window gives any guarantee on the order of elements?
Posted by Yanfei Lei <fr...@gmail.com>.
Hi Qing,
> am I right to think that there will be 1 reduce function per key, and
they will never overlap?
I agree with this, please correct me if I'm wrong. For a certain key, it
will be divided into a certain key group range, and thus sent to a certain
sub-task(reduce function instance). A key is only processed by one sub-task
of an operator, so the subtasks of one key don't overlap. But in turn, a
subtask will process multiple keys.
For example: if we have many pairs like: (k1,v11), ... (k2,v2) ... (k1,v12)
...(k3,v3) ... (k1,v13),... v12 is always processed after v11.
Best,
Yanfei
Qing Lim <q....@mwam.com> 于2022年11月3日周四 16:17写道：
> Hi Yanfei
>
> Thanks for the explanation.
>
>
>
> If I use reduce in the context of keyed stream with window, am I right to
> think that there will be 1 reduce function per key, and they will never
> overlap? Each reduce function instance will only receive elements from the
> same key in order.
>
>
>
> *From:* Yanfei Lei <fr...@gmail.com>
> *Sent:* 03 November 2022 03:06
> *To:* Qing Lim <q....@mwam.com>
> *Cc:* User <us...@flink.apache.org>
> *Subject:* Re: Does reduce function on keyed window gives any guarantee
> on the order of elements?
>
>
>
> Hi Qing,
>
> > Does it guarantee that it will be called in the same order of elements
> in the stream, where value2 is always 1 element after value1?
>
>
> Order is maintained within each parallel stream partition. If the reduce
> operator only has one sending- sub-task, the answer is YES, but if reduce
> operator has multiple sending- sub-task, order among elements is only
> preserved for each pair of sending- and receiving tasks.
>
> The answers under
> https://stackoverflow.com/questions/38354713/ordering-of-records-in-stream might
> help.
>
> Best,
>
> Yanfei
>
>
>
> Qing Lim <q....@mwam.com> 于2022年11月3日周四 01:32写道：
>
> Hi, Flink User Group
>
>
>
> I am trying to use Reduce function, I wonder does it guarantee order when
> its called?
>
>
>
> The signature is as follow:
>
>
>
> T reduce(T value1, T value2) throws Exception;
>
>
>
> Does it guarantee that it will be called in the same order of elements in
> the stream, where value2 is always 1 element after value1?
>
>
>
> Kind regards
>
>
>
>
>
> *Qing Lim *| Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail:
> q.lim@mwam.com | Tel: +44 207 925 4865
>
>
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser. Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>
>
>
>
--
Best,
Yanfei
RE: Does reduce function on keyed window gives any guarantee on the order of elements?
Posted by Qing Lim <q....@mwam.com>.
Hi Yanfei
Thanks for the explanation.
If I use reduce in the context of keyed stream with window, am I right to think that there will be 1 reduce function per key, and they will never overlap? Each reduce function instance will only receive elements from the same key in order.
From: Yanfei Lei <fr...@gmail.com>
Sent: 03 November 2022 03:06
To: Qing Lim <q....@mwam.com>
Cc: User <us...@flink.apache.org>
Subject: Re: Does reduce function on keyed window gives any guarantee on the order of elements?
Hi Qing,
> Does it guarantee that it will be called in the same order of elements in the stream, where value2 is always 1 element after value1?
Order is maintained within each parallel stream partition. If the reduce operator only has one sending- sub-task, the answer is YES, but if reduce operator has multiple sending- sub-task, order among elements is only preserved for each pair of sending- and receiving tasks.
The answers under https://stackoverflow.com/questions/38354713/ordering-of-records-in-stream might help.
Best,
Yanfei
Qing Lim <q....@mwam.com>> 于2022年11月3日周四 01:32写道：
Hi, Flink User Group
I am trying to use Reduce function, I wonder does it guarantee order when its called?
The signature is as follow:
T reduce(T value1, T value2) throws Exception;
Does it guarantee that it will be called in the same order of elements in the stream, where value2 is always 1 element after value1?
Kind regards
Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail: q.lim@mwam.com<ma...@mwam.com> | Tel: +44 207 925 4865
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.
Re: Does reduce function on keyed window gives any guarantee on the order of elements?
Posted by Yanfei Lei <fr...@gmail.com>.
Hi Qing,
> Does it guarantee that it will be called in the same order of elements in
the stream, where value2 is always 1 element after value1?
Order is maintained within each parallel stream partition. If the reduce
operator only has one sending- sub-task, the answer is YES, but if reduce
operator has multiple sending- sub-task, order among elements is only
preserved for each pair of sending- and receiving tasks.
The answers under
https://stackoverflow.com/questions/38354713/ordering-of-records-in-stream
might
help.
Best,
Yanfei
Qing Lim <q....@mwam.com> 于2022年11月3日周四 01:32写道：
> Hi, Flink User Group
>
>
>
> I am trying to use Reduce function, I wonder does it guarantee order when
> its called?
>
>
>
> The signature is as follow:
>
>
>
> T reduce(T value1, T value2) throws Exception;
>
>
>
> Does it guarantee that it will be called in the same order of elements in
> the stream, where value2 is always 1 element after value1?
>
>
>
> Kind regards
>
>
>
>
>
> *Qing Lim *| Marshall Wace LLP, George House, 131 Sloane Street, London | E-mail:
> q.lim@mwam.com | Tel: +44 207 925 4865
>
>
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser. Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>