You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Rex Fenley <Re...@remind101.com> on 2021/01/21 05:13:58 UTC
A few questions about minibatch
Hi,
Our job was experiencing high write amplification on aggregates so we
decided to give mini-batch a go. There's a few things I've noticed that are
different from our previous job and I would like some clarification.
1) Our operators now say they have Watermarks. We never explicitly added
watermarks, and our state is essentially unbounded across all time since it
consumes from Debezium and reshapes our database data into another store.
Why does it say we have Watermarks then?
2) In our sources I see MiniBatchAssigner(interval=[1000ms],
mode=[ProcTime], what does that do?
3) I don't really see anything else different yet in the shape of our plan
even though we've turned on
configuration.setString(
"table.optimizer.agg-phase-strategy",
"TWO_PHASE"
)
is there a way to check that this optimization is on? We use user defined
aggregate functions, does it work for UDAF?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: A few questions about minibatch
Posted by Rex Fenley <Re...@remind101.com>.
Thanks, that all makes sense!
On Wed, Jan 27, 2021 at 7:00 PM Jark Wu <im...@gmail.com> wrote:
> Hi Rex,
>
> Could you share your query here? It would be helpful to identify the root
> cause if we have the query.
>
> 1) watermark
> The framework automatically adds a node (the MiniBatchAssigner) to
> generate watermark events as the mini-batch id to broadcast and trigger
> mini-batch in the pipeline.
>
> 2) MiniBatchAssigner(interval=[1000ms], mode=[ProcTime]
> It generates a new mini-batch id in an interval of 1000ms in system time.
> The mini-batch id is represented by the watermark event.
>
> 3) TWO_PHASE optimization
> If users want to have TWO_PHASE optimization, it requires the aggregate
> functions all support the merge() method and the mini-batch is enabled.
>
> Best,
> Jark
>
>
>
>
> On Tue, 26 Jan 2021 at 19:01, Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> I am pulling Jark and Godfrey who are more familiar with the planner
>> internals.
>>
>> Best,
>>
>> Dawid
>> On 22/01/2021 20:11, Rex Fenley wrote:
>>
>> Hello,
>>
>> Does anyone have any more information here?
>>
>> Thanks!
>>
>> On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley <Re...@remind101.com> wrote:
>>
>>> Hi,
>>>
>>> Our job was experiencing high write amplification on aggregates so we
>>> decided to give mini-batch a go. There's a few things I've noticed that are
>>> different from our previous job and I would like some clarification.
>>>
>>> 1) Our operators now say they have Watermarks. We never explicitly added
>>> watermarks, and our state is essentially unbounded across all time since it
>>> consumes from Debezium and reshapes our database data into another store.
>>> Why does it say we have Watermarks then?
>>>
>>> 2) In our sources I see MiniBatchAssigner(interval=[1000ms],
>>> mode=[ProcTime], what does that do?
>>>
>>> 3) I don't really see anything else different yet in the shape of our
>>> plan even though we've turned on
>>> configuration.setString(
>>> "table.optimizer.agg-phase-strategy",
>>> "TWO_PHASE"
>>> )
>>> is there a way to check that this optimization is on? We use user
>>> defined aggregate functions, does it work for UDAF?
>>>
>>> Thanks!
>>>
>>> --
>>>
>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>
>>>
>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>> <https://www.facebook.com/remindhq>
>>>
>>
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: A few questions about minibatch
Posted by Jark Wu <im...@gmail.com>.
Hi Rex,
Could you share your query here? It would be helpful to identify the root
cause if we have the query.
1) watermark
The framework automatically adds a node (the MiniBatchAssigner) to generate
watermark events as the mini-batch id to broadcast and trigger mini-batch
in the pipeline.
2) MiniBatchAssigner(interval=[1000ms], mode=[ProcTime]
It generates a new mini-batch id in an interval of 1000ms in system time.
The mini-batch id is represented by the watermark event.
3) TWO_PHASE optimization
If users want to have TWO_PHASE optimization, it requires the aggregate
functions all support the merge() method and the mini-batch is enabled.
Best,
Jark
On Tue, 26 Jan 2021 at 19:01, Dawid Wysakowicz <dw...@apache.org>
wrote:
> I am pulling Jark and Godfrey who are more familiar with the planner
> internals.
>
> Best,
>
> Dawid
> On 22/01/2021 20:11, Rex Fenley wrote:
>
> Hello,
>
> Does anyone have any more information here?
>
> Thanks!
>
> On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley <Re...@remind101.com> wrote:
>
>> Hi,
>>
>> Our job was experiencing high write amplification on aggregates so we
>> decided to give mini-batch a go. There's a few things I've noticed that are
>> different from our previous job and I would like some clarification.
>>
>> 1) Our operators now say they have Watermarks. We never explicitly added
>> watermarks, and our state is essentially unbounded across all time since it
>> consumes from Debezium and reshapes our database data into another store.
>> Why does it say we have Watermarks then?
>>
>> 2) In our sources I see MiniBatchAssigner(interval=[1000ms],
>> mode=[ProcTime], what does that do?
>>
>> 3) I don't really see anything else different yet in the shape of our
>> plan even though we've turned on
>> configuration.setString(
>> "table.optimizer.agg-phase-strategy",
>> "TWO_PHASE"
>> )
>> is there a way to check that this optimization is on? We use user defined
>> aggregate functions, does it work for UDAF?
>>
>> Thanks!
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
>
Re: A few questions about minibatch
Posted by Dawid Wysakowicz <dw...@apache.org>.
I am pulling Jark and Godfrey who are more familiar with the planner
internals.
Best,
Dawid
On 22/01/2021 20:11, Rex Fenley wrote:
> Hello,
>
> Does anyone have any more information here?
>
> Thanks!
>
> On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley <Rex@remind101.com
> <ma...@remind101.com>> wrote:
>
> Hi,
>
> Our job was experiencing high write amplification on aggregates so
> we decided to give mini-batch a go. There's a few things I've
> noticed that are different from our previous job and I would like
> some clarification.
>
> 1) Our operators now say they have Watermarks. We never explicitly
> added watermarks, and our state is essentially unbounded across
> all time since it consumes from Debezium and reshapes our database
> data into another store. Why does it say we have Watermarks then?
>
> 2) In our sources I see MiniBatchAssigner(interval=[1000ms],
> mode=[ProcTime], what does that do?
>
> 3) I don't really see anything else different yet in the shape of
> our plan even though we've turned on
> configuration.setString(
> "table.optimizer.agg-phase-strategy",
> "TWO_PHASE"
> )
> is there a way to check that this optimization is on? We use user
> defined aggregate functions, does it work for UDAF?
>
> Thanks!
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/>| BLOG
> <http://blog.remind.com/> | FOLLOW US
> <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
>
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/>| BLOG
> <http://blog.remind.com/> | FOLLOW US
> <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
Re: A few questions about minibatch
Posted by Rex Fenley <Re...@remind101.com>.
Hello,
Does anyone have any more information here?
Thanks!
On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley <Re...@remind101.com> wrote:
> Hi,
>
> Our job was experiencing high write amplification on aggregates so we
> decided to give mini-batch a go. There's a few things I've noticed that are
> different from our previous job and I would like some clarification.
>
> 1) Our operators now say they have Watermarks. We never explicitly added
> watermarks, and our state is essentially unbounded across all time since it
> consumes from Debezium and reshapes our database data into another store.
> Why does it say we have Watermarks then?
>
> 2) In our sources I see MiniBatchAssigner(interval=[1000ms],
> mode=[ProcTime], what does that do?
>
> 3) I don't really see anything else different yet in the shape of our plan
> even though we've turned on
> configuration.setString(
> "table.optimizer.agg-phase-strategy",
> "TWO_PHASE"
> )
> is there a way to check that this optimization is on? We use user defined
> aggregate functions, does it work for UDAF?
>
> Thanks!
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>