You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by David Anderson <da...@apache.org> on 2021/08/19 13:56:45 UTC

Were Bundles meant to be internal?

Most of the table/bundle related classes such as
MapBundleFunction, MapBundleOperator, and CountBundleTrigger aren't marked
as either @Internal or @Public. What was the intention?

I ask because I'm starting to see some interest in using them for
implementing pre-aggregation via the DataStream API, and I'm not convinced
this is a good idea. E.g., see
https://stackoverflow.com/questions/68811184/pre-shuffle-aggregation-in-flink
.

David

Re: Were Bundles meant to be internal?

Posted by Jingsong Li <ji...@gmail.com>.
Hi David,

According to my understanding, all classes that are not marked are
internal classes, and they have no guarantee of compatibility.

You'd better copy them for your own use.

(Implementing pre-aggregation via the DataStream API using
MapBundleFunctions is a good idea)

Best,
Jingsong

On Fri, Aug 20, 2021 at 12:29 AM JING ZHANG <be...@gmail.com> wrote:
>
> Hi David,
> Although the original intention of introducing `MapBundleFunction`,
> `MapBundleOperator`, and `CountBundleTrigger` is to do MiniBatch
> optimization for Flink SQL,
> those classes could also be used in DataStream Job.
> I think it's ok to do local-aggregation optimization based on those classed
> in you case.
>
> Best,
> JING ZHANG
>
> David Anderson <da...@apache.org> 于2021年8月19日周四 下午10:15写道:
>
> > Most of the table/bundle related classes such as
> > MapBundleFunction, MapBundleOperator, and CountBundleTrigger aren't marked
> > as either @Internal or @Public. What was the intention?
> >
> > I ask because I'm starting to see some interest in using them for
> > implementing pre-aggregation via the DataStream API, and I'm not convinced
> > this is a good idea. E.g., see
> >
> > https://stackoverflow.com/questions/68811184/pre-shuffle-aggregation-in-flink
> > .
> >
> > David
> >



-- 
Best, Jingsong Lee

Re: Were Bundles meant to be internal?

Posted by JING ZHANG <be...@gmail.com>.
Hi David,
Although the original intention of introducing `MapBundleFunction`,
`MapBundleOperator`, and `CountBundleTrigger` is to do MiniBatch
optimization for Flink SQL,
those classes could also be used in DataStream Job.
I think it's ok to do local-aggregation optimization based on those classed
in you case.

Best,
JING ZHANG

David Anderson <da...@apache.org> 于2021年8月19日周四 下午10:15写道:

> Most of the table/bundle related classes such as
> MapBundleFunction, MapBundleOperator, and CountBundleTrigger aren't marked
> as either @Internal or @Public. What was the intention?
>
> I ask because I'm starting to see some interest in using them for
> implementing pre-aggregation via the DataStream API, and I'm not convinced
> this is a good idea. E.g., see
>
> https://stackoverflow.com/questions/68811184/pre-shuffle-aggregation-in-flink
> .
>
> David
>