You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Kostas Kloudas <k....@data-artisans.com> on 2018/07/20 20:24:33 UTC

Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

Hi all,

We just merged on the master [FLINK-9753] (commit 66b1f854a0250bdd048808d40f93aa2990476841)
and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e). 

These two commits introduce:
1) a big refactoring to the new StreamingFileSink which allows it to support bulk formats in general (FLINK-9753) and more expressive rolling and bucketing strategies, and 
2) support for Parque, based on the previous refactoring.

Both these features were pretty popular in the mailing list. 

Given that these 2 features are self contained, as they mainly affect a connector,  we would like to cherry pick these commits to the release-1.6 branch. But for this, we would like to have the OK from the community.

Please let me know what you think,
Kostas

https://issues.apache.org/jira/browse/FLINK-9903 <https://issues.apache.org/jira/browse/FLINK-9903>
https://issues.apache.org/jira/browse/FLINK-9753 <https://issues.apache.org/jira/browse/FLINK-9753>

Re: Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

Posted by vino yang <ya...@gmail.com>.

I also see some file sink questions, so +1 for picking the PR about
StreamingFileSink.

Thanks, vino.

2018-07-23 20:53 GMT+08:00 Kostas Kloudas <k....@data-artisans.com>:

> Thanks for the +1’s.
>
> I added the relevant commits to the release-1.6 branch.
>
> Cheers,
> Kostas
>
> > On Jul 22, 2018, at 3:05 PM, Till Rohrmann <tr...@apache.org> wrote:
> >
> > I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
> > because they add more value for the user at a comparable small risk of
> > breaking the system since it is a connector. So +1 for cherry picking
> these
> > two commits.
> >
> > Cheers,
> > Till
> >
> > On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <to...@gmail.com> wrote:
> >
> >> Hi Kostas,
> >>
> >> Nice to see these features merged to the release-1.6 branch. Supporting
> >> Parque format in StreamingFileSink
> >> really makes me eager to apply it to some use cases in our company, but
> I'm
> >> still wondering if the current
> >> StreamingFileSink works well on S3 or I should wait util [FLINK-9752]
> done.
> >>
> >> Anyway, I would still like to see this happen, because it let me have
> more
> >> alternative to upload data with parquet
> >> format. Thanks for your great efforts on this.
> >>
> >> Best Regards,
> >> Tony Wei
> >>
> >> https://issues.apache.org/jira/browse/FLINK-9752
> >>
> >> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <k....@data-artisans.com>:
> >>
> >>> Hi all,
> >>>
> >>> We just merged on the master [FLINK-9753] (commit
> >>> 66b1f854a0250bdd048808d40f93aa2990476841)
> >>> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
> >>>
> >>> These two commits introduce:
> >>> 1) a big refactoring to the new StreamingFileSink which allows it to
> >>> support bulk formats in general (FLINK-9753) and more expressive
> rolling
> >>> and bucketing strategies, and
> >>> 2) support for Parque, based on the previous refactoring.
> >>>
> >>> Both these features were pretty popular in the mailing list.
> >>>
> >>> Given that these 2 features are self contained, as they mainly affect a
> >>> connector,  we would like to cherry pick these commits to the
> release-1.6
> >>> branch. But for this, we would like to have the OK from the community.
> >>>
> >>> Please let me know what you think,
> >>> Kostas
> >>>
> >>> https://issues.apache.org/jira/browse/FLINK-9903 <
> >>> https://issues.apache.org/jira/browse/FLINK-9903>
> >>> https://issues.apache.org/jira/browse/FLINK-9753 <
> >>> https://issues.apache.org/jira/browse/FLINK-9753>
> >>
>
>

Re: Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

Posted by Kostas Kloudas <k....@data-artisans.com>.

Thanks for the +1’s.

I added the relevant commits to the release-1.6 branch.

Cheers,
Kostas

> On Jul 22, 2018, at 3:05 PM, Till Rohrmann <tr...@apache.org> wrote:
> 
> I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
> because they add more value for the user at a comparable small risk of
> breaking the system since it is a connector. So +1 for cherry picking these
> two commits.
> 
> Cheers,
> Till
> 
> On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <to...@gmail.com> wrote:
> 
>> Hi Kostas,
>> 
>> Nice to see these features merged to the release-1.6 branch. Supporting
>> Parque format in StreamingFileSink
>> really makes me eager to apply it to some use cases in our company, but I'm
>> still wondering if the current
>> StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.
>> 
>> Anyway, I would still like to see this happen, because it let me have more
>> alternative to upload data with parquet
>> format. Thanks for your great efforts on this.
>> 
>> Best Regards,
>> Tony Wei
>> 
>> https://issues.apache.org/jira/browse/FLINK-9752
>> 
>> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <k....@data-artisans.com>:
>> 
>>> Hi all,
>>> 
>>> We just merged on the master [FLINK-9753] (commit
>>> 66b1f854a0250bdd048808d40f93aa2990476841)
>>> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
>>> 
>>> These two commits introduce:
>>> 1) a big refactoring to the new StreamingFileSink which allows it to
>>> support bulk formats in general (FLINK-9753) and more expressive rolling
>>> and bucketing strategies, and
>>> 2) support for Parque, based on the previous refactoring.
>>> 
>>> Both these features were pretty popular in the mailing list.
>>> 
>>> Given that these 2 features are self contained, as they mainly affect a
>>> connector,  we would like to cherry pick these commits to the release-1.6
>>> branch. But for this, we would like to have the OK from the community.
>>> 
>>> Please let me know what you think,
>>> Kostas
>>> 
>>> https://issues.apache.org/jira/browse/FLINK-9903 <
>>> https://issues.apache.org/jira/browse/FLINK-9903>
>>> https://issues.apache.org/jira/browse/FLINK-9753 <
>>> https://issues.apache.org/jira/browse/FLINK-9753>
>>

Re: Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

Posted by Till Rohrmann <tr...@apache.org>.

I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
because they add more value for the user at a comparable small risk of
breaking the system since it is a connector. So +1 for cherry picking these
two commits.

Cheers,
Till

On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <to...@gmail.com> wrote:

> Hi Kostas,
>
> Nice to see these features merged to the release-1.6 branch. Supporting
> Parque format in StreamingFileSink
> really makes me eager to apply it to some use cases in our company, but I'm
> still wondering if the current
> StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.
>
> Anyway, I would still like to see this happen, because it let me have more
> alternative to upload data with parquet
> format. Thanks for your great efforts on this.
>
> Best Regards,
> Tony Wei
>
> https://issues.apache.org/jira/browse/FLINK-9752
>
> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <k....@data-artisans.com>:
>
> > Hi all,
> >
> > We just merged on the master [FLINK-9753] (commit
> > 66b1f854a0250bdd048808d40f93aa2990476841)
> > and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
> >
> > These two commits introduce:
> > 1) a big refactoring to the new StreamingFileSink which allows it to
> > support bulk formats in general (FLINK-9753) and more expressive rolling
> > and bucketing strategies, and
> > 2) support for Parque, based on the previous refactoring.
> >
> > Both these features were pretty popular in the mailing list.
> >
> > Given that these 2 features are self contained, as they mainly affect a
> > connector,  we would like to cherry pick these commits to the release-1.6
> > branch. But for this, we would like to have the OK from the community.
> >
> > Please let me know what you think,
> > Kostas
> >
> > https://issues.apache.org/jira/browse/FLINK-9903 <
> > https://issues.apache.org/jira/browse/FLINK-9903>
> > https://issues.apache.org/jira/browse/FLINK-9753 <
> > https://issues.apache.org/jira/browse/FLINK-9753>
>

Re: Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

Posted by Tony Wei <to...@gmail.com>.

Hi Kostas,

Nice to see these features merged to the release-1.6 branch. Supporting
Parque format in StreamingFileSink
really makes me eager to apply it to some use cases in our company, but I'm
still wondering if the current
StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.

Anyway, I would still like to see this happen, because it let me have more
alternative to upload data with parquet
format. Thanks for your great efforts on this.

Best Regards,
Tony Wei

https://issues.apache.org/jira/browse/FLINK-9752

2018-07-21 4:24 GMT+08:00 Kostas Kloudas <k....@data-artisans.com>:

> Hi all,
>
> We just merged on the master [FLINK-9753] (commit
> 66b1f854a0250bdd048808d40f93aa2990476841)
> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
>
> These two commits introduce:
> 1) a big refactoring to the new StreamingFileSink which allows it to
> support bulk formats in general (FLINK-9753) and more expressive rolling
> and bucketing strategies, and
> 2) support for Parque, based on the previous refactoring.
>
> Both these features were pretty popular in the mailing list.
>
> Given that these 2 features are self contained, as they mainly affect a
> connector,  we would like to cherry pick these commits to the release-1.6
> branch. But for this, we would like to have the OK from the community.
>
> Please let me know what you think,
> Kostas
>
> https://issues.apache.org/jira/browse/FLINK-9903 <
> https://issues.apache.org/jira/browse/FLINK-9903>
> https://issues.apache.org/jira/browse/FLINK-9753 <
> https://issues.apache.org/jira/browse/FLINK-9753>