You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2018/01/31 09:38:28 UTC
[DISCUSS] Releasing Flink 1.5.0
Hi Everyone,
When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.
We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.
With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.
What do you think?
Best,
Aljoscha
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Fabian Hueske <fh...@gmail.com>.
A user reported a regression from 1.4.2.
A batch job with a DeltaIteration gets stuck when being executed in a
LocalEnvironment [1].
Cheers, Fabian
[1] https://issues.apache.org/jira/browse/FLINK-9242
2018-04-23 21:09 GMT+02:00 Shuyi Chen <su...@gmail.com>:
> Hi Aljoscha and Till,
>
> I've added 2 PRs to fix and harden the YARN kerberos security for flip-6. I
> think they should go in for 1.5.0 (particularly FLINK-8286
> <https://issues.apache.org/jira/browse/FLINK-8286>).
>
> 1) https://github.com/apache/flink/pull/5896 (FLINK-8286
> <https://issues.apache.org/jira/browse/FLINK-8286>): fixed broken kerberos
> configuration for Yarn TaskExecutor.
> 2) https://github.com/apache/flink/pull/5901 (FLINK-9235
> <https://issues.apache.org/jira/browse/FLINK-9235>): Added integration
> test
> to test YARN security for both legacy and new mode.
>
> Thanks a lot.
> Shuyi
>
> On Thu, Apr 12, 2018 at 9:17 AM, Christophe Jolif <cj...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > A small patch: https://github.com/apache/flink/pull/5789 (JIRA:
> > https://issues.apache.org/jira/browse/FLINK-9103) was issued to help
> with
> > SSL certificates in Kubernetes deployment where you don't control your
> IPs.
> > As this is very small and helpful (at least to me and Edward who issued
> the
> > fix), I was wondering if that could be considered for 1.5?
> >
> > Thanks,
> > --
> > Christophe
> >
> >
> > On Mon, Mar 12, 2018 at 12:42 PM, Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Hi Pavel,
> > >
> > > currently, it is extremely difficult to say when it will happen since
> > Flink
> > > 1.5 includes some very big changes which need thorough testing.
> Depending
> > > on that and what else the community finds on the way, it may go faster
> or
> > > slower. Personally, I hope to finish the release until end of
> > > March/beginning of April.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Mar 8, 2018 at 7:28 PM, Pavel Ciorba <pa...@gmail.com>
> wrote:
> > >
> > > > Approximately when is the release of Flink 1.5 planned?
> > > >
> > > > Best,
> > > >
> > > > 2018-03-01 11:20 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> > > >
> > > > > Thanks for bringing this issue up Shashank. I think Aljoscha is
> > taking
> > > a
> > > > > look at the issue. It looks like a serious bug which we should
> > > definitely
> > > > > fix. What I've heard so far is that it's not so trivial.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <shashank734@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Can we have
> > > > > > https://issues.apache.org/jira/browse/FLINK-7756
> > > > > > <https://issues.apache.org/jira/browse/FLINK-7756> solved in
> > this
> > > > > > version.
> > > > > > Cause unable to use checkpointing with CEP and RocksDB backend.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.
> > > > > nabble.com/
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> "So you have to trust that the dots will somehow connect in your future."
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Shuyi Chen <su...@gmail.com>.
Hi Aljoscha and Till,
I've added 2 PRs to fix and harden the YARN kerberos security for flip-6. I
think they should go in for 1.5.0 (particularly FLINK-8286
<https://issues.apache.org/jira/browse/FLINK-8286>).
1) https://github.com/apache/flink/pull/5896 (FLINK-8286
<https://issues.apache.org/jira/browse/FLINK-8286>): fixed broken kerberos
configuration for Yarn TaskExecutor.
2) https://github.com/apache/flink/pull/5901 (FLINK-9235
<https://issues.apache.org/jira/browse/FLINK-9235>): Added integration test
to test YARN security for both legacy and new mode.
Thanks a lot.
Shuyi
On Thu, Apr 12, 2018 at 9:17 AM, Christophe Jolif <cj...@gmail.com> wrote:
> Hi all,
>
> A small patch: https://github.com/apache/flink/pull/5789 (JIRA:
> https://issues.apache.org/jira/browse/FLINK-9103) was issued to help with
> SSL certificates in Kubernetes deployment where you don't control your IPs.
> As this is very small and helpful (at least to me and Edward who issued the
> fix), I was wondering if that could be considered for 1.5?
>
> Thanks,
> --
> Christophe
>
>
> On Mon, Mar 12, 2018 at 12:42 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Hi Pavel,
> >
> > currently, it is extremely difficult to say when it will happen since
> Flink
> > 1.5 includes some very big changes which need thorough testing. Depending
> > on that and what else the community finds on the way, it may go faster or
> > slower. Personally, I hope to finish the release until end of
> > March/beginning of April.
> >
> > Cheers,
> > Till
> >
> > On Thu, Mar 8, 2018 at 7:28 PM, Pavel Ciorba <pa...@gmail.com> wrote:
> >
> > > Approximately when is the release of Flink 1.5 planned?
> > >
> > > Best,
> > >
> > > 2018-03-01 11:20 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> > >
> > > > Thanks for bringing this issue up Shashank. I think Aljoscha is
> taking
> > a
> > > > look at the issue. It looks like a serious bug which we should
> > definitely
> > > > fix. What I've heard so far is that it's not so trivial.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <sh...@gmail.com>
> > > wrote:
> > > >
> > > > > Can we have
> > > > > https://issues.apache.org/jira/browse/FLINK-7756
> > > > > <https://issues.apache.org/jira/browse/FLINK-7756> solved in
> this
> > > > > version.
> > > > > Cause unable to use checkpointing with CEP and RocksDB backend.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.
> > > > nabble.com/
> > > > >
> > > >
> > >
> >
>
--
"So you have to trust that the dots will somehow connect in your future."
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Christophe Jolif <cj...@gmail.com>.
Hi all,
A small patch: https://github.com/apache/flink/pull/5789 (JIRA:
https://issues.apache.org/jira/browse/FLINK-9103) was issued to help with
SSL certificates in Kubernetes deployment where you don't control your IPs.
As this is very small and helpful (at least to me and Edward who issued the
fix), I was wondering if that could be considered for 1.5?
Thanks,
--
Christophe
On Mon, Mar 12, 2018 at 12:42 PM, Till Rohrmann <tr...@apache.org>
wrote:
> Hi Pavel,
>
> currently, it is extremely difficult to say when it will happen since Flink
> 1.5 includes some very big changes which need thorough testing. Depending
> on that and what else the community finds on the way, it may go faster or
> slower. Personally, I hope to finish the release until end of
> March/beginning of April.
>
> Cheers,
> Till
>
> On Thu, Mar 8, 2018 at 7:28 PM, Pavel Ciorba <pa...@gmail.com> wrote:
>
> > Approximately when is the release of Flink 1.5 planned?
> >
> > Best,
> >
> > 2018-03-01 11:20 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> >
> > > Thanks for bringing this issue up Shashank. I think Aljoscha is taking
> a
> > > look at the issue. It looks like a serious bug which we should
> definitely
> > > fix. What I've heard so far is that it's not so trivial.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <sh...@gmail.com>
> > wrote:
> > >
> > > > Can we have
> > > > https://issues.apache.org/jira/browse/FLINK-7756
> > > > <https://issues.apache.org/jira/browse/FLINK-7756> solved in this
> > > > version.
> > > > Cause unable to use checkpointing with CEP and RocksDB backend.
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.
> > > nabble.com/
> > > >
> > >
> >
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
Hi Pavel,
currently, it is extremely difficult to say when it will happen since Flink
1.5 includes some very big changes which need thorough testing. Depending
on that and what else the community finds on the way, it may go faster or
slower. Personally, I hope to finish the release until end of
March/beginning of April.
Cheers,
Till
On Thu, Mar 8, 2018 at 7:28 PM, Pavel Ciorba <pa...@gmail.com> wrote:
> Approximately when is the release of Flink 1.5 planned?
>
> Best,
>
> 2018-03-01 11:20 GMT+02:00 Till Rohrmann <tr...@apache.org>:
>
> > Thanks for bringing this issue up Shashank. I think Aljoscha is taking a
> > look at the issue. It looks like a serious bug which we should definitely
> > fix. What I've heard so far is that it's not so trivial.
> >
> > Cheers,
> > Till
> >
> > On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <sh...@gmail.com>
> wrote:
> >
> > > Can we have
> > > https://issues.apache.org/jira/browse/FLINK-7756
> > > <https://issues.apache.org/jira/browse/FLINK-7756> solved in this
> > > version.
> > > Cause unable to use checkpointing with CEP and RocksDB backend.
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.
> > nabble.com/
> > >
> >
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Pavel Ciorba <pa...@gmail.com>.
Approximately when is the release of Flink 1.5 planned?
Best,
2018-03-01 11:20 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> Thanks for bringing this issue up Shashank. I think Aljoscha is taking a
> look at the issue. It looks like a serious bug which we should definitely
> fix. What I've heard so far is that it's not so trivial.
>
> Cheers,
> Till
>
> On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <sh...@gmail.com> wrote:
>
> > Can we have
> > https://issues.apache.org/jira/browse/FLINK-7756
> > <https://issues.apache.org/jira/browse/FLINK-7756> solved in this
> > version.
> > Cause unable to use checkpointing with CEP and RocksDB backend.
> >
> >
> >
> > --
> > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/
> >
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
Thanks for bringing this issue up Shashank. I think Aljoscha is taking a
look at the issue. It looks like a serious bug which we should definitely
fix. What I've heard so far is that it's not so trivial.
Cheers,
Till
On Thu, Mar 1, 2018 at 9:56 AM, shashank734 <sh...@gmail.com> wrote:
> Can we have
> https://issues.apache.org/jira/browse/FLINK-7756
> <https://issues.apache.org/jira/browse/FLINK-7756> solved in this
> version.
> Cause unable to use checkpointing with CEP and RocksDB backend.
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by shashank734 <sh...@gmail.com>.
Can we have
https://issues.apache.org/jira/browse/FLINK-7756
<https://issues.apache.org/jira/browse/FLINK-7756> solved in this version.
Cause unable to use checkpointing with CEP and RocksDB backend.
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Aljoscha Krettek <al...@apache.org>.
@Bowen Yes, we're currently looking at getting this in because, as you said, future changes would break the API.
> On 27. Feb 2018, at 15:15, Till Rohrmann <tr...@apache.org> wrote:
>
> The release branch is now created [1]. Please be aware that we should only commit bug fixes to this branch henceforth.
>
> @Bowen, let's wait what Aljoscha says concerning FLINK-8667. If he agrees, then we can still merge it into the release branch.
>
> [1] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5 <https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5>
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 8:03 PM, Bowen Li <bowenli86@gmail.com <ma...@gmail.com>> wrote:
> Hi guys,
>
> Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into 1.5.0? I've been waiting for it for quite a few days.
>
> The reason I really want to get it into 1.5.0 is because KeyedBroadcastProcessFunction will be released in 1.5.0 and there's no compatibility issues. If we don't get FLINK-8667 in, it will be much harder to make it work in post-1.5.0 branches and migrate user apps.
>
> Thanks,
> Bowen
>
>
>
> On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> wrote:
> +1 for cutting the release branch now.
>
> I'm also happy to volunteer as the release manager for Flink 1.5.
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>
> wrote:
>
> > End of last week was the day where we wanted to to the cut of the release
> > branch. There are still a bunch of open blocker issues about bugs in our
> > Jira: [1]. So I'm wondering whether we should actually cut the branch now
> > because some commits would have to be merged on release-1.4, release-1.5,
> > and master. What do you think?
> >
> > Regarding managing the release: I will have two weeks in mid march where I
> > won't have time and this could be the hot release phase. I think that
> > because of this, it would be better to have a release manager other than me
> > and I think Till would be a good candidate for that since he's the lead of
> > FLIP-6 which is the prime new feature of the next release. What do you
> > think about that?
> >
> > [1] https://issues.apache.org/jira/issues/?jql=project%20% <https://issues.apache.org/jira/issues/?jql=project%20%>
> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
> >
> > > On 13. Feb 2018, at 10:00, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> wrote:
> > >
> > > +1 from my side.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <piotr@data-artisans.com <ma...@data-artisans.com>
> > >
> > > wrote:
> > >
> > >> +0.95 from my side.
> > >>
> > >> Network changes are mostly reviewed and should be merged by the end of
> > >> this week.
> > >>
> > >> Piotrek
> > >>
> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
> > >>>
> > >>> I agree with the basic idea. I think there is no need to call it "soft
> > >>> feature freeze" though - it is a feature freeze (no new features get
> > >>> merged) ;-)
> > >>>
> > >>> What you are suggesting is to delay forking of the release-1.5 branch
> > to
> > >>> avoid applying the bug fixes to too many branches. That makes sense.
> > >>> In effect, there is a period of one week (next week) where no post 1.5
> > >>> features can be merged (feature freeze but release branch not forked),
> > >>> which should be okay.
> > >>>
> > >>> Best,
> > >>> Stephan
> > >>>
> > >>>
> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> > >> k.kloudas@data-artisans.com <ma...@data-artisans.com>
> > >>>> wrote:
> > >>>
> > >>>> For me as well +1.
> > >>>>
> > >>>> Cheers,
> > >>>> Kostas
> > >>>>
> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <twalthr@apache.org <ma...@apache.org>>
> > wrote:
> > >>>>>
> > >>>>> Sounds good to me. +1 from my side.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Timo
> > >>>>>
> > >>>>>
> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> > were
> > >>>> we agree to not merge new features to master after that and then to
> > the
> > >>>> actual hard cutting of the release branch a while later.
> > >>>>>>
> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> > soft
> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> > the
> > >>>> release branch?
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Aljoscha
> > >>>>>>
> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> Local state recovery is almost completely done. Only some reviews
> > and
> > >>>>>>> merging of the final PRs is pending.
> > >>>>>>>
> > >>>>>>> The network stack improvements are on a good way to be finished by
> > >> the
> > >>>> end
> > >>>>>>> of this week or beginning of next week. To my knowledge we got
> > >> recently
> > >>>>>>> green Travis builds :-) The network stack changes will also include
> > >> the
> > >>>>>>> application level flow control and the back pressure based
> > checkpoint
> > >>>>>>> alignment. So only the last reviews and merging is missing.
> > >>>>>>>
> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> > >> default.
> > >>>>>>> There are still some smaller things left to be done but I'm
> > confident
> > >>>> that
> > >>>>>>> we can resolve them quickly.
> > >>>>>>>
> > >>>>>>> I agree that due to the big changes we should have a very thorough
> > >> and
> > >>>>>>> principled testing period where we put Flink through the paces.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> > >> chesnay@apache.org <ma...@apache.org>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> > >>>> local
> > >>>>>>>> state recovery) are nearly done.
> > >>>>>>>>
> > >>>>>>>> I'm unsure about local state recovery, but I still see open issues
> > >> for
> > >>>>>>>> FLIP-6 and the network stack rework.
> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
> > >>>>>>>>
> > >>>>>>>> Given the large scope of these features I would very much prefer
> > to
> > >>>> have
> > >>>>>>>> them active on master for a while before a feature-freeze
> > >>>>>>>> to expose them to a wider audience.
> > >>>>>>>>
> > >>>>>>>> IMO it will take at least another month before we can start the
> > >>>> release
> > >>>>>>>> process for 1.5, i.e. the feature freeze.
> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> > to
> > >>>> settle)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>
> > >>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> <
> > >>>>>>>>> https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230>> for that
> > >>>>>>>>> and there are some pending issues related to scala api and
> > >>>> documentation.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Kostas
> > >>>>>>>>>
> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <twalthr@apache.org <ma...@apache.org>>
> > >> wrote:
> > >>>>>>>>>> Hi Shuyi,
> > >>>>>>>>>>
> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> > will
> > >> be
> > >>>>>>>>>> part of 1.5.0.
> > >>>>>>>>>>
> > >>>>>>>>>> Regards,
> > >>>>>>>>>> Timo
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> > lot
> > >> of
> > >>>>>>>>>>> internal users waiting for this feature.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>
> > >]
> > >>>> Support
> > >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> > >> type
> > >>>>>>>>>>> column
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot
> > >>>>>>>>>>> Shuyi
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> > >> cjolif@gmail.com <ma...@gmail.com>
> > >>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi guys,
> > >>>>>>>>>>>> Sorry for jumping in, but I think
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> > >>>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not
> > compatible
> > >>>> with
> > >>>>>>>>>>>> Elasticsearch 5.2+ client
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> > >>>> someone
> > >>>>>>>>>>>> else
> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> > that
> > >>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>> great!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> > >> twalthr@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>> it would be great if we can include the first version of the
> > >> SQL
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> > >>>> week. I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status.
> > It
> > >>>> is far
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> away
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
> > >>>>>>>>>>>>> beginners.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add some
> > >>>> table
> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> > but
> > >>>> this is
> > >>>>>>>>>>>>> optional because a user can implement own table factories at
> > >> the
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> begining.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>> Timo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for starting the discussion.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
> > improvements
> > >>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> > >>>> users have
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> been
> > >>>>>>>>>>>>> asking for them:
> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> > >> timestamp
> > >>>> to
> > >>>>>>>>>>>>>> set
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> up
> > >>>>>>>>>>>>> start offset
> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> > >>>> should
> > >>>>>>>>>>>>>> consider idle partitions
> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> > >>>>>>>>>>>>>> FlinkKinesisConsumer
> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> > >>>> FlinkKafkaConsumer
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
> > FLINK-5479
> > >> is
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>> lacking a pull request.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> aljoscha@apache.org <ma...@apache.org>)
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>> Hi Everyone,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> > >>>> that to
> > >>>>>>>>>>>>>> get
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>> stable release out before putting in a couple of new
> > features.
> > >>>> Back
> > >>>>>>>>>>>>> then,
> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> > >> local
> > >>>> state
> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> > >>>> 1.5.0
> > >>>>>>>>>>>>>> development cycle to allow for those features to become
> > ready
> > >>>> and
> > >>>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> do
> > >>>>>>>>>>>>> the next release.
> > >>>>>>>>>>>>>> We are now approaching the approximate time where we wanted
> > to
> > >>>> do the
> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> > and
> > >>>> gather
> > >>>>>>>>>>>>>> opinions on how we should proceed now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> > >>>> manager for
> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> > >>>>>>>>>>>>>> interested in
> > >>>>>>>>>>>>>> doing that.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What do you think?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
The release branch is now created [1]. Please be aware that we should only
commit bug fixes to this branch henceforth.
@Bowen, let's wait what Aljoscha says concerning FLINK-8667. If he agrees,
then we can still merge it into the release branch.
[1]
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5
Cheers,
Till
On Mon, Feb 26, 2018 at 8:03 PM, Bowen Li <bo...@gmail.com> wrote:
> Hi guys,
>
> Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into
> 1.5.0? I've been waiting for it for quite a few days.
>
> The reason I really want to get it into 1.5.0 is because
> KeyedBroadcastProcessFunction will be released in 1.5.0 and there's no
> compatibility issues. If we don't get FLINK-8667 in, it will be much harder
> to make it work in post-1.5.0 branches and migrate user apps.
>
> Thanks,
> Bowen
>
>
>
> On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> +1 for cutting the release branch now.
>>
>> I'm also happy to volunteer as the release manager for Flink 1.5.
>>
>> Cheers,
>> Till
>>
>> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>> > End of last week was the day where we wanted to to the cut of the
>> release
>> > branch. There are still a bunch of open blocker issues about bugs in our
>> > Jira: [1]. So I'm wondering whether we should actually cut the branch
>> now
>> > because some commits would have to be merged on release-1.4,
>> release-1.5,
>> > and master. What do you think?
>> >
>> > Regarding managing the release: I will have two weeks in mid march
>> where I
>> > won't have time and this could be the hot release phase. I think that
>> > because of this, it would be better to have a release manager other
>> than me
>> > and I think Till would be a good candidate for that since he's the lead
>> of
>> > FLIP-6 which is the prime new feature of the next release. What do you
>> > think about that?
>> >
>> > [1] https://issues.apache.org/jira/issues/?jql=project%20%
>> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
>> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
>> >
>> > > On 13. Feb 2018, at 10:00, Till Rohrmann <tr...@apache.org>
>> wrote:
>> > >
>> > > +1 from my side.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <
>> piotr@data-artisans.com
>> > >
>> > > wrote:
>> > >
>> > >> +0.95 from my side.
>> > >>
>> > >> Network changes are mostly reviewed and should be merged by the end
>> of
>> > >> this week.
>> > >>
>> > >> Piotrek
>> > >>
>> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
>> > >>>
>> > >>> I agree with the basic idea. I think there is no need to call it
>> "soft
>> > >>> feature freeze" though - it is a feature freeze (no new features get
>> > >>> merged) ;-)
>> > >>>
>> > >>> What you are suggesting is to delay forking of the release-1.5
>> branch
>> > to
>> > >>> avoid applying the bug fixes to too many branches. That makes sense.
>> > >>> In effect, there is a period of one week (next week) where no post
>> 1.5
>> > >>> features can be merged (feature freeze but release branch not
>> forked),
>> > >>> which should be okay.
>> > >>>
>> > >>> Best,
>> > >>> Stephan
>> > >>>
>> > >>>
>> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
>> > >> k.kloudas@data-artisans.com
>> > >>>> wrote:
>> > >>>
>> > >>>> For me as well +1.
>> > >>>>
>> > >>>> Cheers,
>> > >>>> Kostas
>> > >>>>
>> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org>
>> > wrote:
>> > >>>>>
>> > >>>>> Sounds good to me. +1 from my side.
>> > >>>>>
>> > >>>>> Regards,
>> > >>>>> Timo
>> > >>>>>
>> > >>>>>
>> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
>> > were
>> > >>>> we agree to not merge new features to master after that and then to
>> > the
>> > >>>> actual hard cutting of the release branch a while later.
>> > >>>>>>
>> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
>> > soft
>> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
>> > the
>> > >>>> release branch?
>> > >>>>>>
>> > >>>>>> What do you think?
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Aljoscha
>> > >>>>>>
>> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org>
>> > >> wrote:
>> > >>>>>>>
>> > >>>>>>> Local state recovery is almost completely done. Only some
>> reviews
>> > and
>> > >>>>>>> merging of the final PRs is pending.
>> > >>>>>>>
>> > >>>>>>> The network stack improvements are on a good way to be finished
>> by
>> > >> the
>> > >>>> end
>> > >>>>>>> of this week or beginning of next week. To my knowledge we got
>> > >> recently
>> > >>>>>>> green Travis builds :-) The network stack changes will also
>> include
>> > >> the
>> > >>>>>>> application level flow control and the back pressure based
>> > checkpoint
>> > >>>>>>> alignment. So only the last reviews and merging is missing.
>> > >>>>>>>
>> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
>> > >> default.
>> > >>>>>>> There are still some smaller things left to be done but I'm
>> > confident
>> > >>>> that
>> > >>>>>>> we can resolve them quickly.
>> > >>>>>>>
>> > >>>>>>> I agree that due to the big changes we should have a very
>> thorough
>> > >> and
>> > >>>>>>> principled testing period where we put Flink through the paces.
>> > >>>>>>>
>> > >>>>>>> Cheers,
>> > >>>>>>> Till
>> > >>>>>>>
>> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
>> > >> chesnay@apache.org>
>> > >>>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on
>> the
>> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack
>> changes,
>> > >>>> local
>> > >>>>>>>> state recovery) are nearly done.
>> > >>>>>>>>
>> > >>>>>>>> I'm unsure about local state recovery, but I still see open
>> issues
>> > >> for
>> > >>>>>>>> FLIP-6 and the network stack rework.
>> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
>> > >>>>>>>>
>> > >>>>>>>> Given the large scope of these features I would very much
>> prefer
>> > to
>> > >>>> have
>> > >>>>>>>> them active on master for a while before a feature-freeze
>> > >>>>>>>> to expose them to a wider audience.
>> > >>>>>>>>
>> > >>>>>>>> IMO it will take at least another month before we can start the
>> > >>>> release
>> > >>>>>>>> process for 1.5, i.e. the feature freeze.
>> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the
>> dust
>> > to
>> > >>>> settle)
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>
>> > >>>>>>>>> I believe that support for Broadcast State should also be in
>> 1.5.
>> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230
>> <
>> > >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
>> > >>>>>>>>> and there are some pending issues related to scala api and
>> > >>>> documentation.
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks,
>> > >>>>>>>>> Kostas
>> > >>>>>>>>>
>> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org>
>> > >> wrote:
>> > >>>>>>>>>> Hi Shuyi,
>> > >>>>>>>>>>
>> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
>> > will
>> > >> be
>> > >>>>>>>>>> part of 1.5.0.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Regards,
>> > >>>>>>>>>> Timo
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
>> > lot
>> > >> of
>> > >>>>>>>>>>> internal users waiting for this feature.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jir
>> a/browse/FLINK-7923
>> > >]
>> > >>>> Support
>> > >>>>>>>>>>> accessing subfields of a Composite element in an Object
>> Array
>> > >> type
>> > >>>>>>>>>>> column
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Thanks a lot
>> > >>>>>>>>>>> Shuyi
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
>> > >> cjolif@gmail.com
>> > >>>>>
>> > >>>>>>>>>>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Hi guys,
>> > >>>>>>>>>>>> Sorry for jumping in, but I think
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>> > >>>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not
>> > compatible
>> > >>>> with
>> > >>>>>>>>>>>> Elasticsearch 5.2+ client
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> have long been awaited and there was one PR from me and
>> from
>> > >>>> someone
>> > >>>>>>>>>>>> else
>> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
>> > that
>> > >>>> would
>> > >>>>>>>>>>>> be
>> > >>>>>>>>>>>> great!
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Thanks!
>> > >>>>>>>>>>>> --
>> > >>>>>>>>>>>> Christophe
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
>> > >> twalthr@apache.org>
>> > >>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>>>>> it would be great if we can include the first version of
>> the
>> > >> SQL
>> > >>>>>>>>>>>>> client
>> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR
>> this
>> > >>>> week. I
>> > >>>>>>>>>>>>> think
>> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha"
>> status.
>> > It
>> > >>>> is far
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>> away
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> from feature completeness but will be a great tool for
>> Flink
>> > >>>>>>>>>>>>> beginners.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add
>> some
>> > >>>> table
>> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
>> > but
>> > >>>> this is
>> > >>>>>>>>>>>>> optional because a user can implement own table factories
>> at
>> > >> the
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>> begining.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> Regards,
>> > >>>>>>>>>>>>> Timo
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks for starting the discussion.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
>> > improvements
>> > >>>> that
>> > >>>>>>>>>>>>>> we
>> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a
>> few
>> > >>>> users have
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> been
>> > >>>>>>>>>>>>> asking for them:
>> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
>> > >> timestamp
>> > >>>> to
>> > >>>>>>>>>>>>>> set
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> up
>> > >>>>>>>>>>>>> start offset
>> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in
>> FlinkKafkaConsumer
>> > >>>> should
>> > >>>>>>>>>>>>>> consider idle partitions
>> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>> > >>>>>>>>>>>>>> FlinkKinesisConsumer
>> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>> > >>>> FlinkKafkaConsumer
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
>> > FLINK-5479
>> > >> is
>> > >>>>>>>>>>>>>> still
>> > >>>>>>>>>>>>>> lacking a pull request.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Cheers,
>> > >>>>>>>>>>>>>> Gordon
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> aljoscha@apache.org)
>> > >>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>> Hi Everyone,
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we
>> did
>> > >>>> that to
>> > >>>>>>>>>>>>>> get
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> a
>> > >>>>>>>>>>>>> stable release out before putting in a couple of new
>> > features.
>> > >>>> Back
>> > >>>>>>>>>>>>> then,
>> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
>> > >> local
>> > >>>> state
>> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a
>> shortened
>> > >>>> 1.5.0
>> > >>>>>>>>>>>>>> development cycle to allow for those features to become
>> > ready
>> > >>>> and
>> > >>>>>>>>>>>>>> then
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> do
>> > >>>>>>>>>>>>> the next release.
>> > >>>>>>>>>>>>>> We are now approaching the approximate time where we
>> wanted
>> > to
>> > >>>> do the
>> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
>> > and
>> > >>>> gather
>> > >>>>>>>>>>>>>> opinions on how we should proceed now.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>> > >>>> manager for
>> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would
>> be
>> > >>>>>>>>>>>>>> interested in
>> > >>>>>>>>>>>>>> doing that.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> What do you think?
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>> Aljoscha
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> --
>> > >>>>>>>>>>>> Christophe
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>
>> > >>>>
>> > >>>>
>> > >>
>> > >>
>> >
>> >
>>
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Bowen Li <bo...@gmail.com>.
Hi guys,
Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into
1.5.0? I've been waiting for it for quite a few days.
The reason I really want to get it into 1.5.0 is because KeyedBroadcastProce
ssFunction will be released in 1.5.0 and there's no compatibility issues.
If we don't get FLINK-8667 in, it will be much harder to make it work in
post-1.5.0 branches and migrate user apps.
Thanks,
Bowen
On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <tr...@apache.org> wrote:
> +1 for cutting the release branch now.
>
> I'm also happy to volunteer as the release manager for Flink 1.5.
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > End of last week was the day where we wanted to to the cut of the release
> > branch. There are still a bunch of open blocker issues about bugs in our
> > Jira: [1]. So I'm wondering whether we should actually cut the branch now
> > because some commits would have to be merged on release-1.4, release-1.5,
> > and master. What do you think?
> >
> > Regarding managing the release: I will have two weeks in mid march where
> I
> > won't have time and this could be the hot release phase. I think that
> > because of this, it would be better to have a release manager other than
> me
> > and I think Till would be a good candidate for that since he's the lead
> of
> > FLIP-6 which is the prime new feature of the next release. What do you
> > think about that?
> >
> > [1] https://issues.apache.org/jira/issues/?jql=project%20%
> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
> >
> > > On 13. Feb 2018, at 10:00, Till Rohrmann <tr...@apache.org> wrote:
> > >
> > > +1 from my side.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <
> piotr@data-artisans.com
> > >
> > > wrote:
> > >
> > >> +0.95 from my side.
> > >>
> > >> Network changes are mostly reviewed and should be merged by the end of
> > >> this week.
> > >>
> > >> Piotrek
> > >>
> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
> > >>>
> > >>> I agree with the basic idea. I think there is no need to call it
> "soft
> > >>> feature freeze" though - it is a feature freeze (no new features get
> > >>> merged) ;-)
> > >>>
> > >>> What you are suggesting is to delay forking of the release-1.5 branch
> > to
> > >>> avoid applying the bug fixes to too many branches. That makes sense.
> > >>> In effect, there is a period of one week (next week) where no post
> 1.5
> > >>> features can be merged (feature freeze but release branch not
> forked),
> > >>> which should be okay.
> > >>>
> > >>> Best,
> > >>> Stephan
> > >>>
> > >>>
> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> > >> k.kloudas@data-artisans.com
> > >>>> wrote:
> > >>>
> > >>>> For me as well +1.
> > >>>>
> > >>>> Cheers,
> > >>>> Kostas
> > >>>>
> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org>
> > wrote:
> > >>>>>
> > >>>>> Sounds good to me. +1 from my side.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Timo
> > >>>>>
> > >>>>>
> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> > were
> > >>>> we agree to not merge new features to master after that and then to
> > the
> > >>>> actual hard cutting of the release branch a while later.
> > >>>>>>
> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> > soft
> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> > the
> > >>>> release branch?
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Aljoscha
> > >>>>>>
> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> Local state recovery is almost completely done. Only some reviews
> > and
> > >>>>>>> merging of the final PRs is pending.
> > >>>>>>>
> > >>>>>>> The network stack improvements are on a good way to be finished
> by
> > >> the
> > >>>> end
> > >>>>>>> of this week or beginning of next week. To my knowledge we got
> > >> recently
> > >>>>>>> green Travis builds :-) The network stack changes will also
> include
> > >> the
> > >>>>>>> application level flow control and the back pressure based
> > checkpoint
> > >>>>>>> alignment. So only the last reviews and merging is missing.
> > >>>>>>>
> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> > >> default.
> > >>>>>>> There are still some smaller things left to be done but I'm
> > confident
> > >>>> that
> > >>>>>>> we can resolve them quickly.
> > >>>>>>>
> > >>>>>>> I agree that due to the big changes we should have a very
> thorough
> > >> and
> > >>>>>>> principled testing period where we put Flink through the paces.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> > >> chesnay@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack
> changes,
> > >>>> local
> > >>>>>>>> state recovery) are nearly done.
> > >>>>>>>>
> > >>>>>>>> I'm unsure about local state recovery, but I still see open
> issues
> > >> for
> > >>>>>>>> FLIP-6 and the network stack rework.
> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
> > >>>>>>>>
> > >>>>>>>> Given the large scope of these features I would very much prefer
> > to
> > >>>> have
> > >>>>>>>> them active on master for a while before a feature-freeze
> > >>>>>>>> to expose them to a wider audience.
> > >>>>>>>>
> > >>>>>>>> IMO it will take at least another month before we can start the
> > >>>> release
> > >>>>>>>> process for 1.5, i.e. the feature freeze.
> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> > to
> > >>>> settle)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>
> > >>>>>>>>> I believe that support for Broadcast State should also be in
> 1.5.
> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230
> <
> > >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
> > >>>>>>>>> and there are some pending issues related to scala api and
> > >>>> documentation.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Kostas
> > >>>>>>>>>
> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org>
> > >> wrote:
> > >>>>>>>>>> Hi Shuyi,
> > >>>>>>>>>>
> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> > will
> > >> be
> > >>>>>>>>>> part of 1.5.0.
> > >>>>>>>>>>
> > >>>>>>>>>> Regards,
> > >>>>>>>>>> Timo
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> > lot
> > >> of
> > >>>>>>>>>>> internal users waiting for this feature.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/
> jira/browse/FLINK-7923
> > >]
> > >>>> Support
> > >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> > >> type
> > >>>>>>>>>>> column
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot
> > >>>>>>>>>>> Shuyi
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> > >> cjolif@gmail.com
> > >>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi guys,
> > >>>>>>>>>>>> Sorry for jumping in, but I think
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> > >>>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not
> > compatible
> > >>>> with
> > >>>>>>>>>>>> Elasticsearch 5.2+ client
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> > >>>> someone
> > >>>>>>>>>>>> else
> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> > that
> > >>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>> great!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> > >> twalthr@apache.org>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>> it would be great if we can include the first version of
> the
> > >> SQL
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> > >>>> week. I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha"
> status.
> > It
> > >>>> is far
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> away
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> from feature completeness but will be a great tool for
> Flink
> > >>>>>>>>>>>>> beginners.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add
> some
> > >>>> table
> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> > but
> > >>>> this is
> > >>>>>>>>>>>>> optional because a user can implement own table factories
> at
> > >> the
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> begining.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>> Timo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for starting the discussion.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
> > improvements
> > >>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> > >>>> users have
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> been
> > >>>>>>>>>>>>> asking for them:
> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> > >> timestamp
> > >>>> to
> > >>>>>>>>>>>>>> set
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> up
> > >>>>>>>>>>>>> start offset
> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in
> FlinkKafkaConsumer
> > >>>> should
> > >>>>>>>>>>>>>> consider idle partitions
> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> > >>>>>>>>>>>>>> FlinkKinesisConsumer
> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> > >>>> FlinkKafkaConsumer
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
> > FLINK-5479
> > >> is
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>> lacking a pull request.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> aljoscha@apache.org)
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>> Hi Everyone,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we
> did
> > >>>> that to
> > >>>>>>>>>>>>>> get
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>> stable release out before putting in a couple of new
> > features.
> > >>>> Back
> > >>>>>>>>>>>>> then,
> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> > >> local
> > >>>> state
> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a
> shortened
> > >>>> 1.5.0
> > >>>>>>>>>>>>>> development cycle to allow for those features to become
> > ready
> > >>>> and
> > >>>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> do
> > >>>>>>>>>>>>> the next release.
> > >>>>>>>>>>>>>> We are now approaching the approximate time where we
> wanted
> > to
> > >>>> do the
> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> > and
> > >>>> gather
> > >>>>>>>>>>>>>> opinions on how we should proceed now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> > >>>> manager for
> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> > >>>>>>>>>>>>>> interested in
> > >>>>>>>>>>>>>> doing that.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What do you think?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
+1 for cutting the release branch now.
I'm also happy to volunteer as the release manager for Flink 1.5.
Cheers,
Till
On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> End of last week was the day where we wanted to to the cut of the release
> branch. There are still a bunch of open blocker issues about bugs in our
> Jira: [1]. So I'm wondering whether we should actually cut the branch now
> because some commits would have to be merged on release-1.4, release-1.5,
> and master. What do you think?
>
> Regarding managing the release: I will have two weeks in mid march where I
> won't have time and this could be the hot release phase. I think that
> because of this, it would be better to have a release manager other than me
> and I think Till would be a good candidate for that since he's the lead of
> FLIP-6 which is the prime new feature of the next release. What do you
> think about that?
>
> [1] https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
>
> > On 13. Feb 2018, at 10:00, Till Rohrmann <tr...@apache.org> wrote:
> >
> > +1 from my side.
> >
> > Cheers,
> > Till
> >
> > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <piotr@data-artisans.com
> >
> > wrote:
> >
> >> +0.95 from my side.
> >>
> >> Network changes are mostly reviewed and should be merged by the end of
> >> this week.
> >>
> >> Piotrek
> >>
> >>> On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
> >>>
> >>> I agree with the basic idea. I think there is no need to call it "soft
> >>> feature freeze" though - it is a feature freeze (no new features get
> >>> merged) ;-)
> >>>
> >>> What you are suggesting is to delay forking of the release-1.5 branch
> to
> >>> avoid applying the bug fixes to too many branches. That makes sense.
> >>> In effect, there is a period of one week (next week) where no post 1.5
> >>> features can be merged (feature freeze but release branch not forked),
> >>> which should be okay.
> >>>
> >>> Best,
> >>> Stephan
> >>>
> >>>
> >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> >> k.kloudas@data-artisans.com
> >>>> wrote:
> >>>
> >>>> For me as well +1.
> >>>>
> >>>> Cheers,
> >>>> Kostas
> >>>>
> >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org>
> wrote:
> >>>>>
> >>>>> Sounds good to me. +1 from my side.
> >>>>>
> >>>>> Regards,
> >>>>> Timo
> >>>>>
> >>>>>
> >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> were
> >>>> we agree to not merge new features to master after that and then to
> the
> >>>> actual hard cutting of the release branch a while later.
> >>>>>>
> >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> soft
> >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> the
> >>>> release branch?
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> --
> >>>>>> Aljoscha
> >>>>>>
> >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> Local state recovery is almost completely done. Only some reviews
> and
> >>>>>>> merging of the final PRs is pending.
> >>>>>>>
> >>>>>>> The network stack improvements are on a good way to be finished by
> >> the
> >>>> end
> >>>>>>> of this week or beginning of next week. To my knowledge we got
> >> recently
> >>>>>>> green Travis builds :-) The network stack changes will also include
> >> the
> >>>>>>> application level flow control and the back pressure based
> checkpoint
> >>>>>>> alignment. So only the last reviews and merging is missing.
> >>>>>>>
> >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> >> default.
> >>>>>>> There are still some smaller things left to be done but I'm
> confident
> >>>> that
> >>>>>>> we can resolve them quickly.
> >>>>>>>
> >>>>>>> I agree that due to the big changes we should have a very thorough
> >> and
> >>>>>>> principled testing period where we put Flink through the paces.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>>
> >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> >> chesnay@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> >>>> local
> >>>>>>>> state recovery) are nearly done.
> >>>>>>>>
> >>>>>>>> I'm unsure about local state recovery, but I still see open issues
> >> for
> >>>>>>>> FLIP-6 and the network stack rework.
> >>>>>>>> As such it doesn't make sense to release 1.5 now.
> >>>>>>>>
> >>>>>>>> Given the large scope of these features I would very much prefer
> to
> >>>> have
> >>>>>>>> them active on master for a while before a feature-freeze
> >>>>>>>> to expose them to a wider audience.
> >>>>>>>>
> >>>>>>>> IMO it will take at least another month before we can start the
> >>>> release
> >>>>>>>> process for 1.5, i.e. the feature freeze.
> >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> to
> >>>> settle)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>>>>>
> >>>>>>>>> Hi Aljoscha,
> >>>>>>>>>
> >>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>>>>>> and there are some pending issues related to scala api and
> >>>> documentation.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Kostas
> >>>>>>>>>
> >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org>
> >> wrote:
> >>>>>>>>>> Hi Shuyi,
> >>>>>>>>>>
> >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> will
> >> be
> >>>>>>>>>> part of 1.5.0.
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Timo
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> lot
> >> of
> >>>>>>>>>>> internal users waiting for this feature.
> >>>>>>>>>>>
> >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923
> >]
> >>>> Support
> >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> >> type
> >>>>>>>>>>> column
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks a lot
> >>>>>>>>>>> Shuyi
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> >> cjolif@gmail.com
> >>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi guys,
> >>>>>>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>>>>>
> >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not
> compatible
> >>>> with
> >>>>>>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>>>>>
> >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> >>>> someone
> >>>>>>>>>>>> else
> >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> that
> >>>> would
> >>>>>>>>>>>> be
> >>>>>>>>>>>> great!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Christophe
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> >> twalthr@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>>> it would be great if we can include the first version of the
> >> SQL
> >>>>>>>>>>>>> client
> >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> >>>> week. I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status.
> It
> >>>> is far
> >>>>>>>>>>>>>
> >>>>>>>>>>>> away
> >>>>>>>>>>>>
> >>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>>>>>> beginners.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In order to use the SQL client we would need to also add some
> >>>> table
> >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> but
> >>>> this is
> >>>>>>>>>>>>> optional because a user can implement own table factories at
> >> the
> >>>>>>>>>>>>>
> >>>>>>>>>>>> begining.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Timo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think there’s a few connector related must-have
> improvements
> >>>> that
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> >>>> users have
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> been
> >>>>>>>>>>>>> asking for them:
> >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> >> timestamp
> >>>> to
> >>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> up
> >>>>>>>>>>>>> start offset
> >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> >>>> should
> >>>>>>>>>>>>>> consider idle partitions
> >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> >>>> FlinkKafkaConsumer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> These are still missing in the master branch. Only
> FLINK-5479
> >> is
> >>>>>>>>>>>>>> still
> >>>>>>>>>>>>>> lacking a pull request.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> Gordon
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> aljoscha@apache.org)
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> >>>> that to
> >>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>> stable release out before putting in a couple of new
> features.
> >>>> Back
> >>>>>>>>>>>>> then,
> >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> >> local
> >>>> state
> >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> >>>> 1.5.0
> >>>>>>>>>>>>>> development cycle to allow for those features to become
> ready
> >>>> and
> >>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> do
> >>>>>>>>>>>>> the next release.
> >>>>>>>>>>>>>> We are now approaching the approximate time where we wanted
> to
> >>>> do the
> >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> and
> >>>> gather
> >>>>>>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> >>>> manager for
> >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>>>>>> interested in
> >>>>>>>>>>>>>> doing that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Aljoscha
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>> Christophe
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Aljoscha Krettek <al...@apache.org>.
End of last week was the day where we wanted to to the cut of the release branch. There are still a bunch of open blocker issues about bugs in our Jira: [1]. So I'm wondering whether we should actually cut the branch now because some commits would have to be merged on release-1.4, release-1.5, and master. What do you think?
Regarding managing the release: I will have two weeks in mid march where I won't have time and this could be the hot release phase. I think that because of this, it would be better to have a release manager other than me and I think Till would be a good candidate for that since he's the lead of FLIP-6 which is the prime new feature of the next release. What do you think about that?
[1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20and%20priority%20%3D%20blocker%20and%20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
> On 13. Feb 2018, at 10:00, Till Rohrmann <tr...@apache.org> wrote:
>
> +1 from my side.
>
> Cheers,
> Till
>
> On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <pi...@data-artisans.com>
> wrote:
>
>> +0.95 from my side.
>>
>> Network changes are mostly reviewed and should be merged by the end of
>> this week.
>>
>> Piotrek
>>
>>> On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
>>>
>>> I agree with the basic idea. I think there is no need to call it "soft
>>> feature freeze" though - it is a feature freeze (no new features get
>>> merged) ;-)
>>>
>>> What you are suggesting is to delay forking of the release-1.5 branch to
>>> avoid applying the bug fixes to too many branches. That makes sense.
>>> In effect, there is a period of one week (next week) where no post 1.5
>>> features can be merged (feature freeze but release branch not forked),
>>> which should be okay.
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
>> k.kloudas@data-artisans.com
>>>> wrote:
>>>
>>>> For me as well +1.
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org> wrote:
>>>>>
>>>>> Sounds good to me. +1 from my side.
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>>>>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
>>>> we agree to not merge new features to master after that and then to the
>>>> actual hard cutting of the release branch a while later.
>>>>>>
>>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
>>>> feature freeze and end of next week (23.02.2018) as the hard cut of the
>>>> release branch?
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> --
>>>>>> Aljoscha
>>>>>>
>>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org>
>> wrote:
>>>>>>>
>>>>>>> Local state recovery is almost completely done. Only some reviews and
>>>>>>> merging of the final PRs is pending.
>>>>>>>
>>>>>>> The network stack improvements are on a good way to be finished by
>> the
>>>> end
>>>>>>> of this week or beginning of next week. To my knowledge we got
>> recently
>>>>>>> green Travis builds :-) The network stack changes will also include
>> the
>>>>>>> application level flow control and the back pressure based checkpoint
>>>>>>> alignment. So only the last reviews and merging is missing.
>>>>>>>
>>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
>> default.
>>>>>>> There are still some smaller things left to be done but I'm confident
>>>> that
>>>>>>> we can resolve them quickly.
>>>>>>>
>>>>>>> I agree that due to the big changes we should have a very thorough
>> and
>>>>>>> principled testing period where we put Flink through the paces.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
>> chesnay@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
>>>> local
>>>>>>>> state recovery) are nearly done.
>>>>>>>>
>>>>>>>> I'm unsure about local state recovery, but I still see open issues
>> for
>>>>>>>> FLIP-6 and the network stack rework.
>>>>>>>> As such it doesn't make sense to release 1.5 now.
>>>>>>>>
>>>>>>>> Given the large scope of these features I would very much prefer to
>>>> have
>>>>>>>> them active on master for a while before a feature-freeze
>>>>>>>> to expose them to a wider audience.
>>>>>>>>
>>>>>>>> IMO it will take at least another month before we can start the
>>>> release
>>>>>>>> process for 1.5, i.e. the feature freeze.
>>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
>>>> settle)
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>>>>>
>>>>>>>>> Hi Aljoscha,
>>>>>>>>>
>>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>>>>>> and there are some pending issues related to scala api and
>>>> documentation.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kostas
>>>>>>>>>
>>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org>
>> wrote:
>>>>>>>>>> Hi Shuyi,
>>>>>>>>>>
>>>>>>>>>> I will take a look at it again this week. I'm pretty sure it will
>> be
>>>>>>>>>> part of 1.5.0.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Timo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>>>>>
>>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot
>> of
>>>>>>>>>>> internal users waiting for this feature.
>>>>>>>>>>>
>>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
>>>> Support
>>>>>>>>>>> accessing subfields of a Composite element in an Object Array
>> type
>>>>>>>>>>> column
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot
>>>>>>>>>>> Shuyi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
>> cjolif@gmail.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>>>>>
>>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible
>>>> with
>>>>>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>>>>>
>>>>>>>>>>>> have long been awaited and there was one PR from me and from
>>>> someone
>>>>>>>>>>>> else
>>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
>>>> would
>>>>>>>>>>>> be
>>>>>>>>>>>> great!
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> --
>>>>>>>>>>>> Christophe
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
>> twalthr@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>>> it would be great if we can include the first version of the
>> SQL
>>>>>>>>>>>>> client
>>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
>>>> week. I
>>>>>>>>>>>>> think
>>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
>>>> is far
>>>>>>>>>>>>>
>>>>>>>>>>>> away
>>>>>>>>>>>>
>>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>>>>>> beginners.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In order to use the SQL client we would need to also add some
>>>> table
>>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
>>>> this is
>>>>>>>>>>>>> optional because a user can implement own table factories at
>> the
>>>>>>>>>>>>>
>>>>>>>>>>>> begining.
>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Timo
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think there’s a few connector related must-have improvements
>>>> that
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
>>>> users have
>>>>>>>>>>>>>>
>>>>>>>>>>>>> been
>>>>>>>>>>>>> asking for them:
>>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
>> timestamp
>>>> to
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>
>>>>>>>>>>>>> up
>>>>>>>>>>>>> start offset
>>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
>>>> should
>>>>>>>>>>>>>> consider idle partitions
>>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>>>> FlinkKafkaConsumer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479
>> is
>>>>>>>>>>>>>> still
>>>>>>>>>>>>>> lacking a pull request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Gordon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>>>>>
>>>>>>>>>>>>> aljoscha@apache.org)
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
>>>> that to
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>
>>>>>>>>>>>>> a
>>>>>>>>>>>>> stable release out before putting in a couple of new features.
>>>> Back
>>>>>>>>>>>>> then,
>>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
>> local
>>>> state
>>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
>>>> 1.5.0
>>>>>>>>>>>>>> development cycle to allow for those features to become ready
>>>> and
>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>
>>>>>>>>>>>>> do
>>>>>>>>>>>>> the next release.
>>>>>>>>>>>>>> We are now approaching the approximate time where we wanted to
>>>> do the
>>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
>>>> gather
>>>>>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>>>> manager for
>>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>>>>>> interested in
>>>>>>>>>>>>>> doing that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>> Christophe
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
+1 from my side.
Cheers,
Till
On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <pi...@data-artisans.com>
wrote:
> +0.95 from my side.
>
> Network changes are mostly reviewed and should be merged by the end of
> this week.
>
> Piotrek
>
> > On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
> >
> > I agree with the basic idea. I think there is no need to call it "soft
> > feature freeze" though - it is a feature freeze (no new features get
> > merged) ;-)
> >
> > What you are suggesting is to delay forking of the release-1.5 branch to
> > avoid applying the bug fixes to too many branches. That makes sense.
> > In effect, there is a period of one week (next week) where no post 1.5
> > features can be merged (feature freeze but release branch not forked),
> > which should be okay.
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> k.kloudas@data-artisans.com
> >> wrote:
> >
> >> For me as well +1.
> >>
> >> Cheers,
> >> Kostas
> >>
> >>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org> wrote:
> >>>
> >>> Sounds good to me. +1 from my side.
> >>>
> >>> Regards,
> >>> Timo
> >>>
> >>>
> >>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
> >> we agree to not merge new features to master after that and then to the
> >> actual hard cutting of the release branch a while later.
> >>>>
> >>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
> >> feature freeze and end of next week (23.02.2018) as the hard cut of the
> >> release branch?
> >>>>
> >>>> What do you think?
> >>>>
> >>>> --
> >>>> Aljoscha
> >>>>
> >>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org>
> wrote:
> >>>>>
> >>>>> Local state recovery is almost completely done. Only some reviews and
> >>>>> merging of the final PRs is pending.
> >>>>>
> >>>>> The network stack improvements are on a good way to be finished by
> the
> >> end
> >>>>> of this week or beginning of next week. To my knowledge we got
> recently
> >>>>> green Travis builds :-) The network stack changes will also include
> the
> >>>>> application level flow control and the back pressure based checkpoint
> >>>>> alignment. So only the last reviews and merging is missing.
> >>>>>
> >>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> default.
> >>>>> There are still some smaller things left to be done but I'm confident
> >> that
> >>>>> we can resolve them quickly.
> >>>>>
> >>>>> I agree that due to the big changes we should have a very thorough
> and
> >>>>> principled testing period where we put Flink through the paces.
> >>>>>
> >>>>> Cheers,
> >>>>> Till
> >>>>>
> >>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> chesnay@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> >> local
> >>>>>> state recovery) are nearly done.
> >>>>>>
> >>>>>> I'm unsure about local state recovery, but I still see open issues
> for
> >>>>>> FLIP-6 and the network stack rework.
> >>>>>> As such it doesn't make sense to release 1.5 now.
> >>>>>>
> >>>>>> Given the large scope of these features I would very much prefer to
> >> have
> >>>>>> them active on master for a while before a feature-freeze
> >>>>>> to expose them to a wider audience.
> >>>>>>
> >>>>>> IMO it will take at least another month before we can start the
> >> release
> >>>>>> process for 1.5, i.e. the feature freeze.
> >>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
> >> settle)
> >>>>>>
> >>>>>>
> >>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>>>
> >>>>>>> Hi Aljoscha,
> >>>>>>>
> >>>>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>>>> and there are some pending issues related to scala api and
> >> documentation.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Kostas
> >>>>>>>
> >>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org>
> wrote:
> >>>>>>>> Hi Shuyi,
> >>>>>>>>
> >>>>>>>> I will take a look at it again this week. I'm pretty sure it will
> be
> >>>>>>>> part of 1.5.0.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Timo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>>>
> >>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot
> of
> >>>>>>>>> internal users waiting for this feature.
> >>>>>>>>>
> >>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
> >> Support
> >>>>>>>>> accessing subfields of a Composite element in an Object Array
> type
> >>>>>>>>> column
> >>>>>>>>>
> >>>>>>>>> Thanks a lot
> >>>>>>>>> Shuyi
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> cjolif@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi guys,
> >>>>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>>>
> >>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible
> >> with
> >>>>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>>>
> >>>>>>>>>> have long been awaited and there was one PR from me and from
> >> someone
> >>>>>>>>>> else
> >>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
> >> would
> >>>>>>>>>> be
> >>>>>>>>>> great!
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>> --
> >>>>>>>>>> Christophe
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> twalthr@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>> it would be great if we can include the first version of the
> SQL
> >>>>>>>>>>> client
> >>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> >> week. I
> >>>>>>>>>>> think
> >>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
> >> is far
> >>>>>>>>>>>
> >>>>>>>>>> away
> >>>>>>>>>>
> >>>>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>>>> beginners.
> >>>>>>>>>>>
> >>>>>>>>>>> In order to use the SQL client we would need to also add some
> >> table
> >>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
> >> this is
> >>>>>>>>>>> optional because a user can implement own table factories at
> the
> >>>>>>>>>>>
> >>>>>>>>>> begining.
> >>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Timo
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think there’s a few connector related must-have improvements
> >> that
> >>>>>>>>>>>> we
> >>>>>>>>>>>> should get in before the feature freeze, since quite a few
> >> users have
> >>>>>>>>>>>>
> >>>>>>>>>>> been
> >>>>>>>>>>> asking for them:
> >>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> timestamp
> >> to
> >>>>>>>>>>>> set
> >>>>>>>>>>>>
> >>>>>>>>>>> up
> >>>>>>>>>>> start offset
> >>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> >> should
> >>>>>>>>>>>> consider idle partitions
> >>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> >> FlinkKafkaConsumer
> >>>>>>>>>>>>
> >>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479
> is
> >>>>>>>>>>>> still
> >>>>>>>>>>>> lacking a pull request.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Gordon
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>>>
> >>>>>>>>>>> aljoscha@apache.org)
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>
> >>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> >> that to
> >>>>>>>>>>>> get
> >>>>>>>>>>>>
> >>>>>>>>>>> a
> >>>>>>>>>>> stable release out before putting in a couple of new features.
> >> Back
> >>>>>>>>>>> then,
> >>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> local
> >> state
> >>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> >> 1.5.0
> >>>>>>>>>>>> development cycle to allow for those features to become ready
> >> and
> >>>>>>>>>>>> then
> >>>>>>>>>>>>
> >>>>>>>>>>> do
> >>>>>>>>>>> the next release.
> >>>>>>>>>>>> We are now approaching the approximate time where we wanted to
> >> do the
> >>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
> >> gather
> >>>>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>>>
> >>>>>>>>>>>> With this, I'd also like to propose myself as the release
> >> manager for
> >>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>>>> interested in
> >>>>>>>>>>>> doing that.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Aljoscha
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>> Christophe
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> >>
> >>
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Piotr Nowojski <pi...@data-artisans.com>.
+0.95 from my side.
Network changes are mostly reviewed and should be merged by the end of this week.
Piotrek
> On 12 Feb 2018, at 17:41, Stephan Ewen <se...@apache.org> wrote:
>
> I agree with the basic idea. I think there is no need to call it "soft
> feature freeze" though - it is a feature freeze (no new features get
> merged) ;-)
>
> What you are suggesting is to delay forking of the release-1.5 branch to
> avoid applying the bug fixes to too many branches. That makes sense.
> In effect, there is a period of one week (next week) where no post 1.5
> features can be merged (feature freeze but release branch not forked),
> which should be okay.
>
> Best,
> Stephan
>
>
> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <k.kloudas@data-artisans.com
>> wrote:
>
>> For me as well +1.
>>
>> Cheers,
>> Kostas
>>
>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org> wrote:
>>>
>>> Sounds good to me. +1 from my side.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
>> we agree to not merge new features to master after that and then to the
>> actual hard cutting of the release branch a while later.
>>>>
>>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
>> feature freeze and end of next week (23.02.2018) as the hard cut of the
>> release branch?
>>>>
>>>> What do you think?
>>>>
>>>> --
>>>> Aljoscha
>>>>
>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org> wrote:
>>>>>
>>>>> Local state recovery is almost completely done. Only some reviews and
>>>>> merging of the final PRs is pending.
>>>>>
>>>>> The network stack improvements are on a good way to be finished by the
>> end
>>>>> of this week or beginning of next week. To my knowledge we got recently
>>>>> green Travis builds :-) The network stack changes will also include the
>>>>> application level flow control and the back pressure based checkpoint
>>>>> alignment. So only the last reviews and merging is missing.
>>>>>
>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>>>>> There are still some smaller things left to be done but I'm confident
>> that
>>>>> we can resolve them quickly.
>>>>>
>>>>> I agree that due to the big changes we should have a very thorough and
>>>>> principled testing period where we put Flink through the paces.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
>> local
>>>>>> state recovery) are nearly done.
>>>>>>
>>>>>> I'm unsure about local state recovery, but I still see open issues for
>>>>>> FLIP-6 and the network stack rework.
>>>>>> As such it doesn't make sense to release 1.5 now.
>>>>>>
>>>>>> Given the large scope of these features I would very much prefer to
>> have
>>>>>> them active on master for a while before a feature-freeze
>>>>>> to expose them to a wider audience.
>>>>>>
>>>>>> IMO it will take at least another month before we can start the
>> release
>>>>>> process for 1.5, i.e. the feature freeze.
>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
>> settle)
>>>>>>
>>>>>>
>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>>>> and there are some pending issues related to scala api and
>> documentation.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kostas
>>>>>>>
>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>>>>>>> Hi Shuyi,
>>>>>>>>
>>>>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>>>>> part of 1.5.0.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Timo
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>>>
>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>>>>> internal users waiting for this feature.
>>>>>>>>>
>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
>> Support
>>>>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>>>>> column
>>>>>>>>>
>>>>>>>>> Thanks a lot
>>>>>>>>> Shuyi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cjolif@gmail.com
>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>>>
>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible
>> with
>>>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>>>
>>>>>>>>>> have long been awaited and there was one PR from me and from
>> someone
>>>>>>>>>> else
>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
>> would
>>>>>>>>>> be
>>>>>>>>>> great!
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> --
>>>>>>>>>> Christophe
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>>>>> client
>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
>> week. I
>>>>>>>>>>> think
>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
>> is far
>>>>>>>>>>>
>>>>>>>>>> away
>>>>>>>>>>
>>>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>>>> beginners.
>>>>>>>>>>>
>>>>>>>>>>> In order to use the SQL client we would need to also add some
>> table
>>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
>> this is
>>>>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>>>>
>>>>>>>>>> begining.
>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Timo
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>>>
>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>>>
>>>>>>>>>>>> I think there’s a few connector related must-have improvements
>> that
>>>>>>>>>>>> we
>>>>>>>>>>>> should get in before the feature freeze, since quite a few
>> users have
>>>>>>>>>>>>
>>>>>>>>>>> been
>>>>>>>>>>> asking for them:
>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp
>> to
>>>>>>>>>>>> set
>>>>>>>>>>>>
>>>>>>>>>>> up
>>>>>>>>>>> start offset
>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
>> should
>>>>>>>>>>>> consider idle partitions
>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>> FlinkKafkaConsumer
>>>>>>>>>>>>
>>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>>>>> still
>>>>>>>>>>>> lacking a pull request.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Gordon
>>>>>>>>>>>>
>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>>>
>>>>>>>>>>> aljoscha@apache.org)
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
>> that to
>>>>>>>>>>>> get
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>> stable release out before putting in a couple of new features.
>> Back
>>>>>>>>>>> then,
>>>>>>>>>>> some of those new features (FLIP-6, network stack changes, local
>> state
>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
>> 1.5.0
>>>>>>>>>>>> development cycle to allow for those features to become ready
>> and
>>>>>>>>>>>> then
>>>>>>>>>>>>
>>>>>>>>>>> do
>>>>>>>>>>> the next release.
>>>>>>>>>>>> We are now approaching the approximate time where we wanted to
>> do the
>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
>> gather
>>>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>>>
>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>> manager for
>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>>>> interested in
>>>>>>>>>>>> doing that.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Christophe
>>>>>>>>>>
>>>>>>>>>>
>>>
>>
>>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Stephan Ewen <se...@apache.org>.
I agree with the basic idea. I think there is no need to call it "soft
feature freeze" though - it is a feature freeze (no new features get
merged) ;-)
What you are suggesting is to delay forking of the release-1.5 branch to
avoid applying the bug fixes to too many branches. That makes sense.
In effect, there is a period of one week (next week) where no post 1.5
features can be merged (feature freeze but release branch not forked),
which should be okay.
Best,
Stephan
On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <k.kloudas@data-artisans.com
> wrote:
> For me as well +1.
>
> Cheers,
> Kostas
>
> > On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org> wrote:
> >
> > Sounds good to me. +1 from my side.
> >
> > Regards,
> > Timo
> >
> >
> > Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >> I agree with Chesnay: we should do a soft "feature freeze" first, were
> we agree to not merge new features to master after that and then to the
> actual hard cutting of the release branch a while later.
> >>
> >> For actual dates, I'm proposing end of this week (16.02.2018) as soft
> feature freeze and end of next week (23.02.2018) as the hard cut of the
> release branch?
> >>
> >> What do you think?
> >>
> >> --
> >> Aljoscha
> >>
> >>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org> wrote:
> >>>
> >>> Local state recovery is almost completely done. Only some reviews and
> >>> merging of the final PRs is pending.
> >>>
> >>> The network stack improvements are on a good way to be finished by the
> end
> >>> of this week or beginning of next week. To my knowledge we got recently
> >>> green Travis builds :-) The network stack changes will also include the
> >>> application level flow control and the back pressure based checkpoint
> >>> alignment. So only the last reviews and merging is missing.
> >>>
> >>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
> >>> There are still some smaller things left to be done but I'm confident
> that
> >>> we can resolve them quickly.
> >>>
> >>> I agree that due to the big changes we should have a very thorough and
> >>> principled testing period where we put Flink through the paces.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
> >>> wrote:
> >>>
> >>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>> assumption that the 3 big features (FLIP-6, network stack changes,
> local
> >>>> state recovery) are nearly done.
> >>>>
> >>>> I'm unsure about local state recovery, but I still see open issues for
> >>>> FLIP-6 and the network stack rework.
> >>>> As such it doesn't make sense to release 1.5 now.
> >>>>
> >>>> Given the large scope of these features I would very much prefer to
> have
> >>>> them active on master for a while before a feature-freeze
> >>>> to expose them to a wider audience.
> >>>>
> >>>> IMO it will take at least another month before we can start the
> release
> >>>> process for 1.5, i.e. the feature freeze.
> >>>> (2 more weeks for implementation, 2 weeks on master for the dust to
> settle)
> >>>>
> >>>>
> >>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>
> >>>>> Hi Aljoscha,
> >>>>>
> >>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>> and there are some pending issues related to scala api and
> documentation.
> >>>>>
> >>>>> Thanks,
> >>>>> Kostas
> >>>>>
> >>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
> >>>>>> Hi Shuyi,
> >>>>>>
> >>>>>> I will take a look at it again this week. I'm pretty sure it will be
> >>>>>> part of 1.5.0.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Timo
> >>>>>>
> >>>>>>
> >>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>
> >>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
> >>>>>>> internal users waiting for this feature.
> >>>>>>>
> >>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
> Support
> >>>>>>> accessing subfields of a Composite element in an Object Array type
> >>>>>>> column
> >>>>>>>
> >>>>>>> Thanks a lot
> >>>>>>> Shuyi
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cjolif@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi guys,
> >>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>
> >>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible
> with
> >>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>
> >>>>>>>> have long been awaited and there was one PR from me and from
> someone
> >>>>>>>> else
> >>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
> would
> >>>>>>>> be
> >>>>>>>> great!
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>> --
> >>>>>>>> Christophe
> >>>>>>>>
> >>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Aljoscha,
> >>>>>>>>> it would be great if we can include the first version of the SQL
> >>>>>>>>> client
> >>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> week. I
> >>>>>>>>> think
> >>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
> is far
> >>>>>>>>>
> >>>>>>>> away
> >>>>>>>>
> >>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>> beginners.
> >>>>>>>>>
> >>>>>>>>> In order to use the SQL client we would need to also add some
> table
> >>>>>>>>> sources with the new unified table factories (FLINK-8535), but
> this is
> >>>>>>>>> optional because a user can implement own table factories at the
> >>>>>>>>>
> >>>>>>>> begining.
> >>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Timo
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>
> >>>>>>>>> Hi Aljoscha,
> >>>>>>>>>
> >>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>
> >>>>>>>>>> I think there’s a few connector related must-have improvements
> that
> >>>>>>>>>> we
> >>>>>>>>>> should get in before the feature freeze, since quite a few
> users have
> >>>>>>>>>>
> >>>>>>>>> been
> >>>>>>>>> asking for them:
> >>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp
> to
> >>>>>>>>>> set
> >>>>>>>>>>
> >>>>>>>>> up
> >>>>>>>>> start offset
> >>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> should
> >>>>>>>>>> consider idle partitions
> >>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> FlinkKafkaConsumer
> >>>>>>>>>>
> >>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
> >>>>>>>>>> still
> >>>>>>>>>> lacking a pull request.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Gordon
> >>>>>>>>>>
> >>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>
> >>>>>>>>> aljoscha@apache.org)
> >>>>>>>>> wrote:
> >>>>>>>>>> Hi Everyone,
> >>>>>>>>>>
> >>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> that to
> >>>>>>>>>> get
> >>>>>>>>>>
> >>>>>>>>> a
> >>>>>>>>> stable release out before putting in a couple of new features.
> Back
> >>>>>>>>> then,
> >>>>>>>>> some of those new features (FLIP-6, network stack changes, local
> state
> >>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> 1.5.0
> >>>>>>>>>> development cycle to allow for those features to become ready
> and
> >>>>>>>>>> then
> >>>>>>>>>>
> >>>>>>>>> do
> >>>>>>>>> the next release.
> >>>>>>>>>> We are now approaching the approximate time where we wanted to
> do the
> >>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
> gather
> >>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>
> >>>>>>>>>> With this, I'd also like to propose myself as the release
> manager for
> >>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>> interested in
> >>>>>>>>>> doing that.
> >>>>>>>>>>
> >>>>>>>>>> What do you think?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Aljoscha
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>> Christophe
> >>>>>>>>
> >>>>>>>>
> >
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Kostas Kloudas <k....@data-artisans.com>.
For me as well +1.
Cheers,
Kostas
> On Feb 12, 2018, at 2:59 PM, Timo Walther <tw...@apache.org> wrote:
>
> Sounds good to me. +1 from my side.
>
> Regards,
> Timo
>
>
> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>> I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.
>>
>> For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?
>>
>> What do you think?
>>
>> --
>> Aljoscha
>>
>>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org> wrote:
>>>
>>> Local state recovery is almost completely done. Only some reviews and
>>> merging of the final PRs is pending.
>>>
>>> The network stack improvements are on a good way to be finished by the end
>>> of this week or beginning of next week. To my knowledge we got recently
>>> green Travis builds :-) The network stack changes will also include the
>>> application level flow control and the back pressure based checkpoint
>>> alignment. So only the last reviews and merging is missing.
>>>
>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>>> There are still some smaller things left to be done but I'm confident that
>>> we can resolve them quickly.
>>>
>>> I agree that due to the big changes we should have a very thorough and
>>> principled testing period where we put Flink through the paces.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>> assumption that the 3 big features (FLIP-6, network stack changes, local
>>>> state recovery) are nearly done.
>>>>
>>>> I'm unsure about local state recovery, but I still see open issues for
>>>> FLIP-6 and the network stack rework.
>>>> As such it doesn't make sense to release 1.5 now.
>>>>
>>>> Given the large scope of these features I would very much prefer to have
>>>> them active on master for a while before a feature-freeze
>>>> to expose them to a wider audience.
>>>>
>>>> IMO it will take at least another month before we can start the release
>>>> process for 1.5, i.e. the feature freeze.
>>>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>>>
>>>>
>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>
>>>>> Hi Aljoscha,
>>>>>
>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>> and there are some pending issues related to scala api and documentation.
>>>>>
>>>>> Thanks,
>>>>> Kostas
>>>>>
>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>>>>> Hi Shuyi,
>>>>>>
>>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>>> part of 1.5.0.
>>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>>
>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>
>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>>> internal users waiting for this feature.
>>>>>>>
>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>>> column
>>>>>>>
>>>>>>> Thanks a lot
>>>>>>> Shuyi
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi guys,
>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>
>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>
>>>>>>>> have long been awaited and there was one PR from me and from someone
>>>>>>>> else
>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>>>> be
>>>>>>>> great!
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> --
>>>>>>>> Christophe
>>>>>>>>
>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Aljoscha,
>>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>>> client
>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>>>> think
>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>>>
>>>>>>>> away
>>>>>>>>
>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>> beginners.
>>>>>>>>>
>>>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>>
>>>>>>>> begining.
>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Timo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>
>>>>>>>>> Hi Aljoscha,
>>>>>>>>>
>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>
>>>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>>>> we
>>>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>>>
>>>>>>>>> been
>>>>>>>>> asking for them:
>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>>>> set
>>>>>>>>>>
>>>>>>>>> up
>>>>>>>>> start offset
>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>>>> consider idle partitions
>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>>>
>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>>> still
>>>>>>>>>> lacking a pull request.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Gordon
>>>>>>>>>>
>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>
>>>>>>>>> aljoscha@apache.org)
>>>>>>>>> wrote:
>>>>>>>>>> Hi Everyone,
>>>>>>>>>>
>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>>>> get
>>>>>>>>>>
>>>>>>>>> a
>>>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>>> then,
>>>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>>>> then
>>>>>>>>>>
>>>>>>>>> do
>>>>>>>>> the next release.
>>>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>
>>>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>> interested in
>>>>>>>>>> doing that.
>>>>>>>>>>
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Aljoscha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Christophe
>>>>>>>>
>>>>>>>>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Timo Walther <tw...@apache.org>.
Sounds good to me. +1 from my side.
Regards,
Timo
Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.
>
> For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?
>
> What do you think?
>
> --
> Aljoscha
>
>> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org> wrote:
>>
>> Local state recovery is almost completely done. Only some reviews and
>> merging of the final PRs is pending.
>>
>> The network stack improvements are on a good way to be finished by the end
>> of this week or beginning of next week. To my knowledge we got recently
>> green Travis builds :-) The network stack changes will also include the
>> application level flow control and the back pressure based checkpoint
>> alignment. So only the last reviews and merging is missing.
>>
>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>> There are still some smaller things left to be done but I'm confident that
>> we can resolve them quickly.
>>
>> I agree that due to the big changes we should have a very thorough and
>> principled testing period where we put Flink through the paces.
>>
>> Cheers,
>> Till
>>
>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>> assumption that the 3 big features (FLIP-6, network stack changes, local
>>> state recovery) are nearly done.
>>>
>>> I'm unsure about local state recovery, but I still see open issues for
>>> FLIP-6 and the network stack rework.
>>> As such it doesn't make sense to release 1.5 now.
>>>
>>> Given the large scope of these features I would very much prefer to have
>>> them active on master for a while before a feature-freeze
>>> to expose them to a wider audience.
>>>
>>> IMO it will take at least another month before we can start the release
>>> process for 1.5, i.e. the feature freeze.
>>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>>
>>>
>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>
>>>> Hi Aljoscha,
>>>>
>>>> I believe that support for Broadcast State should also be in 1.5.
>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>> https://github.com/apache/flink/pull/5230> for that
>>>> and there are some pending issues related to scala api and documentation.
>>>>
>>>> Thanks,
>>>> Kostas
>>>>
>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>>>> Hi Shuyi,
>>>>>
>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>> part of 1.5.0.
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>
>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>> internal users waiting for this feature.
>>>>>>
>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>> column
>>>>>>
>>>>>> Thanks a lot
>>>>>> Shuyi
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>> Sorry for jumping in, but I think
>>>>>>>
>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>>>>>> Elasticsearch 5.2+ client
>>>>>>>
>>>>>>> have long been awaited and there was one PR from me and from someone
>>>>>>> else
>>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>>> be
>>>>>>> great!
>>>>>>>
>>>>>>> Thanks!
>>>>>>> --
>>>>>>> Christophe
>>>>>>>
>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>> client
>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>>> think
>>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>>
>>>>>>> away
>>>>>>>
>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>> beginners.
>>>>>>>>
>>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>
>>>>>>> begining.
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Timo
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>
>>>>>>>> Hi Aljoscha,
>>>>>>>>
>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>
>>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>>> we
>>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>>
>>>>>>>> been
>>>>>>>> asking for them:
>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>>> set
>>>>>>>>>
>>>>>>>> up
>>>>>>>> start offset
>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>>> consider idle partitions
>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>>
>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>> still
>>>>>>>>> lacking a pull request.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Gordon
>>>>>>>>>
>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>
>>>>>>>> aljoscha@apache.org)
>>>>>>>> wrote:
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>>> get
>>>>>>>>>
>>>>>>>> a
>>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>> then,
>>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>>> then
>>>>>>>>>
>>>>>>>> do
>>>>>>>> the next release.
>>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>
>>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>> interested in
>>>>>>>>> doing that.
>>>>>>>>>
>>>>>>>>> What do you think?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Aljoscha
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>> Christophe
>>>>>>>
>>>>>>>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Aljoscha Krettek <al...@apache.org>.
I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.
For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?
What do you think?
--
Aljoscha
> On 8. Feb 2018, at 10:15, Till Rohrmann <tr...@apache.org> wrote:
>
> Local state recovery is almost completely done. Only some reviews and
> merging of the final PRs is pending.
>
> The network stack improvements are on a good way to be finished by the end
> of this week or beginning of next week. To my knowledge we got recently
> green Travis builds :-) The network stack changes will also include the
> application level flow control and the back pressure based checkpoint
> alignment. So only the last reviews and merging is missing.
>
> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
> There are still some smaller things left to be done but I'm confident that
> we can resolve them quickly.
>
> I agree that due to the big changes we should have a very thorough and
> principled testing period where we put Flink through the paces.
>
> Cheers,
> Till
>
> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>> assumption that the 3 big features (FLIP-6, network stack changes, local
>> state recovery) are nearly done.
>>
>> I'm unsure about local state recovery, but I still see open issues for
>> FLIP-6 and the network stack rework.
>> As such it doesn't make sense to release 1.5 now.
>>
>> Given the large scope of these features I would very much prefer to have
>> them active on master for a while before a feature-freeze
>> to expose them to a wider audience.
>>
>> IMO it will take at least another month before we can start the release
>> process for 1.5, i.e. the feature freeze.
>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>
>>
>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>
>>> Hi Aljoscha,
>>>
>>> I believe that support for Broadcast State should also be in 1.5.
>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>> https://github.com/apache/flink/pull/5230> for that
>>> and there are some pending issues related to scala api and documentation.
>>>
>>> Thanks,
>>> Kostas
>>>
>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>>>
>>>> Hi Shuyi,
>>>>
>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>> part of 1.5.0.
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>
>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>> internal users waiting for this feature.
>>>>>
>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>> accessing subfields of a Composite element in an Object Array type
>>>>> column
>>>>>
>>>>> Thanks a lot
>>>>> Shuyi
>>>>>
>>>>>
>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi guys,
>>>>>>
>>>>>> Sorry for jumping in, but I think
>>>>>>
>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>>>>> Elasticsearch 5.2+ client
>>>>>>
>>>>>> have long been awaited and there was one PR from me and from someone
>>>>>> else
>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>> be
>>>>>> great!
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> Christophe
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>> client
>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>> think
>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>
>>>>>> away
>>>>>>
>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>> beginners.
>>>>>>>
>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>> optional because a user can implement own table factories at the
>>>>>>>
>>>>>> begining.
>>>>>>
>>>>>>> Regards,
>>>>>>> Timo
>>>>>>>
>>>>>>>
>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>>> Thanks for starting the discussion.
>>>>>>>>
>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>> we
>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>
>>>>>>> been
>>>>>>
>>>>>>> asking for them:
>>>>>>>>
>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>> set
>>>>>>>>
>>>>>>> up
>>>>>>
>>>>>>> start offset
>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>> consider idle partitions
>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>> FlinkKinesisConsumer
>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>
>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>> still
>>>>>>>> lacking a pull request.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Gordon
>>>>>>>>
>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>
>>>>>>> aljoscha@apache.org)
>>>>>>
>>>>>>> wrote:
>>>>>>>> Hi Everyone,
>>>>>>>>
>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>> get
>>>>>>>>
>>>>>>> a
>>>>>>
>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>>
>>>>>>> then,
>>>>>>
>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>> then
>>>>>>>>
>>>>>>> do
>>>>>>
>>>>>>> the next release.
>>>>>>>>
>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>> opinions on how we should proceed now.
>>>>>>>>
>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>> interested in
>>>>>>>> doing that.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Aljoscha
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>> Christophe
>>>>>>
>>>>>>
>>>>>
>>>
>>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Till Rohrmann <tr...@apache.org>.
Local state recovery is almost completely done. Only some reviews and
merging of the final PRs is pending.
The network stack improvements are on a good way to be finished by the end
of this week or beginning of next week. To my knowledge we got recently
green Travis builds :-) The network stack changes will also include the
application level flow control and the back pressure based checkpoint
alignment. So only the last reviews and merging is missing.
Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
There are still some smaller things left to be done but I'm confident that
we can resolve them quickly.
I agree that due to the big changes we should have a very thorough and
principled testing period where we put Flink through the paces.
Cheers,
Till
On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <ch...@apache.org>
wrote:
> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> assumption that the 3 big features (FLIP-6, network stack changes, local
> state recovery) are nearly done.
>
> I'm unsure about local state recovery, but I still see open issues for
> FLIP-6 and the network stack rework.
> As such it doesn't make sense to release 1.5 now.
>
> Given the large scope of these features I would very much prefer to have
> them active on master for a while before a feature-freeze
> to expose them to a wider audience.
>
> IMO it will take at least another month before we can start the release
> process for 1.5, i.e. the feature freeze.
> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>
>
> On 05.02.2018 22:39, Kostas Kloudas wrote:
>
>> Hi Aljoscha,
>>
>> I believe that support for Broadcast State should also be in 1.5.
>> There is an open PR https://github.com/apache/flink/pull/5230 <
>> https://github.com/apache/flink/pull/5230> for that
>> and there are some pending issues related to scala api and documentation.
>>
>> Thanks,
>> Kostas
>>
>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>>
>>> Hi Shuyi,
>>>
>>> I will take a look at it again this week. I'm pretty sure it will be
>>> part of 1.5.0.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>
>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>> internal users waiting for this feature.
>>>>
>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>> accessing subfields of a Composite element in an Object Array type
>>>> column
>>>>
>>>> Thanks a lot
>>>> Shuyi
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi guys,
>>>>>
>>>>> Sorry for jumping in, but I think
>>>>>
>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>>>> Elasticsearch 5.2+ client
>>>>>
>>>>> have long been awaited and there was one PR from me and from someone
>>>>> else
>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>> be
>>>>> great!
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> Christophe
>>>>>
>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org>
>>>>> wrote:
>>>>>
>>>>> Hi Aljoscha,
>>>>>>
>>>>>> it would be great if we can include the first version of the SQL
>>>>>> client
>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>> think
>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>
>>>>> away
>>>>>
>>>>>> from feature completeness but will be a great tool for Flink
>>>>>> beginners.
>>>>>>
>>>>>> In order to use the SQL client we would need to also add some table
>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>> optional because a user can implement own table factories at the
>>>>>>
>>>>> begining.
>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>>
>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>
>>>>>> Hi Aljoscha,
>>>>>>
>>>>>>> Thanks for starting the discussion.
>>>>>>>
>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>> we
>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>
>>>>>> been
>>>>>
>>>>>> asking for them:
>>>>>>>
>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>> set
>>>>>>>
>>>>>> up
>>>>>
>>>>>> start offset
>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>> consider idle partitions
>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>> FlinkKinesisConsumer
>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>
>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>> still
>>>>>>> lacking a pull request.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Gordon
>>>>>>>
>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>
>>>>>> aljoscha@apache.org)
>>>>>
>>>>>> wrote:
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>> get
>>>>>>>
>>>>>> a
>>>>>
>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>
>>>>>> then,
>>>>>
>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>> development cycle to allow for those features to become ready and
>>>>>>> then
>>>>>>>
>>>>>> do
>>>>>
>>>>>> the next release.
>>>>>>>
>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>> opinions on how we should proceed now.
>>>>>>>
>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>> interested in
>>>>>>> doing that.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Best,
>>>>>>> Aljoscha
>>>>>>>
>>>>>>>
>>>>>> --
>>>>> Christophe
>>>>>
>>>>>
>>>>
>>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Chesnay Schepler <ch...@apache.org>.
As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
assumption that the 3 big features (FLIP-6, network stack changes, local
state recovery) are nearly done.
I'm unsure about local state recovery, but I still see open issues for
FLIP-6 and the network stack rework.
As such it doesn't make sense to release 1.5 now.
Given the large scope of these features I would very much prefer to have
them active on master for a while before a feature-freeze
to expose them to a wider audience.
IMO it will take at least another month before we can start the release
process for 1.5, i.e. the feature freeze.
(2 more weeks for implementation, 2 weeks on master for the dust to settle)
On 05.02.2018 22:39, Kostas Kloudas wrote:
> Hi Aljoscha,
>
> I believe that support for Broadcast State should also be in 1.5.
> There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> for that
> and there are some pending issues related to scala api and documentation.
>
> Thanks,
> Kostas
>
>> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>>
>> Hi Shuyi,
>>
>> I will take a look at it again this week. I'm pretty sure it will be part of 1.5.0.
>>
>> Regards,
>> Timo
>>
>>
>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>> internal users waiting for this feature.
>>>
>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>> accessing subfields of a Composite element in an Object Array type column
>>>
>>> Thanks a lot
>>> Shuyi
>>>
>>>
>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> Sorry for jumping in, but I think
>>>>
>>>> [FLINK-8101] Elasticsearch 6.X support
>>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>>> Elasticsearch 5.2+ client
>>>>
>>>> have long been awaited and there was one PR from me and from someone else
>>>> showing the interest ;) So if you could consider it for 1.5 that would be
>>>> great!
>>>>
>>>> Thanks!
>>>> --
>>>> Christophe
>>>>
>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org> wrote:
>>>>
>>>>> Hi Aljoscha,
>>>>>
>>>>> it would be great if we can include the first version of the SQL client
>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>> away
>>>>> from feature completeness but will be a great tool for Flink beginners.
>>>>>
>>>>> In order to use the SQL client we would need to also add some table
>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>> optional because a user can implement own table factories at the
>>>> begining.
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>
>>>>> Hi Aljoscha,
>>>>>> Thanks for starting the discussion.
>>>>>>
>>>>>> I think there’s a few connector related must-have improvements that we
>>>>>> should get in before the feature freeze, since quite a few users have
>>>> been
>>>>>> asking for them:
>>>>>>
>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>>>> up
>>>>>> start offset
>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>> consider idle partitions
>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>> FlinkKinesisConsumer
>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>
>>>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>>>> lacking a pull request.
>>>>>>
>>>>>> Cheers,
>>>>>> Gordon
>>>>>>
>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>> aljoscha@apache.org)
>>>>>> wrote:
>>>>>> Hi Everyone,
>>>>>>
>>>>>> When we decided to do the 1.4.0 release a while back we did that to get
>>>> a
>>>>>> stable release out before putting in a couple of new features. Back
>>>> then,
>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>> development cycle to allow for those features to become ready and then
>>>> do
>>>>>> the next release.
>>>>>>
>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>> opinions on how we should proceed now.
>>>>>>
>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>>>> doing that.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> Best,
>>>>>> Aljoscha
>>>>>>
>>>>>
>>>> --
>>>> Christophe
>>>>
>>>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Aljoscha,
I believe that support for Broadcast State should also be in 1.5.
There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> for that
and there are some pending issues related to scala api and documentation.
Thanks,
Kostas
> On Feb 5, 2018, at 5:37 PM, Timo Walther <tw...@apache.org> wrote:
>
> Hi Shuyi,
>
> I will take a look at it again this week. I'm pretty sure it will be part of 1.5.0.
>
> Regards,
> Timo
>
>
> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>> internal users waiting for this feature.
>>
>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>> accessing subfields of a Composite element in an Object Array type column
>>
>> Thanks a lot
>> Shuyi
>>
>>
>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> Sorry for jumping in, but I think
>>>
>>> [FLINK-8101] Elasticsearch 6.X support
>>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>>> Elasticsearch 5.2+ client
>>>
>>> have long been awaited and there was one PR from me and from someone else
>>> showing the interest ;) So if you could consider it for 1.5 that would be
>>> great!
>>>
>>> Thanks!
>>> --
>>> Christophe
>>>
>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org> wrote:
>>>
>>>> Hi Aljoscha,
>>>>
>>>> it would be great if we can include the first version of the SQL client
>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>> away
>>>> from feature completeness but will be a great tool for Flink beginners.
>>>>
>>>> In order to use the SQL client we would need to also add some table
>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>> optional because a user can implement own table factories at the
>>> begining.
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>
>>>> Hi Aljoscha,
>>>>> Thanks for starting the discussion.
>>>>>
>>>>> I think there’s a few connector related must-have improvements that we
>>>>> should get in before the feature freeze, since quite a few users have
>>> been
>>>>> asking for them:
>>>>>
>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>>> up
>>>>> start offset
>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>> consider idle partitions
>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>> FlinkKinesisConsumer
>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>
>>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>>> lacking a pull request.
>>>>>
>>>>> Cheers,
>>>>> Gordon
>>>>>
>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>> aljoscha@apache.org)
>>>>> wrote:
>>>>> Hi Everyone,
>>>>>
>>>>> When we decided to do the 1.4.0 release a while back we did that to get
>>> a
>>>>> stable release out before putting in a couple of new features. Back
>>> then,
>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>> development cycle to allow for those features to become ready and then
>>> do
>>>>> the next release.
>>>>>
>>>>> We are now approaching the approximate time where we wanted to do the
>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>> opinions on how we should proceed now.
>>>>>
>>>>> With this, I'd also like to propose myself as the release manager for
>>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>>> doing that.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Best,
>>>>> Aljoscha
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Christophe
>>>
>>
>>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Timo Walther <tw...@apache.org>.
Hi Shuyi,
I will take a look at it again this week. I'm pretty sure it will be
part of 1.5.0.
Regards,
Timo
Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
> internal users waiting for this feature.
>
> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
> accessing subfields of a Composite element in an Object Array type column
>
> Thanks a lot
> Shuyi
>
>
> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com> wrote:
>
>> Hi guys,
>>
>> Sorry for jumping in, but I think
>>
>> [FLINK-8101] Elasticsearch 6.X support
>> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
>> Elasticsearch 5.2+ client
>>
>> have long been awaited and there was one PR from me and from someone else
>> showing the interest ;) So if you could consider it for 1.5 that would be
>> great!
>>
>> Thanks!
>> --
>> Christophe
>>
>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org> wrote:
>>
>>> Hi Aljoscha,
>>>
>>> it would be great if we can include the first version of the SQL client
>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>> we can merge this with explicit "experimental/alpha" status. It is far
>> away
>>> from feature completeness but will be a great tool for Flink beginners.
>>>
>>> In order to use the SQL client we would need to also add some table
>>> sources with the new unified table factories (FLINK-8535), but this is
>>> optional because a user can implement own table factories at the
>> begining.
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>
>>> Hi Aljoscha,
>>>> Thanks for starting the discussion.
>>>>
>>>> I think there’s a few connector related must-have improvements that we
>>>> should get in before the feature freeze, since quite a few users have
>> been
>>>> asking for them:
>>>>
>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>> up
>>>> start offset
>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>> consider idle partitions
>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>> FlinkKinesisConsumer
>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>
>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>> lacking a pull request.
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>> aljoscha@apache.org)
>>>> wrote:
>>>> Hi Everyone,
>>>>
>>>> When we decided to do the 1.4.0 release a while back we did that to get
>> a
>>>> stable release out before putting in a couple of new features. Back
>> then,
>>>> some of those new features (FLIP-6, network stack changes, local state
>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>> development cycle to allow for those features to become ready and then
>> do
>>>> the next release.
>>>>
>>>> We are now approaching the approximate time where we wanted to do the
>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>> opinions on how we should proceed now.
>>>>
>>>> With this, I'd also like to propose myself as the release manager for
>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>> doing that.
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>
>>>
>>
>> --
>> Christophe
>>
>
>
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Shuyi Chen <su...@gmail.com>.
Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
internal users waiting for this feature.
[FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
accessing subfields of a Composite element in an Object Array type column
Thanks a lot
Shuyi
On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <cj...@gmail.com> wrote:
> Hi guys,
>
> Sorry for jumping in, but I think
>
> [FLINK-8101] Elasticsearch 6.X support
> [FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
> Elasticsearch 5.2+ client
>
> have long been awaited and there was one PR from me and from someone else
> showing the interest ;) So if you could consider it for 1.5 that would be
> great!
>
> Thanks!
> --
> Christophe
>
> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org> wrote:
>
> > Hi Aljoscha,
> >
> > it would be great if we can include the first version of the SQL client
> > (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> > we can merge this with explicit "experimental/alpha" status. It is far
> away
> > from feature completeness but will be a great tool for Flink beginners.
> >
> > In order to use the SQL client we would need to also add some table
> > sources with the new unified table factories (FLINK-8535), but this is
> > optional because a user can implement own table factories at the
> begining.
> >
> > Regards,
> > Timo
> >
> >
> > Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >
> > Hi Aljoscha,
> >>
> >> Thanks for starting the discussion.
> >>
> >> I think there’s a few connector related must-have improvements that we
> >> should get in before the feature freeze, since quite a few users have
> been
> >> asking for them:
> >>
> >> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
> up
> >> start offset
> >> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
> >> consider idle partitions
> >> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >> FlinkKinesisConsumer
> >> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
> >>
> >> These are still missing in the master branch. Only FLINK-5479 is still
> >> lacking a pull request.
> >>
> >> Cheers,
> >> Gordon
> >>
> >> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> aljoscha@apache.org)
> >> wrote:
> >> Hi Everyone,
> >>
> >> When we decided to do the 1.4.0 release a while back we did that to get
> a
> >> stable release out before putting in a couple of new features. Back
> then,
> >> some of those new features (FLIP-6, network stack changes, local state
> >> recovery) were almost ready and we wanted to do a shortened 1.5.0
> >> development cycle to allow for those features to become ready and then
> do
> >> the next release.
> >>
> >> We are now approaching the approximate time where we wanted to do the
> >> Flink 1.5.0 release so I would like to gauge where we are and gather
> >> opinions on how we should proceed now.
> >>
> >> With this, I'd also like to propose myself as the release manager for
> >> 1.5.0 but I'm very happy to yield if someone else would be interested in
> >> doing that.
> >>
> >> What do you think?
> >>
> >> Best,
> >> Aljoscha
> >>
> >
> >
> >
>
>
> --
> Christophe
>
--
"So you have to trust that the dots will somehow connect in your future."
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Christophe Jolif <cj...@gmail.com>.
Hi guys,
Sorry for jumping in, but I think
[FLINK-8101] Elasticsearch 6.X support
[FLINK-7386] Flink Elasticsearch 5 connector is not compatible with
Elasticsearch 5.2+ client
have long been awaited and there was one PR from me and from someone else
showing the interest ;) So if you could consider it for 1.5 that would be
great!
Thanks!
--
Christophe
On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <tw...@apache.org> wrote:
> Hi Aljoscha,
>
> it would be great if we can include the first version of the SQL client
> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> we can merge this with explicit "experimental/alpha" status. It is far away
> from feature completeness but will be a great tool for Flink beginners.
>
> In order to use the SQL client we would need to also add some table
> sources with the new unified table factories (FLINK-8535), but this is
> optional because a user can implement own table factories at the begining.
>
> Regards,
> Timo
>
>
> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>
> Hi Aljoscha,
>>
>> Thanks for starting the discussion.
>>
>> I think there’s a few connector related must-have improvements that we
>> should get in before the feature freeze, since quite a few users have been
>> asking for them:
>>
>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up
>> start offset
>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>> consider idle partitions
>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>> FlinkKinesisConsumer
>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>
>> These are still missing in the master branch. Only FLINK-5479 is still
>> lacking a pull request.
>>
>> Cheers,
>> Gordon
>>
>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljoscha@apache.org)
>> wrote:
>> Hi Everyone,
>>
>> When we decided to do the 1.4.0 release a while back we did that to get a
>> stable release out before putting in a couple of new features. Back then,
>> some of those new features (FLIP-6, network stack changes, local state
>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>> development cycle to allow for those features to become ready and then do
>> the next release.
>>
>> We are now approaching the approximate time where we wanted to do the
>> Flink 1.5.0 release so I would like to gauge where we are and gather
>> opinions on how we should proceed now.
>>
>> With this, I'd also like to propose myself as the release manager for
>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>> doing that.
>>
>> What do you think?
>>
>> Best,
>> Aljoscha
>>
>
>
>
--
Christophe
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by Timo Walther <tw...@apache.org>.
Hi Aljoscha,
it would be great if we can include the first version of the SQL client
(see FLIP-24, Implementation Plan 1). I will open a PR this week. I
think we can merge this with explicit "experimental/alpha" status. It is
far away from feature completeness but will be a great tool for Flink
beginners.
In order to use the SQL client we would need to also add some table
sources with the new unified table factories (FLINK-8535), but this is
optional because a user can implement own table factories at the begining.
Regards,
Timo
Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> Hi Aljoscha,
>
> Thanks for starting the discussion.
>
> I think there’s a few connector related must-have improvements that we should get in before the feature freeze, since quite a few users have been asking for them:
>
> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start offset
> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
> [FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>
> These are still missing in the master branch. Only FLINK-5479 is still lacking a pull request.
>
> Cheers,
> Gordon
>
> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljoscha@apache.org) wrote:
> Hi Everyone,
>
> When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.
>
> We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.
>
> With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.
>
> What do you think?
>
> Best,
> Aljoscha
Re: [DISCUSS] Releasing Flink 1.5.0
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Aljoscha,
Thanks for starting the discussion.
I think there’s a few connector related must-have improvements that we should get in before the feature freeze, since quite a few users have been asking for them:
[FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start offset
[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
[FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
[FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
These are still missing in the master branch. Only FLINK-5479 is still lacking a pull request.
Cheers,
Gordon
On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljoscha@apache.org) wrote:
Hi Everyone,
When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.
We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.
With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.
What do you think?
Best,
Aljoscha