You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2014/08/27 19:21:07 UTC

Planning Flink release 0.7-incubating

Hi,

since we have our release infrastructure in place now, I would suggest to
aim for a 0.7-incubating release in the near future (say 3-5 weeks).
While 0.6-incubating was mainly about getting the release infra / legal
stuff sorted out, I think it would be nice to have a "feature" release soon.

The following new features would make a great 0.7-incubating release:
 - *Flink Streaming* "Beta Preview". I would suggest to ship the streaming,
but clearly mark it as a preview in the documentation.
 -* Java API Pojo improvements*: Code generation, key selection using a
string-expression: https://issues.apache.org/jira/browse/FLINK-1032
  - *Reworked Scala API*. Bring the Scala API in sync with the latest
developments in the Java API:
https://issues.apache.org/jira/browse/FLINK-641
  -* Akka-based RPC service*:
https://issues.apache.org/jira/browse/FLINK-1019
  - *Kryo-based serialization*. This feature has been requested by many
users. Mostly because they wanted to use Collections inside POJOs:
https://issues.apache.org/jira/browse/FLINK-610


Opinions?

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
Done with reverting and merging the individual commits.

Sorry, didn't wanted to sound too negative. I just think merged commits are
an important incentive for contributors.
And thanks for merging of course ;-)

2014-10-08 12:25 GMT+02:00 Robert Metzger <rm...@apache.org>:

> Oh, damn. You're right. I've been using the "merge_pull_request.sh" script
> which is squashing together the commits automatically. In this particular
> case, its quite unfortunate because it contains part of the work of Artems
> GSoC summer project, and also another pull request from Timo.
>
> I agree if you revert and push the commits individually in this case.
>
> On Wed, Oct 8, 2014 at 12:19 PM, Fabian Hueske <fh...@apache.org> wrote:
>
> > I checked the merged Hadoop function compatibility pull request.
> > The PR consisted of four individual commits of three authors. All commits
> > have been squashed into a single commit such that two authors basically
> > lost their contributions.
> > I think this is not correct and should never be done. It might cause
> > frustration among contributors and eventually cause contributors to leave
> > the community.
> >
> > I propose to squash commits in pull requests only if it is really
> necessary
> > and in that case only commits that have the same author.
> >
> > For the current issue with the Hadoop function compat PR, I will prepare
> a
> > revert patch for the master and add the four commits of the PR on top.
> >
> > Fabian
> >
> > 2014-10-08 12:06 GMT+02:00 Robert Metzger <rm...@apache.org>:
> >
> > > I merged the Hadoop compat and POJOs, currently rebasing stephan's
> fault
> > > tolerance ....
> > >
> > > On Wed, Oct 8, 2014 at 10:50 AM, Fabian Hueske <fh...@apache.org>
> > wrote:
> > >
> > > > I think, the question is which PRs to merge into master before
> forking
> > of
> > > > the release candidate branch.
> > > > You did already merge #126 into the master branch, so it will be in
> ;-)
> > > >
> > > > 2014-10-08 10:37 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> > > >
> > > > > We can also include the blob manager. The corresponding PR is #126.
> > > > >
> > > > > On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Pushed the resolution for FLINK-1103 and the streaming bugfix
> > commit.
> > > > > >
> > > > > > We are good for now on our side for an rc. Maybe I should add the
> > > > > streaming
> > > > > > connectors dependency fix that Stephan suggested here an the
> > mailing
> > > > list
> > > > > > soon.
> > > > > >
> > > > > > Marton
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > We can add it and keep it initially undocumented (experimental)
> > > until
> > > > > > > further tests...
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <
> > > rmetzger@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I would add #142.
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <
> > sewen@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree, having a release candidate out would be nice.
> > > > > > > > >
> > > > > > > > > What is your opinion on issue #142? Fault tolerance is
> > inactive
> > > > by
> > > > > > > > default,
> > > > > > > > > but can be activated through the configuration.
> > > > > > > > >
> > > > > > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <
> > > > > rmetzger@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Oh. I didn't know this. The last mail from Gyula sounded
> > like
> > > > the
> > > > > > > > > Streaming
> > > > > > > > > > part is ready2release.
> > > > > > > > > >
> > > > > > > > > > My personal goal was to prepare a release candiate today
> so
> > > > that
> > > > > we
> > > > > > > > have
> > > > > > > > > a
> > > > > > > > > > reference point against which we can test (which would
> also
> > > > mean
> > > > > > > > forking
> > > > > > > > > of
> > > > > > > > > > a 0.7-release branch and basically doing a feature
> freeze).
> > > > > > > > > > I also though that it might be a good idea to have a
> > release
> > > > > > > candidate
> > > > > > > > by
> > > > > > > > > > tomorrow because it could be of use for the Hackathon
> > > tomorrow
> > > > in
> > > > > > > > > > Stockholm.
> > > > > > > > > > But since the Scala POJO changes and the streaming
> examples
> > > are
> > > > > > both
> > > > > > > > > > outstanding, we can probably scratch that.
> > > > > > > > > >
> > > > > > > > > > Doing a fork, release candidate and a feature freeze
> > doesn't
> > > > mean
> > > > > > we
> > > > > > > > can
> > > > > > > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > > > > > > >
> > > > > > > > > > What is your best-case scenario for FLINK-1103? (as far
> as
> > I
> > > > can
> > > > > > see
> > > > > > > > the
> > > > > > > > > > only outstanding streaming feature for the release)
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Robert
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > > > > > > balassi.marton@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As for the streaming side we would like to push some
> > bugfix
> > > > > > commits
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > > > > > > >
> > > > > > > > > > > These are more or less ready, hopefully will be
> available
> > > at
> > > > > the
> > > > > > > end
> > > > > > > > of
> > > > > > > > > > > this week.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > > > > > > rmetzger@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I've merged the record api deprecation already.
> > > > > > > > > > > > I'll merge #141 once Aljoscha provided his Scala
> > changes.
> > > > > > > > > > > >
> > > > > > > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > > > > > > fhueske@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > So which PRs will be included in the candidate?
> > > > > > > > > > > > >
> > > > > > > > > > > > > #141 POJOs
> > > > > > > > > > > > > #144 Deprecate Record API
> > > > > > > > > > > > > #136 Fixed example quickstart
> > > > > > > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop
> > function
> > > > > > wrappers
> > > > > > > > > > > (includes
> > > > > > > > > > > > > PR #131)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <
> > > > sewen@apache.org
> > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Great news, looking forward to seeing this in the
> > > > master!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > > > > > > rmetzger@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As an update for everyone: My POJO feature is
> > > > finished,
> > > > > > > > > including
> > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > Aljoscha is currently adopting the Scala API to
> > > have
> > > > > > > support
> > > > > > > > > for
> > > > > > > > > > > > > (nested)
> > > > > > > > > > > > > > > POJOs as well.
> > > > > > > > > > > > > > > Once that is done, I'll merge everything and
> > > create a
> > > > > > first
> > > > > > > > > > > candidate
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > we can use for testing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian
> Hueske <
> > > > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I just checked the "Run Example" quickstart
> and
> > > it
> > > > > > needs
> > > > > > > a
> > > > > > > > > bit
> > > > > > > > > > of
> > > > > > > > > > > > > work.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > > > > > > rmetzger@apache.org
> > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm working hard on getting the POJOs
> ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We also should do a pass over our
> > > documentation,
> > > > > the
> > > > > > > > > > > quickstarts
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > website to see if everything is in a good
> > state
> > > > > (for
> > > > > > > > > example
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > collection-based execution needs
> > documentation
> > > as
> > > > > > > well).
> > > > > > > > We
> > > > > > > > > > > > should
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > finally document the hadoop-input format
> > > wrappers
> > > > > (I
> > > > > > > > think
> > > > > > > > > > Timo
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > working
> > > > > > > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > > > > > > This page mentions the
> > LocalDistributedExecutor
> > > > and
> > > > > > > > > contains
> > > > > > > > > > > some
> > > > > > > > > > > > > > (most
> > > > > > > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We also need to deprecate the old record
> api
> > (
> > > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/FLINK-1106
> > > > ).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan
> > Ewen <
> > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > > > > > > Last issues are going in and we started
> > > working
> > > > > on
> > > > > > > > > > > dependencies
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > test
> > > > > > > > > > > > > > > > > > platform diversity in order to make
> > > stabilizing
> > > > > > > phase.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We should have an official feature freeze
> > > soon
> > > > > and
> > > > > > > fork
> > > > > > > > > the
> > > > > > > > > > > > > > > 0.7-release
> > > > > > > > > > > > > > > > > > branch. I personally vote to include the
> > POJO
> > > > > > support
> > > > > > > > (I
> > > > > > > > > > > think
> > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > sorting that one out and is close to
> > > > completion),
> > > > > > > and I
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > collection-based execution (today or
> > > tomorrow).
> > > > > > Till
> > > > > > > > has
> > > > > > > > > > the
> > > > > > > > > > > > BLOB
> > > > > > > > > > > > > > > > manager
> > > > > > > > > > > > > > > > > > ready, which would be good to include
> > (better
> > > > > > support
> > > > > > > > > large
> > > > > > > > > > > > > > libraries
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula
> > Fora <
> > > > > > > > > > > gyfora@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > So what is the current decision
> regarding
> > > the
> > > > > > time
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > upcoming
> > > > > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As for the streaming component, we
> > included
> > > > all
> > > > > > the
> > > > > > > > > > > features
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > wanted,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > will start to test everything tomorrow,
> > > > making
> > > > > > sure
> > > > > > > > > that
> > > > > > > > > > > all
> > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > > > > > > We are also almost finished with
> cleaning
> > > up
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > > > > > dependencies
> > > > > > > > > > > > > > > > > > > that Stephan pointed out, should be
> > > finished
> > > > by
> > > > > > > > > tomorrow.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian
> Hueske <
> > > > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if
> possible)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha
> > > > Krettek <
> > > > > > > > > > > > > > aljoscha@apache.org
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> Can we not manage this stuff on
> Jira?
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM,
> > > Stephan
> > > > > > Ewen <
> > > > > > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >>> I personally like the idea of SOFT
> > > > > time-based
> > > > > > > > > feature
> > > > > > > > > > > > > > freezes.
> > > > > > > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > > > > > > >>> releases will get delayed again and
> > > > again,
> > > > > > > > because
> > > > > > > > > of
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Having reached the freeze point
> > > already,
> > > > we
> > > > > > > could
> > > > > > > > > > still
> > > > > > > > > > > > > > include
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >>> features that are pending ready
> state
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > > > days
> > > > > > > > > > > > > > > > (streaming,
> > > > > > > > > > > > > > > > > > blob
> > > > > > > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head
> > > for a
> > > > > > > release
> > > > > > > > > > > state.
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> We had a mail listing the issues to
> > > > include
> > > > > > > into
> > > > > > > > > 0.7,
> > > > > > > > > > > > but a
> > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > page
> > > > > > > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > > > > > > >>> probably be better. In that sense,
> we
> > > > could
> > > > > > > start
> > > > > > > > > > > > > collecting
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb
> "Daniel
> > > > > > Warneke" <
> > > > > > > > > > > > > > > warneke@apache.org
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a
> > wiki
> > > > page
> > > > > > (or
> > > > > > > > > > > something
> > > > > > > > > > > > > > > > similar)
> > > > > > > > > > > > > > > > > > > where
> > > > > > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb
> Fabian
> > > > > Hueske:
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>>>> I agree, a hard feature stop
> > deadline
> > > > > might
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > the
> > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > >>>>> How about the following
> procedure:
> > > > > > > > > > > > > > > > > > > >>>>> We decide two (or three) weeks
> > > before a
> > > > > > > > targeted
> > > > > > > > > > > > release
> > > > > > > > > > > > > > date
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are
> > > > selected
> > > > > > > for a
> > > > > > > > > > > release
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > > > > > > >>>>> or really close to completion
> (via
> > > > > progress
> > > > > > > > > > estimates
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > JIRA).
> > > > > > > > > > > > > > > > > > > >>>>> After we decided which JIRAs to
> > > include
> > > > > in
> > > > > > a
> > > > > > > > > > release,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > > > > > >>>>> track the progress and dedicate
> > > another
> > > > > > week
> > > > > > > > > > > > exclusively
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > testing
> > > > > > > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton
> > > > > Balassi <
> > > > > > > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > >>>>> As for the streaming team we're
> > also
> > > > > > getting
> > > > > > > > > ready
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > release,
> > > > > > > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > > > > > > >>>>>> couple of days will be needed to
> > > > finish
> > > > > > the
> > > > > > > > > > features
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>    - A little work is still
> needed
> > > for
> > > > > > > reduce
> > > > > > > > > > > > operations
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >>>>>>    groups/connected streams (any
> > > > comment
> > > > > > on
> > > > > > > > > > Gyula's
> > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > e-mail
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > > > > > > >>>>>>    - The examples are being
> > updated
> > > to
> > > > > > match
> > > > > > > > the
> > > > > > > > > > > > > standard,
> > > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some
> > deja
> > > > vu.
> > > > > :)
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM,
> > > Ufuk
> > > > > > > Celebi <
> > > > > > > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert
> > > > > Metzger <
> > > > > > > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we
> > discussed
> > > to
> > > > > do
> > > > > > a
> > > > > > > > > > feature
> > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta
> > > Preview".
> > > > I
> > > > > > > would
> > > > > > > > > > > suggest
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > ship
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a
> preview
> > > in
> > > > > the
> > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo
> improvements*:
> > > Code
> > > > > > > > > generation,
> > > > > > > > > > > key
> > > > > > > > > > > > > > > > selection
> > > > > > > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring
> > the
> > > > > Scala
> > > > > > > API
> > > > > > > > > in
> > > > > > > > > > > sync
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*.
> > > This
> > > > > > > feature
> > > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > requested
> > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > > > > > > >>>>>>>> users. Mostly because they
> > wanted
> > > to
> > > > > use
> > > > > > > > > > > Collections
> > > > > > > > > > > > > > > inside
> > > > > > > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals
> to
> > > > > support
> > > > > > > > > > > incremental
> > > > > > > > > > > > > > > program
> > > > > > > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic
> memory
> > > > > > > assignments
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> The following features are in
> > the
> > > > > > master,
> > > > > > > as
> > > > > > > > > of
> > > > > > > > > > > > today:
> > > > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> We certainly need some days to
> > > test
> > > > > > > > everything
> > > > > > > > > > > until
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> Based on our experience with
> the
> > > > last
> > > > > > > major
> > > > > > > > > > > > release, I
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing
> > > BEFORE
> > > > > the
> > > > > > > > first
> > > > > > > > > > > > release
> > > > > > > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had
> 6
> > > > > > > candidates)
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> How do you guys feel about
> this?
> > > > > Should
> > > > > > we
> > > > > > > > > wait
> > > > > > > > > > a
> > > > > > > > > > > > few
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > days
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > > >>>>>>>> release so that a few more
> > > features
> > > > > make
> > > > > > > it
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the
> > one
> > > > > hand,
> > > > > > > its
> > > > > > > > > > really
> > > > > > > > > > > > > nice
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also
> > eats
> > > > up
> > > > > > some
> > > > > > > > > time
> > > > > > > > > > > and
> > > > > > > > > > > > > > causes
> > > > > > > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > > > > > > >>>>>>>> I would really like to have
> the
> > > Java
> > > > > API
> > > > > > > > Pojo
> > > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> I think I can finish it until
> > end
> > > of
> > > > > > this
> > > > > > > > > week.
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> I agree that the finished
> > features
> > > > > > > > (especially
> > > > > > > > > > the
> > > > > > > > > > > > > Scala
> > > > > > > > > > > > > > > API)
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> a new release, but still I
> would
> > > like
> > > > > to
> > > > > > > > wait a
> > > > > > > > > > few
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > days.
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>> Some of the missing features
> are
> > on
> > > > the
> > > > > > > brink
> > > > > > > > > of
> > > > > > > > > > > > being
> > > > > > > > > > > > > > > > finished
> > > > > > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't
> > want
> > > > to
> > > > > > > > invest a
> > > > > > > > > > > week
> > > > > > > > > > > > in
> > > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > > > > > >>>>>>> doing the release vote, when
> the
> > > new
> > > > > > > features
> > > > > > > > > are
> > > > > > > > > > > > > likely
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And
> > the
> > > > > > > upcoming
> > > > > > > > > > > features
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > > > > > > >>>>>>> worth a release, so users can
> > work
> > > > with
> > > > > > > them.
> > > > > > > > > ;)
> > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
Oh, damn. You're right. I've been using the "merge_pull_request.sh" script
which is squashing together the commits automatically. In this particular
case, its quite unfortunate because it contains part of the work of Artems
GSoC summer project, and also another pull request from Timo.

I agree if you revert and push the commits individually in this case.

On Wed, Oct 8, 2014 at 12:19 PM, Fabian Hueske <fh...@apache.org> wrote:

> I checked the merged Hadoop function compatibility pull request.
> The PR consisted of four individual commits of three authors. All commits
> have been squashed into a single commit such that two authors basically
> lost their contributions.
> I think this is not correct and should never be done. It might cause
> frustration among contributors and eventually cause contributors to leave
> the community.
>
> I propose to squash commits in pull requests only if it is really necessary
> and in that case only commits that have the same author.
>
> For the current issue with the Hadoop function compat PR, I will prepare a
> revert patch for the master and add the four commits of the PR on top.
>
> Fabian
>
> 2014-10-08 12:06 GMT+02:00 Robert Metzger <rm...@apache.org>:
>
> > I merged the Hadoop compat and POJOs, currently rebasing stephan's fault
> > tolerance ....
> >
> > On Wed, Oct 8, 2014 at 10:50 AM, Fabian Hueske <fh...@apache.org>
> wrote:
> >
> > > I think, the question is which PRs to merge into master before forking
> of
> > > the release candidate branch.
> > > You did already merge #126 into the master branch, so it will be in ;-)
> > >
> > > 2014-10-08 10:37 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> > >
> > > > We can also include the blob manager. The corresponding PR is #126.
> > > >
> > > > On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <
> > balassi.marton@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Pushed the resolution for FLINK-1103 and the streaming bugfix
> commit.
> > > > >
> > > > > We are good for now on our side for an rc. Maybe I should add the
> > > > streaming
> > > > > connectors dependency fix that Stephan suggested here an the
> mailing
> > > list
> > > > > soon.
> > > > >
> > > > > Marton
> > > > >
> > > > > On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > We can add it and keep it initially undocumented (experimental)
> > until
> > > > > > further tests...
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I would add #142.
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <
> sewen@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > I agree, having a release candidate out would be nice.
> > > > > > > >
> > > > > > > > What is your opinion on issue #142? Fault tolerance is
> inactive
> > > by
> > > > > > > default,
> > > > > > > > but can be activated through the configuration.
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <
> > > > rmetzger@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Oh. I didn't know this. The last mail from Gyula sounded
> like
> > > the
> > > > > > > > Streaming
> > > > > > > > > part is ready2release.
> > > > > > > > >
> > > > > > > > > My personal goal was to prepare a release candiate today so
> > > that
> > > > we
> > > > > > > have
> > > > > > > > a
> > > > > > > > > reference point against which we can test (which would also
> > > mean
> > > > > > > forking
> > > > > > > > of
> > > > > > > > > a 0.7-release branch and basically doing a feature freeze).
> > > > > > > > > I also though that it might be a good idea to have a
> release
> > > > > > candidate
> > > > > > > by
> > > > > > > > > tomorrow because it could be of use for the Hackathon
> > tomorrow
> > > in
> > > > > > > > > Stockholm.
> > > > > > > > > But since the Scala POJO changes and the streaming examples
> > are
> > > > > both
> > > > > > > > > outstanding, we can probably scratch that.
> > > > > > > > >
> > > > > > > > > Doing a fork, release candidate and a feature freeze
> doesn't
> > > mean
> > > > > we
> > > > > > > can
> > > > > > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > > > > > >
> > > > > > > > > What is your best-case scenario for FLINK-1103? (as far as
> I
> > > can
> > > > > see
> > > > > > > the
> > > > > > > > > only outstanding streaming feature for the release)
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Robert
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > > > > > balassi.marton@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As for the streaming side we would like to push some
> bugfix
> > > > > commits
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > > > > > >
> > > > > > > > > > These are more or less ready, hopefully will be available
> > at
> > > > the
> > > > > > end
> > > > > > > of
> > > > > > > > > > this week.
> > > > > > > > > >
> > > > > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > > > > > rmetzger@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I've merged the record api deprecation already.
> > > > > > > > > > > I'll merge #141 once Aljoscha provided his Scala
> changes.
> > > > > > > > > > >
> > > > > > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > > > > > fhueske@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > So which PRs will be included in the candidate?
> > > > > > > > > > > >
> > > > > > > > > > > > #141 POJOs
> > > > > > > > > > > > #144 Deprecate Record API
> > > > > > > > > > > > #136 Fixed example quickstart
> > > > > > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop
> function
> > > > > wrappers
> > > > > > > > > > (includes
> > > > > > > > > > > > PR #131)
> > > > > > > > > > > >
> > > > > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <
> > > sewen@apache.org
> > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Great news, looking forward to seeing this in the
> > > master!
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > > > > > rmetzger@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As an update for everyone: My POJO feature is
> > > finished,
> > > > > > > > including
> > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > Aljoscha is currently adopting the Scala API to
> > have
> > > > > > support
> > > > > > > > for
> > > > > > > > > > > > (nested)
> > > > > > > > > > > > > > POJOs as well.
> > > > > > > > > > > > > > Once that is done, I'll merge everything and
> > create a
> > > > > first
> > > > > > > > > > candidate
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > we can use for testing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I just checked the "Run Example" quickstart and
> > it
> > > > > needs
> > > > > > a
> > > > > > > > bit
> > > > > > > > > of
> > > > > > > > > > > > work.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > > > > > rmetzger@apache.org
> > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We also should do a pass over our
> > documentation,
> > > > the
> > > > > > > > > > quickstarts
> > > > > > > > > > > > and
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > website to see if everything is in a good
> state
> > > > (for
> > > > > > > > example
> > > > > > > > > > the
> > > > > > > > > > > > > > > > collection-based execution needs
> documentation
> > as
> > > > > > well).
> > > > > > > We
> > > > > > > > > > > should
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > finally document the hadoop-input format
> > wrappers
> > > > (I
> > > > > > > think
> > > > > > > > > Timo
> > > > > > > > > > > is
> > > > > > > > > > > > > > > working
> > > > > > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > > > > > This page mentions the
> LocalDistributedExecutor
> > > and
> > > > > > > > contains
> > > > > > > > > > some
> > > > > > > > > > > > > (most
> > > > > > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We also need to deprecate the old record api
> (
> > > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/FLINK-1106
> > > ).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan
> Ewen <
> > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > > > > > Last issues are going in and we started
> > working
> > > > on
> > > > > > > > > > dependencies
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > test
> > > > > > > > > > > > > > > > > platform diversity in order to make
> > stabilizing
> > > > > > phase.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We should have an official feature freeze
> > soon
> > > > and
> > > > > > fork
> > > > > > > > the
> > > > > > > > > > > > > > 0.7-release
> > > > > > > > > > > > > > > > > branch. I personally vote to include the
> POJO
> > > > > support
> > > > > > > (I
> > > > > > > > > > think
> > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > sorting that one out and is close to
> > > completion),
> > > > > > and I
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > add
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > collection-based execution (today or
> > tomorrow).
> > > > > Till
> > > > > > > has
> > > > > > > > > the
> > > > > > > > > > > BLOB
> > > > > > > > > > > > > > > manager
> > > > > > > > > > > > > > > > > ready, which would be good to include
> (better
> > > > > support
> > > > > > > > large
> > > > > > > > > > > > > libraries
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula
> Fora <
> > > > > > > > > > gyfora@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > So what is the current decision regarding
> > the
> > > > > time
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > upcoming
> > > > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As for the streaming component, we
> included
> > > all
> > > > > the
> > > > > > > > > > features
> > > > > > > > > > > we
> > > > > > > > > > > > > > > wanted,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > will start to test everything tomorrow,
> > > making
> > > > > sure
> > > > > > > > that
> > > > > > > > > > all
> > > > > > > > > > > > > works
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > > > > > We are also almost finished with cleaning
> > up
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > > > > > dependencies
> > > > > > > > > > > > > > > > > > that Stephan pointed out, should be
> > finished
> > > by
> > > > > > > > tomorrow.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha
> > > Krettek <
> > > > > > > > > > > > > aljoscha@apache.org
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM,
> > Stephan
> > > > > Ewen <
> > > > > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >>> I personally like the idea of SOFT
> > > > time-based
> > > > > > > > feature
> > > > > > > > > > > > > freezes.
> > > > > > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > > > > > >>> releases will get delayed again and
> > > again,
> > > > > > > because
> > > > > > > > of
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Having reached the freeze point
> > already,
> > > we
> > > > > > could
> > > > > > > > > still
> > > > > > > > > > > > > include
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >>> features that are pending ready state
> > in
> > > > the
> > > > > > next
> > > > > > > > > days
> > > > > > > > > > > > > > > (streaming,
> > > > > > > > > > > > > > > > > blob
> > > > > > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head
> > for a
> > > > > > release
> > > > > > > > > > state.
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> We had a mail listing the issues to
> > > include
> > > > > > into
> > > > > > > > 0.7,
> > > > > > > > > > > but a
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > page
> > > > > > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > > > > > >>> probably be better. In that sense, we
> > > could
> > > > > > start
> > > > > > > > > > > > collecting
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel
> > > > > Warneke" <
> > > > > > > > > > > > > > warneke@apache.org
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a
> wiki
> > > page
> > > > > (or
> > > > > > > > > > something
> > > > > > > > > > > > > > > similar)
> > > > > > > > > > > > > > > > > > where
> > > > > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian
> > > > Hueske:
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>>>> I agree, a hard feature stop
> deadline
> > > > might
> > > > > > not
> > > > > > > > be
> > > > > > > > > > the
> > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > > > > > > >>>>> We decide two (or three) weeks
> > before a
> > > > > > > targeted
> > > > > > > > > > > release
> > > > > > > > > > > > > date
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are
> > > selected
> > > > > > for a
> > > > > > > > > > release
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > > > > > >>>>> or really close to completion (via
> > > > progress
> > > > > > > > > estimates
> > > > > > > > > > > in
> > > > > > > > > > > > > > JIRA).
> > > > > > > > > > > > > > > > > > >>>>> After we decided which JIRAs to
> > include
> > > > in
> > > > > a
> > > > > > > > > release,
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > > > > >>>>> track the progress and dedicate
> > another
> > > > > week
> > > > > > > > > > > exclusively
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > testing
> > > > > > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton
> > > > Balassi <
> > > > > > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > >>>>> As for the streaming team we're
> also
> > > > > getting
> > > > > > > > ready
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > release,
> > > > > > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > > > > > >>>>>> couple of days will be needed to
> > > finish
> > > > > the
> > > > > > > > > features
> > > > > > > > > > > > that
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>    - A little work is still needed
> > for
> > > > > > reduce
> > > > > > > > > > > operations
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >>>>>>    groups/connected streams (any
> > > comment
> > > > > on
> > > > > > > > > Gyula's
> > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > e-mail
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > > > > > >>>>>>    - The examples are being
> updated
> > to
> > > > > match
> > > > > > > the
> > > > > > > > > > > > standard,
> > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some
> deja
> > > vu.
> > > > :)
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM,
> > Ufuk
> > > > > > Celebi <
> > > > > > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert
> > > > Metzger <
> > > > > > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we
> discussed
> > to
> > > > do
> > > > > a
> > > > > > > > > feature
> > > > > > > > > > > > freeze
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta
> > Preview".
> > > I
> > > > > > would
> > > > > > > > > > suggest
> > > > > > > > > > > to
> > > > > > > > > > > > > > ship
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview
> > in
> > > > the
> > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*:
> > Code
> > > > > > > > generation,
> > > > > > > > > > key
> > > > > > > > > > > > > > > selection
> > > > > > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring
> the
> > > > Scala
> > > > > > API
> > > > > > > > in
> > > > > > > > > > sync
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*.
> > This
> > > > > > feature
> > > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > > > requested
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > > > > > >>>>>>>> users. Mostly because they
> wanted
> > to
> > > > use
> > > > > > > > > > Collections
> > > > > > > > > > > > > > inside
> > > > > > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to
> > > > support
> > > > > > > > > > incremental
> > > > > > > > > > > > > > program
> > > > > > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> > > > > > assignments
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> The following features are in
> the
> > > > > master,
> > > > > > as
> > > > > > > > of
> > > > > > > > > > > today:
> > > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> We certainly need some days to
> > test
> > > > > > > everything
> > > > > > > > > > until
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> Based on our experience with the
> > > last
> > > > > > major
> > > > > > > > > > > release, I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing
> > BEFORE
> > > > the
> > > > > > > first
> > > > > > > > > > > release
> > > > > > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> > > > > > candidates)
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> How do you guys feel about this?
> > > > Should
> > > > > we
> > > > > > > > wait
> > > > > > > > > a
> > > > > > > > > > > few
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > days
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > >>>>>>>> release so that a few more
> > features
> > > > make
> > > > > > it
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the
> one
> > > > hand,
> > > > > > its
> > > > > > > > > really
> > > > > > > > > > > > nice
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also
> eats
> > > up
> > > > > some
> > > > > > > > time
> > > > > > > > > > and
> > > > > > > > > > > > > causes
> > > > > > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > > > > > >>>>>>>> I would really like to have the
> > Java
> > > > API
> > > > > > > Pojo
> > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> I think I can finish it until
> end
> > of
> > > > > this
> > > > > > > > week.
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> I agree that the finished
> features
> > > > > > > (especially
> > > > > > > > > the
> > > > > > > > > > > > Scala
> > > > > > > > > > > > > > API)
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> a new release, but still I would
> > like
> > > > to
> > > > > > > wait a
> > > > > > > > > few
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > days.
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>> Some of the missing features are
> on
> > > the
> > > > > > brink
> > > > > > > > of
> > > > > > > > > > > being
> > > > > > > > > > > > > > > finished
> > > > > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't
> want
> > > to
> > > > > > > invest a
> > > > > > > > > > week
> > > > > > > > > > > in
> > > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > > > > >>>>>>> doing the release vote, when the
> > new
> > > > > > features
> > > > > > > > are
> > > > > > > > > > > > likely
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And
> the
> > > > > > upcoming
> > > > > > > > > > features
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > > > > > >>>>>>> worth a release, so users can
> work
> > > with
> > > > > > them.
> > > > > > > > ;)
> > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
I checked the merged Hadoop function compatibility pull request.
The PR consisted of four individual commits of three authors. All commits
have been squashed into a single commit such that two authors basically
lost their contributions.
I think this is not correct and should never be done. It might cause
frustration among contributors and eventually cause contributors to leave
the community.

I propose to squash commits in pull requests only if it is really necessary
and in that case only commits that have the same author.

For the current issue with the Hadoop function compat PR, I will prepare a
revert patch for the master and add the four commits of the PR on top.

Fabian

2014-10-08 12:06 GMT+02:00 Robert Metzger <rm...@apache.org>:

> I merged the Hadoop compat and POJOs, currently rebasing stephan's fault
> tolerance ....
>
> On Wed, Oct 8, 2014 at 10:50 AM, Fabian Hueske <fh...@apache.org> wrote:
>
> > I think, the question is which PRs to merge into master before forking of
> > the release candidate branch.
> > You did already merge #126 into the master branch, so it will be in ;-)
> >
> > 2014-10-08 10:37 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> >
> > > We can also include the blob manager. The corresponding PR is #126.
> > >
> > > On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <
> balassi.marton@gmail.com
> > >
> > > wrote:
> > >
> > > > Pushed the resolution for FLINK-1103 and the streaming bugfix commit.
> > > >
> > > > We are good for now on our side for an rc. Maybe I should add the
> > > streaming
> > > > connectors dependency fix that Stephan suggested here an the mailing
> > list
> > > > soon.
> > > >
> > > > Marton
> > > >
> > > > On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > We can add it and keep it initially undocumented (experimental)
> until
> > > > > further tests...
> > > > >
> > > > > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <
> rmetzger@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > I would add #142.
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > I agree, having a release candidate out would be nice.
> > > > > > >
> > > > > > > What is your opinion on issue #142? Fault tolerance is inactive
> > by
> > > > > > default,
> > > > > > > but can be activated through the configuration.
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Oh. I didn't know this. The last mail from Gyula sounded like
> > the
> > > > > > > Streaming
> > > > > > > > part is ready2release.
> > > > > > > >
> > > > > > > > My personal goal was to prepare a release candiate today so
> > that
> > > we
> > > > > > have
> > > > > > > a
> > > > > > > > reference point against which we can test (which would also
> > mean
> > > > > > forking
> > > > > > > of
> > > > > > > > a 0.7-release branch and basically doing a feature freeze).
> > > > > > > > I also though that it might be a good idea to have a release
> > > > > candidate
> > > > > > by
> > > > > > > > tomorrow because it could be of use for the Hackathon
> tomorrow
> > in
> > > > > > > > Stockholm.
> > > > > > > > But since the Scala POJO changes and the streaming examples
> are
> > > > both
> > > > > > > > outstanding, we can probably scratch that.
> > > > > > > >
> > > > > > > > Doing a fork, release candidate and a feature freeze doesn't
> > mean
> > > > we
> > > > > > can
> > > > > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > > > > >
> > > > > > > > What is your best-case scenario for FLINK-1103? (as far as I
> > can
> > > > see
> > > > > > the
> > > > > > > > only outstanding streaming feature for the release)
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > > > > balassi.marton@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As for the streaming side we would like to push some bugfix
> > > > commits
> > > > > > and
> > > > > > > > the
> > > > > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > > > > >
> > > > > > > > > These are more or less ready, hopefully will be available
> at
> > > the
> > > > > end
> > > > > > of
> > > > > > > > > this week.
> > > > > > > > >
> > > > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > > > > rmetzger@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I've merged the record api deprecation already.
> > > > > > > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > > > > > > >
> > > > > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > > > > >
> > > > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > > > > fhueske@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > So which PRs will be included in the candidate?
> > > > > > > > > > >
> > > > > > > > > > > #141 POJOs
> > > > > > > > > > > #144 Deprecate Record API
> > > > > > > > > > > #136 Fixed example quickstart
> > > > > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function
> > > > wrappers
> > > > > > > > > (includes
> > > > > > > > > > > PR #131)
> > > > > > > > > > >
> > > > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <
> > sewen@apache.org
> > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Great news, looking forward to seeing this in the
> > master!
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > > > > rmetzger@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > As an update for everyone: My POJO feature is
> > finished,
> > > > > > > including
> > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > Aljoscha is currently adopting the Scala API to
> have
> > > > > support
> > > > > > > for
> > > > > > > > > > > (nested)
> > > > > > > > > > > > > POJOs as well.
> > > > > > > > > > > > > Once that is done, I'll merge everything and
> create a
> > > > first
> > > > > > > > > candidate
> > > > > > > > > > > > that
> > > > > > > > > > > > > we can use for testing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I just checked the "Run Example" quickstart and
> it
> > > > needs
> > > > > a
> > > > > > > bit
> > > > > > > > of
> > > > > > > > > > > work.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > > > > rmetzger@apache.org
> > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We also should do a pass over our
> documentation,
> > > the
> > > > > > > > > quickstarts
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > website to see if everything is in a good state
> > > (for
> > > > > > > example
> > > > > > > > > the
> > > > > > > > > > > > > > > collection-based execution needs documentation
> as
> > > > > well).
> > > > > > We
> > > > > > > > > > should
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > finally document the hadoop-input format
> wrappers
> > > (I
> > > > > > think
> > > > > > > > Timo
> > > > > > > > > > is
> > > > > > > > > > > > > > working
> > > > > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > > > > This page mentions the LocalDistributedExecutor
> > and
> > > > > > > contains
> > > > > > > > > some
> > > > > > > > > > > > (most
> > > > > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/FLINK-1106
> > ).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > > > > Last issues are going in and we started
> working
> > > on
> > > > > > > > > dependencies
> > > > > > > > > > > and
> > > > > > > > > > > > > > test
> > > > > > > > > > > > > > > > platform diversity in order to make
> stabilizing
> > > > > phase.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We should have an official feature freeze
> soon
> > > and
> > > > > fork
> > > > > > > the
> > > > > > > > > > > > > 0.7-release
> > > > > > > > > > > > > > > > branch. I personally vote to include the POJO
> > > > support
> > > > > > (I
> > > > > > > > > think
> > > > > > > > > > > > Robert
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > sorting that one out and is close to
> > completion),
> > > > > and I
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > collection-based execution (today or
> tomorrow).
> > > > Till
> > > > > > has
> > > > > > > > the
> > > > > > > > > > BLOB
> > > > > > > > > > > > > > manager
> > > > > > > > > > > > > > > > ready, which would be good to include (better
> > > > support
> > > > > > > large
> > > > > > > > > > > > libraries
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > > > > > > gyfora@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So what is the current decision regarding
> the
> > > > time
> > > > > of
> > > > > > > the
> > > > > > > > > > > > upcoming
> > > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As for the streaming component, we included
> > all
> > > > the
> > > > > > > > > features
> > > > > > > > > > we
> > > > > > > > > > > > > > wanted,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > will start to test everything tomorrow,
> > making
> > > > sure
> > > > > > > that
> > > > > > > > > all
> > > > > > > > > > > > works
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > > > > We are also almost finished with cleaning
> up
> > > the
> > > > > > > > connector
> > > > > > > > > > > > > > dependencies
> > > > > > > > > > > > > > > > > that Stephan pointed out, should be
> finished
> > by
> > > > > > > tomorrow.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha
> > Krettek <
> > > > > > > > > > > > aljoscha@apache.org
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM,
> Stephan
> > > > Ewen <
> > > > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >>> I personally like the idea of SOFT
> > > time-based
> > > > > > > feature
> > > > > > > > > > > > freezes.
> > > > > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > > > > >>> releases will get delayed again and
> > again,
> > > > > > because
> > > > > > > of
> > > > > > > > > > > > features
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Having reached the freeze point
> already,
> > we
> > > > > could
> > > > > > > > still
> > > > > > > > > > > > include
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>> features that are pending ready state
> in
> > > the
> > > > > next
> > > > > > > > days
> > > > > > > > > > > > > > (streaming,
> > > > > > > > > > > > > > > > blob
> > > > > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head
> for a
> > > > > release
> > > > > > > > > state.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> We had a mail listing the issues to
> > include
> > > > > into
> > > > > > > 0.7,
> > > > > > > > > > but a
> > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > page
> > > > > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > > > > >>> probably be better. In that sense, we
> > could
> > > > > start
> > > > > > > > > > > collecting
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel
> > > > Warneke" <
> > > > > > > > > > > > > warneke@apache.org
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki
> > page
> > > > (or
> > > > > > > > > something
> > > > > > > > > > > > > > similar)
> > > > > > > > > > > > > > > > > where
> > > > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian
> > > Hueske:
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline
> > > might
> > > > > not
> > > > > > > be
> > > > > > > > > the
> > > > > > > > > > > best
> > > > > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > > > > > >>>>> We decide two (or three) weeks
> before a
> > > > > > targeted
> > > > > > > > > > release
> > > > > > > > > > > > date
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are
> > selected
> > > > > for a
> > > > > > > > > release
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > > > > >>>>> or really close to completion (via
> > > progress
> > > > > > > > estimates
> > > > > > > > > > in
> > > > > > > > > > > > > JIRA).
> > > > > > > > > > > > > > > > > >>>>> After we decided which JIRAs to
> include
> > > in
> > > > a
> > > > > > > > release,
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > > > >>>>> track the progress and dedicate
> another
> > > > week
> > > > > > > > > > exclusively
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > testing
> > > > > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton
> > > Balassi <
> > > > > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> As for the streaming team we're also
> > > > getting
> > > > > > > ready
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > release,
> > > > > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > > > > >>>>>> couple of days will be needed to
> > finish
> > > > the
> > > > > > > > features
> > > > > > > > > > > that
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>    - A little work is still needed
> for
> > > > > reduce
> > > > > > > > > > operations
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >>>>>>    groups/connected streams (any
> > comment
> > > > on
> > > > > > > > Gyula's
> > > > > > > > > > > recent
> > > > > > > > > > > > > > > e-mail
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > > > > >>>>>>    - The examples are being updated
> to
> > > > match
> > > > > > the
> > > > > > > > > > > standard,
> > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > > > > >>>>>>
> > > > streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja
> > vu.
> > > :)
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM,
> Ufuk
> > > > > Celebi <
> > > > > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert
> > > Metzger <
> > > > > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed
> to
> > > do
> > > > a
> > > > > > > > feature
> > > > > > > > > > > freeze
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta
> Preview".
> > I
> > > > > would
> > > > > > > > > suggest
> > > > > > > > > > to
> > > > > > > > > > > > > ship
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview
> in
> > > the
> > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*:
> Code
> > > > > > > generation,
> > > > > > > > > key
> > > > > > > > > > > > > > selection
> > > > > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the
> > > Scala
> > > > > API
> > > > > > > in
> > > > > > > > > sync
> > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*.
> This
> > > > > feature
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > > > requested
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted
> to
> > > use
> > > > > > > > > Collections
> > > > > > > > > > > > > inside
> > > > > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to
> > > support
> > > > > > > > > incremental
> > > > > > > > > > > > > program
> > > > > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> > > > > assignments
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> The following features are in the
> > > > master,
> > > > > as
> > > > > > > of
> > > > > > > > > > today:
> > > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> We certainly need some days to
> test
> > > > > > everything
> > > > > > > > > until
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> Based on our experience with the
> > last
> > > > > major
> > > > > > > > > > release, I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing
> BEFORE
> > > the
> > > > > > first
> > > > > > > > > > release
> > > > > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> > > > > candidates)
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> How do you guys feel about this?
> > > Should
> > > > we
> > > > > > > wait
> > > > > > > > a
> > > > > > > > > > few
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > days
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > >>>>>>>> release so that a few more
> features
> > > make
> > > > > it
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one
> > > hand,
> > > > > its
> > > > > > > > really
> > > > > > > > > > > nice
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats
> > up
> > > > some
> > > > > > > time
> > > > > > > > > and
> > > > > > > > > > > > causes
> > > > > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > > > > >>>>>>>> I would really like to have the
> Java
> > > API
> > > > > > Pojo
> > > > > > > > > > > > improvements
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> I think I can finish it until end
> of
> > > > this
> > > > > > > week.
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> I agree that the finished features
> > > > > > (especially
> > > > > > > > the
> > > > > > > > > > > Scala
> > > > > > > > > > > > > API)
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> a new release, but still I would
> like
> > > to
> > > > > > wait a
> > > > > > > > few
> > > > > > > > > > > more
> > > > > > > > > > > > > > days.
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>> Some of the missing features are on
> > the
> > > > > brink
> > > > > > > of
> > > > > > > > > > being
> > > > > > > > > > > > > > finished
> > > > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want
> > to
> > > > > > invest a
> > > > > > > > > week
> > > > > > > > > > in
> > > > > > > > > > > > bug
> > > > > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > > > >>>>>>> doing the release vote, when the
> new
> > > > > features
> > > > > > > are
> > > > > > > > > > > likely
> > > > > > > > > > > > to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the
> > > > > upcoming
> > > > > > > > > features
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > > > > >>>>>>> worth a release, so users can work
> > with
> > > > > them.
> > > > > > > ;)
> > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
I merged the Hadoop compat and POJOs, currently rebasing stephan's fault
tolerance ....

On Wed, Oct 8, 2014 at 10:50 AM, Fabian Hueske <fh...@apache.org> wrote:

> I think, the question is which PRs to merge into master before forking of
> the release candidate branch.
> You did already merge #126 into the master branch, so it will be in ;-)
>
> 2014-10-08 10:37 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
>
> > We can also include the blob manager. The corresponding PR is #126.
> >
> > On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <balassi.marton@gmail.com
> >
> > wrote:
> >
> > > Pushed the resolution for FLINK-1103 and the streaming bugfix commit.
> > >
> > > We are good for now on our side for an rc. Maybe I should add the
> > streaming
> > > connectors dependency fix that Stephan suggested here an the mailing
> list
> > > soon.
> > >
> > > Marton
> > >
> > > On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > We can add it and keep it initially undocumented (experimental) until
> > > > further tests...
> > > >
> > > > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <rmetzger@apache.org
> >
> > > > wrote:
> > > >
> > > > > I would add #142.
> > > > >
> > > > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > I agree, having a release candidate out would be nice.
> > > > > >
> > > > > > What is your opinion on issue #142? Fault tolerance is inactive
> by
> > > > > default,
> > > > > > but can be activated through the configuration.
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Oh. I didn't know this. The last mail from Gyula sounded like
> the
> > > > > > Streaming
> > > > > > > part is ready2release.
> > > > > > >
> > > > > > > My personal goal was to prepare a release candiate today so
> that
> > we
> > > > > have
> > > > > > a
> > > > > > > reference point against which we can test (which would also
> mean
> > > > > forking
> > > > > > of
> > > > > > > a 0.7-release branch and basically doing a feature freeze).
> > > > > > > I also though that it might be a good idea to have a release
> > > > candidate
> > > > > by
> > > > > > > tomorrow because it could be of use for the Hackathon tomorrow
> in
> > > > > > > Stockholm.
> > > > > > > But since the Scala POJO changes and the streaming examples are
> > > both
> > > > > > > outstanding, we can probably scratch that.
> > > > > > >
> > > > > > > Doing a fork, release candidate and a feature freeze doesn't
> mean
> > > we
> > > > > can
> > > > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > > > >
> > > > > > > What is your best-case scenario for FLINK-1103? (as far as I
> can
> > > see
> > > > > the
> > > > > > > only outstanding streaming feature for the release)
> > > > > > >
> > > > > > > Best,
> > > > > > > Robert
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > > > balassi.marton@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As for the streaming side we would like to push some bugfix
> > > commits
> > > > > and
> > > > > > > the
> > > > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > > > >
> > > > > > > > These are more or less ready, hopefully will be available at
> > the
> > > > end
> > > > > of
> > > > > > > > this week.
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > > > rmetzger@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I've merged the record api deprecation already.
> > > > > > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > > > > > >
> > > > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > > > >
> > > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > > > fhueske@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > So which PRs will be included in the candidate?
> > > > > > > > > >
> > > > > > > > > > #141 POJOs
> > > > > > > > > > #144 Deprecate Record API
> > > > > > > > > > #136 Fixed example quickstart
> > > > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function
> > > wrappers
> > > > > > > > (includes
> > > > > > > > > > PR #131)
> > > > > > > > > >
> > > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <
> sewen@apache.org
> > >:
> > > > > > > > > >
> > > > > > > > > > > Great news, looking forward to seeing this in the
> master!
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > > > rmetzger@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > As an update for everyone: My POJO feature is
> finished,
> > > > > > including
> > > > > > > > > > > > documentation.
> > > > > > > > > > > > Aljoscha is currently adopting the Scala API to have
> > > > support
> > > > > > for
> > > > > > > > > > (nested)
> > > > > > > > > > > > POJOs as well.
> > > > > > > > > > > > Once that is done, I'll merge everything and create a
> > > first
> > > > > > > > candidate
> > > > > > > > > > > that
> > > > > > > > > > > > we can use for testing.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I just checked the "Run Example" quickstart and it
> > > needs
> > > > a
> > > > > > bit
> > > > > > > of
> > > > > > > > > > work.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > > > rmetzger@apache.org
> > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We also should do a pass over our documentation,
> > the
> > > > > > > > quickstarts
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > website to see if everything is in a good state
> > (for
> > > > > > example
> > > > > > > > the
> > > > > > > > > > > > > > collection-based execution needs documentation as
> > > > well).
> > > > > We
> > > > > > > > > should
> > > > > > > > > > > also
> > > > > > > > > > > > > > finally document the hadoop-input format wrappers
> > (I
> > > > > think
> > > > > > > Timo
> > > > > > > > > is
> > > > > > > > > > > > > working
> > > > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > > > This page mentions the LocalDistributedExecutor
> and
> > > > > > contains
> > > > > > > > some
> > > > > > > > > > > (most
> > > > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106
> ).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > > > > > sewen@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > > > Last issues are going in and we started working
> > on
> > > > > > > > dependencies
> > > > > > > > > > and
> > > > > > > > > > > > > test
> > > > > > > > > > > > > > > platform diversity in order to make stabilizing
> > > > phase.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We should have an official feature freeze soon
> > and
> > > > fork
> > > > > > the
> > > > > > > > > > > > 0.7-release
> > > > > > > > > > > > > > > branch. I personally vote to include the POJO
> > > support
> > > > > (I
> > > > > > > > think
> > > > > > > > > > > Robert
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > sorting that one out and is close to
> completion),
> > > > and I
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > collection-based execution (today or tomorrow).
> > > Till
> > > > > has
> > > > > > > the
> > > > > > > > > BLOB
> > > > > > > > > > > > > manager
> > > > > > > > > > > > > > > ready, which would be good to include (better
> > > support
> > > > > > large
> > > > > > > > > > > libraries
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > > > > > gyfora@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So what is the current decision regarding the
> > > time
> > > > of
> > > > > > the
> > > > > > > > > > > upcoming
> > > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As for the streaming component, we included
> all
> > > the
> > > > > > > > features
> > > > > > > > > we
> > > > > > > > > > > > > wanted,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > will start to test everything tomorrow,
> making
> > > sure
> > > > > > that
> > > > > > > > all
> > > > > > > > > > > works
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > > > We are also almost finished with cleaning up
> > the
> > > > > > > connector
> > > > > > > > > > > > > dependencies
> > > > > > > > > > > > > > > > that Stephan pointed out, should be finished
> by
> > > > > > tomorrow.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > > > > fhueske@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha
> Krettek <
> > > > > > > > > > > aljoscha@apache.org
> > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan
> > > Ewen <
> > > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >>> I personally like the idea of SOFT
> > time-based
> > > > > > feature
> > > > > > > > > > > freezes.
> > > > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > > > >>> releases will get delayed again and
> again,
> > > > > because
> > > > > > of
> > > > > > > > > > > features
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Having reached the freeze point already,
> we
> > > > could
> > > > > > > still
> > > > > > > > > > > include
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>> features that are pending ready state in
> > the
> > > > next
> > > > > > > days
> > > > > > > > > > > > > (streaming,
> > > > > > > > > > > > > > > blob
> > > > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a
> > > > release
> > > > > > > > state.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> We had a mail listing the issues to
> include
> > > > into
> > > > > > 0.7,
> > > > > > > > > but a
> > > > > > > > > > > > wiki
> > > > > > > > > > > > > > page
> > > > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > > > >>> probably be better. In that sense, we
> could
> > > > start
> > > > > > > > > > collecting
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel
> > > Warneke" <
> > > > > > > > > > > > warneke@apache.org
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki
> page
> > > (or
> > > > > > > > something
> > > > > > > > > > > > > similar)
> > > > > > > > > > > > > > > > where
> > > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian
> > Hueske:
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline
> > might
> > > > not
> > > > > > be
> > > > > > > > the
> > > > > > > > > > best
> > > > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > > > > >>>>> We decide two (or three) weeks before a
> > > > > targeted
> > > > > > > > > release
> > > > > > > > > > > date
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are
> selected
> > > > for a
> > > > > > > > release
> > > > > > > > > > > > should
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > > > >>>>> or really close to completion (via
> > progress
> > > > > > > estimates
> > > > > > > > > in
> > > > > > > > > > > > JIRA).
> > > > > > > > > > > > > > > > >>>>> After we decided which JIRAs to include
> > in
> > > a
> > > > > > > release,
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > > >>>>> track the progress and dedicate another
> > > week
> > > > > > > > > exclusively
> > > > > > > > > > > for
> > > > > > > > > > > > > > > testing
> > > > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton
> > Balassi <
> > > > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> As for the streaming team we're also
> > > getting
> > > > > > ready
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > release,
> > > > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > > > >>>>>> couple of days will be needed to
> finish
> > > the
> > > > > > > features
> > > > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>    - A little work is still needed for
> > > > reduce
> > > > > > > > > operations
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > >>>>>>    groups/connected streams (any
> comment
> > > on
> > > > > > > Gyula's
> > > > > > > > > > recent
> > > > > > > > > > > > > > e-mail
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > > > >>>>>>    - The examples are being updated to
> > > match
> > > > > the
> > > > > > > > > > standard,
> > > > > > > > > > > > > check
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > > > >>>>>>
> > > streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja
> vu.
> > :)
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk
> > > > Celebi <
> > > > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert
> > Metzger <
> > > > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to
> > do
> > > a
> > > > > > > feature
> > > > > > > > > > freeze
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview".
> I
> > > > would
> > > > > > > > suggest
> > > > > > > > > to
> > > > > > > > > > > > ship
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in
> > the
> > > > > > > > > documentation.
> > > > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> > > > > > generation,
> > > > > > > > key
> > > > > > > > > > > > > selection
> > > > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the
> > Scala
> > > > API
> > > > > > in
> > > > > > > > sync
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This
> > > > feature
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > > > requested
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to
> > use
> > > > > > > > Collections
> > > > > > > > > > > > inside
> > > > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to
> > support
> > > > > > > > incremental
> > > > > > > > > > > > program
> > > > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> > > > assignments
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> The following features are in the
> > > master,
> > > > as
> > > > > > of
> > > > > > > > > today:
> > > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> We certainly need some days to test
> > > > > everything
> > > > > > > > until
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> Based on our experience with the
> last
> > > > major
> > > > > > > > > release, I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE
> > the
> > > > > first
> > > > > > > > > release
> > > > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> > > > candidates)
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> How do you guys feel about this?
> > Should
> > > we
> > > > > > wait
> > > > > > > a
> > > > > > > > > few
> > > > > > > > > > > more
> > > > > > > > > > > > > > days
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > >>>>>>>> release so that a few more features
> > make
> > > > it
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > > > release?
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one
> > hand,
> > > > its
> > > > > > > really
> > > > > > > > > > nice
> > > > > > > > > > > to
> > > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats
> up
> > > some
> > > > > > time
> > > > > > > > and
> > > > > > > > > > > causes
> > > > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > > > >>>>>>>> I would really like to have the Java
> > API
> > > > > Pojo
> > > > > > > > > > > improvements
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> I think I can finish it until end of
> > > this
> > > > > > week.
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> I agree that the finished features
> > > > > (especially
> > > > > > > the
> > > > > > > > > > Scala
> > > > > > > > > > > > API)
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>>>> a new release, but still I would like
> > to
> > > > > wait a
> > > > > > > few
> > > > > > > > > > more
> > > > > > > > > > > > > days.
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>> Some of the missing features are on
> the
> > > > brink
> > > > > > of
> > > > > > > > > being
> > > > > > > > > > > > > finished
> > > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want
> to
> > > > > invest a
> > > > > > > > week
> > > > > > > > > in
> > > > > > > > > > > bug
> > > > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > > >>>>>>> doing the release vote, when the new
> > > > features
> > > > > > are
> > > > > > > > > > likely
> > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the
> > > > upcoming
> > > > > > > > features
> > > > > > > > > > > will
> > > > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > > > >>>>>>> worth a release, so users can work
> with
> > > > them.
> > > > > > ;)
> > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
I think, the question is which PRs to merge into master before forking of
the release candidate branch.
You did already merge #126 into the master branch, so it will be in ;-)

2014-10-08 10:37 GMT+02:00 Till Rohrmann <ti...@gmail.com>:

> We can also include the blob manager. The corresponding PR is #126.
>
> On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Pushed the resolution for FLINK-1103 and the streaming bugfix commit.
> >
> > We are good for now on our side for an rc. Maybe I should add the
> streaming
> > connectors dependency fix that Stephan suggested here an the mailing list
> > soon.
> >
> > Marton
> >
> > On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > We can add it and keep it initially undocumented (experimental) until
> > > further tests...
> > >
> > > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > I would add #142.
> > > >
> > > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > I agree, having a release candidate out would be nice.
> > > > >
> > > > > What is your opinion on issue #142? Fault tolerance is inactive by
> > > > default,
> > > > > but can be activated through the configuration.
> > > > >
> > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Oh. I didn't know this. The last mail from Gyula sounded like the
> > > > > Streaming
> > > > > > part is ready2release.
> > > > > >
> > > > > > My personal goal was to prepare a release candiate today so that
> we
> > > > have
> > > > > a
> > > > > > reference point against which we can test (which would also mean
> > > > forking
> > > > > of
> > > > > > a 0.7-release branch and basically doing a feature freeze).
> > > > > > I also though that it might be a good idea to have a release
> > > candidate
> > > > by
> > > > > > tomorrow because it could be of use for the Hackathon tomorrow in
> > > > > > Stockholm.
> > > > > > But since the Scala POJO changes and the streaming examples are
> > both
> > > > > > outstanding, we can probably scratch that.
> > > > > >
> > > > > > Doing a fork, release candidate and a feature freeze doesn't mean
> > we
> > > > can
> > > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > > >
> > > > > > What is your best-case scenario for FLINK-1103? (as far as I can
> > see
> > > > the
> > > > > > only outstanding streaming feature for the release)
> > > > > >
> > > > > > Best,
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > > balassi.marton@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > As for the streaming side we would like to push some bugfix
> > commits
> > > > and
> > > > > > the
> > > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > > >
> > > > > > > These are more or less ready, hopefully will be available at
> the
> > > end
> > > > of
> > > > > > > this week.
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I've merged the record api deprecation already.
> > > > > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > > > > >
> > > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > > fhueske@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > So which PRs will be included in the candidate?
> > > > > > > > >
> > > > > > > > > #141 POJOs
> > > > > > > > > #144 Deprecate Record API
> > > > > > > > > #136 Fixed example quickstart
> > > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function
> > wrappers
> > > > > > > (includes
> > > > > > > > > PR #131)
> > > > > > > > >
> > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <sewen@apache.org
> >:
> > > > > > > > >
> > > > > > > > > > Great news, looking forward to seeing this in the master!
> > > > > > > > > >
> > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > > rmetzger@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As an update for everyone: My POJO feature is finished,
> > > > > including
> > > > > > > > > > > documentation.
> > > > > > > > > > > Aljoscha is currently adopting the Scala API to have
> > > support
> > > > > for
> > > > > > > > > (nested)
> > > > > > > > > > > POJOs as well.
> > > > > > > > > > > Once that is done, I'll merge everything and create a
> > first
> > > > > > > candidate
> > > > > > > > > > that
> > > > > > > > > > > we can use for testing.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > > > fhueske@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I just checked the "Run Example" quickstart and it
> > needs
> > > a
> > > > > bit
> > > > > > of
> > > > > > > > > work.
> > > > > > > > > > > >
> > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > > rmetzger@apache.org
> > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We also should do a pass over our documentation,
> the
> > > > > > > quickstarts
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > website to see if everything is in a good state
> (for
> > > > > example
> > > > > > > the
> > > > > > > > > > > > > collection-based execution needs documentation as
> > > well).
> > > > We
> > > > > > > > should
> > > > > > > > > > also
> > > > > > > > > > > > > finally document the hadoop-input format wrappers
> (I
> > > > think
> > > > > > Timo
> > > > > > > > is
> > > > > > > > > > > > working
> > > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > > This page mentions the LocalDistributedExecutor and
> > > > > contains
> > > > > > > some
> > > > > > > > > > (most
> > > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > > >
> > > > > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > > > > sewen@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > > Last issues are going in and we started working
> on
> > > > > > > dependencies
> > > > > > > > > and
> > > > > > > > > > > > test
> > > > > > > > > > > > > > platform diversity in order to make stabilizing
> > > phase.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We should have an official feature freeze soon
> and
> > > fork
> > > > > the
> > > > > > > > > > > 0.7-release
> > > > > > > > > > > > > > branch. I personally vote to include the POJO
> > support
> > > > (I
> > > > > > > think
> > > > > > > > > > Robert
> > > > > > > > > > > > is
> > > > > > > > > > > > > > sorting that one out and is close to completion),
> > > and I
> > > > > > want
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > the
> > > > > > > > > > > > > > collection-based execution (today or tomorrow).
> > Till
> > > > has
> > > > > > the
> > > > > > > > BLOB
> > > > > > > > > > > > manager
> > > > > > > > > > > > > > ready, which would be good to include (better
> > support
> > > > > large
> > > > > > > > > > libraries
> > > > > > > > > > > > or
> > > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > > > > gyfora@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So what is the current decision regarding the
> > time
> > > of
> > > > > the
> > > > > > > > > > upcoming
> > > > > > > > > > > > > > release?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As for the streaming component, we included all
> > the
> > > > > > > features
> > > > > > > > we
> > > > > > > > > > > > wanted,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > will start to test everything tomorrow, making
> > sure
> > > > > that
> > > > > > > all
> > > > > > > > > > works
> > > > > > > > > > > as
> > > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > > We are also almost finished with cleaning up
> the
> > > > > > connector
> > > > > > > > > > > > dependencies
> > > > > > > > > > > > > > > that Stephan pointed out, should be finished by
> > > > > tomorrow.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > > > fhueske@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > > > > > > aljoscha@apache.org
> > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan
> > Ewen <
> > > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >>> I personally like the idea of SOFT
> time-based
> > > > > feature
> > > > > > > > > > freezes.
> > > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > > >>> releases will get delayed again and again,
> > > > because
> > > > > of
> > > > > > > > > > features
> > > > > > > > > > > > that
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Having reached the freeze point already, we
> > > could
> > > > > > still
> > > > > > > > > > include
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > >>> features that are pending ready state in
> the
> > > next
> > > > > > days
> > > > > > > > > > > > (streaming,
> > > > > > > > > > > > > > blob
> > > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a
> > > release
> > > > > > > state.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> We had a mail listing the issues to include
> > > into
> > > > > 0.7,
> > > > > > > > but a
> > > > > > > > > > > wiki
> > > > > > > > > > > > > page
> > > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > > >>> probably be better. In that sense, we could
> > > start
> > > > > > > > > collecting
> > > > > > > > > > > the
> > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel
> > Warneke" <
> > > > > > > > > > > warneke@apache.org
> > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page
> > (or
> > > > > > > something
> > > > > > > > > > > > similar)
> > > > > > > > > > > > > > > where
> > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian
> Hueske:
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline
> might
> > > not
> > > > > be
> > > > > > > the
> > > > > > > > > best
> > > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > > > >>>>> We decide two (or three) weeks before a
> > > > targeted
> > > > > > > > release
> > > > > > > > > > date
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected
> > > for a
> > > > > > > release
> > > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > > >>>>> or really close to completion (via
> progress
> > > > > > estimates
> > > > > > > > in
> > > > > > > > > > > JIRA).
> > > > > > > > > > > > > > > >>>>> After we decided which JIRAs to include
> in
> > a
> > > > > > release,
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > use
> > > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > >>>>> track the progress and dedicate another
> > week
> > > > > > > > exclusively
> > > > > > > > > > for
> > > > > > > > > > > > > > testing
> > > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton
> Balassi <
> > > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> As for the streaming team we're also
> > getting
> > > > > ready
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > release,
> > > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > > >>>>>> couple of days will be needed to finish
> > the
> > > > > > features
> > > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>    - A little work is still needed for
> > > reduce
> > > > > > > > operations
> > > > > > > > > > and
> > > > > > > > > > > > > > > >>>>>>    groups/connected streams (any comment
> > on
> > > > > > Gyula's
> > > > > > > > > recent
> > > > > > > > > > > > > e-mail
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > > >>>>>>    - The examples are being updated to
> > match
> > > > the
> > > > > > > > > standard,
> > > > > > > > > > > > check
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>
> > > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > > >>>>>>
> > > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > > >>>>>>
> > > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > > >>>>>>
> > streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu.
> :)
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk
> > > Celebi <
> > > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert
> Metzger <
> > > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to
> do
> > a
> > > > > > feature
> > > > > > > > > freeze
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I
> > > would
> > > > > > > suggest
> > > > > > > > to
> > > > > > > > > > > ship
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in
> the
> > > > > > > > documentation.
> > > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> > > > > generation,
> > > > > > > key
> > > > > > > > > > > > selection
> > > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the
> Scala
> > > API
> > > > > in
> > > > > > > sync
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This
> > > feature
> > > > > has
> > > > > > > been
> > > > > > > > > > > > requested
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to
> use
> > > > > > > Collections
> > > > > > > > > > > inside
> > > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to
> support
> > > > > > > incremental
> > > > > > > > > > > program
> > > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> > > assignments
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> The following features are in the
> > master,
> > > as
> > > > > of
> > > > > > > > today:
> > > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> We certainly need some days to test
> > > > everything
> > > > > > > until
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> Based on our experience with the last
> > > major
> > > > > > > > release, I
> > > > > > > > > > > would
> > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE
> the
> > > > first
> > > > > > > > release
> > > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> > > candidates)
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> How do you guys feel about this?
> Should
> > we
> > > > > wait
> > > > > > a
> > > > > > > > few
> > > > > > > > > > more
> > > > > > > > > > > > > days
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > >>>>>>>> release so that a few more features
> make
> > > it
> > > > > into
> > > > > > > the
> > > > > > > > > > > > release?
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one
> hand,
> > > its
> > > > > > really
> > > > > > > > > nice
> > > > > > > > > > to
> > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up
> > some
> > > > > time
> > > > > > > and
> > > > > > > > > > causes
> > > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > > >>>>>>>> I would really like to have the Java
> API
> > > > Pojo
> > > > > > > > > > improvements
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> I think I can finish it until end of
> > this
> > > > > week.
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > >>>>>>> I agree that the finished features
> > > > (especially
> > > > > > the
> > > > > > > > > Scala
> > > > > > > > > > > API)
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>>>> a new release, but still I would like
> to
> > > > wait a
> > > > > > few
> > > > > > > > > more
> > > > > > > > > > > > days.
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>> Some of the missing features are on the
> > > brink
> > > > > of
> > > > > > > > being
> > > > > > > > > > > > finished
> > > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to
> > > > invest a
> > > > > > > week
> > > > > > > > in
> > > > > > > > > > bug
> > > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > >>>>>>> doing the release vote, when the new
> > > features
> > > > > are
> > > > > > > > > likely
> > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the
> > > upcoming
> > > > > > > features
> > > > > > > > > > will
> > > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > > >>>>>>> worth a release, so users can work with
> > > them.
> > > > > ;)
> > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Till Rohrmann <ti...@gmail.com>.
We can also include the blob manager. The corresponding PR is #126.

On Wed, Oct 8, 2014 at 1:51 AM, Márton Balassi <ba...@gmail.com>
wrote:

> Pushed the resolution for FLINK-1103 and the streaming bugfix commit.
>
> We are good for now on our side for an rc. Maybe I should add the streaming
> connectors dependency fix that Stephan suggested here an the mailing list
> soon.
>
> Marton
>
> On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > We can add it and keep it initially undocumented (experimental) until
> > further tests...
> >
> > On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > I would add #142.
> > >
> > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > I agree, having a release candidate out would be nice.
> > > >
> > > > What is your opinion on issue #142? Fault tolerance is inactive by
> > > default,
> > > > but can be activated through the configuration.
> > > >
> > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > Oh. I didn't know this. The last mail from Gyula sounded like the
> > > > Streaming
> > > > > part is ready2release.
> > > > >
> > > > > My personal goal was to prepare a release candiate today so that we
> > > have
> > > > a
> > > > > reference point against which we can test (which would also mean
> > > forking
> > > > of
> > > > > a 0.7-release branch and basically doing a feature freeze).
> > > > > I also though that it might be a good idea to have a release
> > candidate
> > > by
> > > > > tomorrow because it could be of use for the Hackathon tomorrow in
> > > > > Stockholm.
> > > > > But since the Scala POJO changes and the streaming examples are
> both
> > > > > outstanding, we can probably scratch that.
> > > > >
> > > > > Doing a fork, release candidate and a feature freeze doesn't mean
> we
> > > can
> > > > > not supply bugfixes anymore for the 0.7-release ;)
> > > > >
> > > > > What is your best-case scenario for FLINK-1103? (as far as I can
> see
> > > the
> > > > > only outstanding streaming feature for the release)
> > > > >
> > > > > Best,
> > > > > Robert
> > > > >
> > > > >
> > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > As for the streaming side we would like to push some bugfix
> commits
> > > and
> > > > > the
> > > > > > resolution of the FLINK-1103 JIRA issue.
> > > > > >
> > > > > > These are more or less ready, hopefully will be available at the
> > end
> > > of
> > > > > > this week.
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > I've merged the record api deprecation already.
> > > > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > > > >
> > > > > > > We certainly should merge #136 and #143 as well.
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> > fhueske@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > So which PRs will be included in the candidate?
> > > > > > > >
> > > > > > > > #141 POJOs
> > > > > > > > #144 Deprecate Record API
> > > > > > > > #136 Fixed example quickstart
> > > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function
> wrappers
> > > > > > (includes
> > > > > > > > PR #131)
> > > > > > > >
> > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > > > >
> > > > > > > > > Great news, looking forward to seeing this in the master!
> > > > > > > > >
> > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > > rmetzger@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As an update for everyone: My POJO feature is finished,
> > > > including
> > > > > > > > > > documentation.
> > > > > > > > > > Aljoscha is currently adopting the Scala API to have
> > support
> > > > for
> > > > > > > > (nested)
> > > > > > > > > > POJOs as well.
> > > > > > > > > > Once that is done, I'll merge everything and create a
> first
> > > > > > candidate
> > > > > > > > > that
> > > > > > > > > > we can use for testing.
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > > fhueske@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I just checked the "Run Example" quickstart and it
> needs
> > a
> > > > bit
> > > > > of
> > > > > > > > work.
> > > > > > > > > > >
> > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > > rmetzger@apache.org
> > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > > >
> > > > > > > > > > > > We also should do a pass over our documentation, the
> > > > > > quickstarts
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > website to see if everything is in a good state (for
> > > > example
> > > > > > the
> > > > > > > > > > > > collection-based execution needs documentation as
> > well).
> > > We
> > > > > > > should
> > > > > > > > > also
> > > > > > > > > > > > finally document the hadoop-input format wrappers (I
> > > think
> > > > > Timo
> > > > > > > is
> > > > > > > > > > > working
> > > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > > This page mentions the LocalDistributedExecutor and
> > > > contains
> > > > > > some
> > > > > > > > > (most
> > > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > > >
> > > > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > > > sewen@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > > Last issues are going in and we started working on
> > > > > > dependencies
> > > > > > > > and
> > > > > > > > > > > test
> > > > > > > > > > > > > platform diversity in order to make stabilizing
> > phase.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We should have an official feature freeze soon and
> > fork
> > > > the
> > > > > > > > > > 0.7-release
> > > > > > > > > > > > > branch. I personally vote to include the POJO
> support
> > > (I
> > > > > > think
> > > > > > > > > Robert
> > > > > > > > > > > is
> > > > > > > > > > > > > sorting that one out and is close to completion),
> > and I
> > > > > want
> > > > > > to
> > > > > > > > add
> > > > > > > > > > the
> > > > > > > > > > > > > collection-based execution (today or tomorrow).
> Till
> > > has
> > > > > the
> > > > > > > BLOB
> > > > > > > > > > > manager
> > > > > > > > > > > > > ready, which would be good to include (better
> support
> > > > large
> > > > > > > > > libraries
> > > > > > > > > > > or
> > > > > > > > > > > > > fat jars).
> > > > > > > > > > > > >
> > > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > > > gyfora@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So what is the current decision regarding the
> time
> > of
> > > > the
> > > > > > > > > upcoming
> > > > > > > > > > > > > release?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As for the streaming component, we included all
> the
> > > > > > features
> > > > > > > we
> > > > > > > > > > > wanted,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > will start to test everything tomorrow, making
> sure
> > > > that
> > > > > > all
> > > > > > > > > works
> > > > > > > > > > as
> > > > > > > > > > > > > > intended.
> > > > > > > > > > > > > > We are also almost finished with cleaning up the
> > > > > connector
> > > > > > > > > > > dependencies
> > > > > > > > > > > > > > that Stephan pointed out, should be finished by
> > > > tomorrow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Gyula
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > > fhueske@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > > > > > aljoscha@apache.org
> > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan
> Ewen <
> > > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>> I personally like the idea of SOFT time-based
> > > > feature
> > > > > > > > > freezes.
> > > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > > >>> releases will get delayed again and again,
> > > because
> > > > of
> > > > > > > > > features
> > > > > > > > > > > that
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Having reached the freeze point already, we
> > could
> > > > > still
> > > > > > > > > include
> > > > > > > > > > > the
> > > > > > > > > > > > > > >>> features that are pending ready state in the
> > next
> > > > > days
> > > > > > > > > > > (streaming,
> > > > > > > > > > > > > blob
> > > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a
> > release
> > > > > > state.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> We had a mail listing the issues to include
> > into
> > > > 0.7,
> > > > > > > but a
> > > > > > > > > > wiki
> > > > > > > > > > > > page
> > > > > > > > > > > > > > >> would
> > > > > > > > > > > > > > >>> probably be better. In that sense, we could
> > start
> > > > > > > > collecting
> > > > > > > > > > the
> > > > > > > > > > > > > issues
> > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel
> Warneke" <
> > > > > > > > > > warneke@apache.org
> > > > > > > > > > > >:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page
> (or
> > > > > > something
> > > > > > > > > > > similar)
> > > > > > > > > > > > > > where
> > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline might
> > not
> > > > be
> > > > > > the
> > > > > > > > best
> > > > > > > > > > > > > practice.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > > >>>>> We decide two (or three) weeks before a
> > > targeted
> > > > > > > release
> > > > > > > > > date
> > > > > > > > > > > > about
> > > > > > > > > > > > > > >> which
> > > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected
> > for a
> > > > > > release
> > > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > > >>>>> or really close to completion (via progress
> > > > > estimates
> > > > > > > in
> > > > > > > > > > JIRA).
> > > > > > > > > > > > > > >>>>> After we decided which JIRAs to include in
> a
> > > > > release,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > use
> > > > > > > > > > > > > JIRA
> > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > >>>>> track the progress and dedicate another
> week
> > > > > > > exclusively
> > > > > > > > > for
> > > > > > > > > > > > > testing
> > > > > > > > > > > > > > >> after
> > > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> As for the streaming team we're also
> getting
> > > > ready
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > release,
> > > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > > >>>>>> couple of days will be needed to finish
> the
> > > > > features
> > > > > > > > that
> > > > > > > > > we
> > > > > > > > > > > > would
> > > > > > > > > > > > > > >> like
> > > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>    - A little work is still needed for
> > reduce
> > > > > > > operations
> > > > > > > > > and
> > > > > > > > > > > > > > >>>>>>    groups/connected streams (any comment
> on
> > > > > Gyula's
> > > > > > > > recent
> > > > > > > > > > > > e-mail
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > > >>>>>>    - The examples are being updated to
> match
> > > the
> > > > > > > > standard,
> > > > > > > > > > > check
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > > >>>>>>
> > > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > > >>>>>>
> > > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > > >>>>>>
> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk
> > Celebi <
> > > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > > > > > > rmetzger@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do
> a
> > > > > feature
> > > > > > > > freeze
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I
> > would
> > > > > > suggest
> > > > > > > to
> > > > > > > > > > ship
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > > > > > > documentation.
> > > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> > > > generation,
> > > > > > key
> > > > > > > > > > > selection
> > > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala
> > API
> > > > in
> > > > > > sync
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > > >>>>>>>>
> > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > > >>>>>>>>
> > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This
> > feature
> > > > has
> > > > > > been
> > > > > > > > > > > requested
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > >> many
> > > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> > > > > > Collections
> > > > > > > > > > inside
> > > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > > >>>>>>>>
> > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> > > > > > incremental
> > > > > > > > > > program
> > > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> > assignments
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> The following features are in the
> master,
> > as
> > > > of
> > > > > > > today:
> > > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> We certainly need some days to test
> > > everything
> > > > > > until
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > > start
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>>> Based on our experience with the last
> > major
> > > > > > > release, I
> > > > > > > > > > would
> > > > > > > > > > > > > > really
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the
> > > first
> > > > > > > release
> > > > > > > > > > > > candidate.
> > > > > > > > > > > > > > For
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> > candidates)
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> How do you guys feel about this? Should
> we
> > > > wait
> > > > > a
> > > > > > > few
> > > > > > > > > more
> > > > > > > > > > > > days
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >>>>>>>> release so that a few more features make
> > it
> > > > into
> > > > > > the
> > > > > > > > > > > release?
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand,
> > its
> > > > > really
> > > > > > > > nice
> > > > > > > > > to
> > > > > > > > > > > > > release
> > > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up
> some
> > > > time
> > > > > > and
> > > > > > > > > causes
> > > > > > > > > > > > > > overhead
> > > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > > >>>>>>>> I would really like to have the Java API
> > > Pojo
> > > > > > > > > improvements
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>>> I think I can finish it until end of
> this
> > > > week.
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>> I agree that the finished features
> > > (especially
> > > > > the
> > > > > > > > Scala
> > > > > > > > > > API)
> > > > > > > > > > > > are
> > > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>> a new release, but still I would like to
> > > wait a
> > > > > few
> > > > > > > > more
> > > > > > > > > > > days.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> Some of the missing features are on the
> > brink
> > > > of
> > > > > > > being
> > > > > > > > > > > finished
> > > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to
> > > invest a
> > > > > > week
> > > > > > > in
> > > > > > > > > bug
> > > > > > > > > > > > > fixing
> > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > >>>>>>> doing the release vote, when the new
> > features
> > > > are
> > > > > > > > likely
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the
> > upcoming
> > > > > > features
> > > > > > > > > will
> > > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > > >>>>>>> worth a release, so users can work with
> > them.
> > > > ;)
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Márton Balassi <ba...@gmail.com>.
Pushed the resolution for FLINK-1103 and the streaming bugfix commit.

We are good for now on our side for an rc. Maybe I should add the streaming
connectors dependency fix that Stephan suggested here an the mailing list
soon.

Marton

On Tue, Oct 7, 2014 at 11:44 PM, Stephan Ewen <se...@apache.org> wrote:

> We can add it and keep it initially undocumented (experimental) until
> further tests...
>
> On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > I would add #142.
> >
> > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > I agree, having a release candidate out would be nice.
> > >
> > > What is your opinion on issue #142? Fault tolerance is inactive by
> > default,
> > > but can be activated through the configuration.
> > >
> > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Oh. I didn't know this. The last mail from Gyula sounded like the
> > > Streaming
> > > > part is ready2release.
> > > >
> > > > My personal goal was to prepare a release candiate today so that we
> > have
> > > a
> > > > reference point against which we can test (which would also mean
> > forking
> > > of
> > > > a 0.7-release branch and basically doing a feature freeze).
> > > > I also though that it might be a good idea to have a release
> candidate
> > by
> > > > tomorrow because it could be of use for the Hackathon tomorrow in
> > > > Stockholm.
> > > > But since the Scala POJO changes and the streaming examples are both
> > > > outstanding, we can probably scratch that.
> > > >
> > > > Doing a fork, release candidate and a feature freeze doesn't mean we
> > can
> > > > not supply bugfixes anymore for the 0.7-release ;)
> > > >
> > > > What is your best-case scenario for FLINK-1103? (as far as I can see
> > the
> > > > only outstanding streaming feature for the release)
> > > >
> > > > Best,
> > > > Robert
> > > >
> > > >
> > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> > balassi.marton@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > As for the streaming side we would like to push some bugfix commits
> > and
> > > > the
> > > > > resolution of the FLINK-1103 JIRA issue.
> > > > >
> > > > > These are more or less ready, hopefully will be available at the
> end
> > of
> > > > > this week.
> > > > >
> > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > I've merged the record api deprecation already.
> > > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > > >
> > > > > > We certainly should merge #136 and #143 as well.
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <
> fhueske@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > So which PRs will be included in the candidate?
> > > > > > >
> > > > > > > #141 POJOs
> > > > > > > #144 Deprecate Record API
> > > > > > > #136 Fixed example quickstart
> > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> > > > > (includes
> > > > > > > PR #131)
> > > > > > >
> > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > > >
> > > > > > > > Great news, looking forward to seeing this in the master!
> > > > > > > >
> > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > > rmetzger@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As an update for everyone: My POJO feature is finished,
> > > including
> > > > > > > > > documentation.
> > > > > > > > > Aljoscha is currently adopting the Scala API to have
> support
> > > for
> > > > > > > (nested)
> > > > > > > > > POJOs as well.
> > > > > > > > > Once that is done, I'll merge everything and create a first
> > > > > candidate
> > > > > > > > that
> > > > > > > > > we can use for testing.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > > fhueske@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I just checked the "Run Example" quickstart and it needs
> a
> > > bit
> > > > of
> > > > > > > work.
> > > > > > > > > >
> > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > > rmetzger@apache.org
> > > > >:
> > > > > > > > > >
> > > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > > >
> > > > > > > > > > > We also should do a pass over our documentation, the
> > > > > quickstarts
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > website to see if everything is in a good state (for
> > > example
> > > > > the
> > > > > > > > > > > collection-based execution needs documentation as
> well).
> > We
> > > > > > should
> > > > > > > > also
> > > > > > > > > > > finally document the hadoop-input format wrappers (I
> > think
> > > > Timo
> > > > > > is
> > > > > > > > > > working
> > > > > > > > > > > on a pull request for that).
> > > > > > > > > > > This page mentions the LocalDistributedExecutor and
> > > contains
> > > > > some
> > > > > > > > (most
> > > > > > > > > > > likely outdated) scala code:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > > >
> > > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > > sewen@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > > Last issues are going in and we started working on
> > > > > dependencies
> > > > > > > and
> > > > > > > > > > test
> > > > > > > > > > > > platform diversity in order to make stabilizing
> phase.
> > > > > > > > > > > >
> > > > > > > > > > > > We should have an official feature freeze soon and
> fork
> > > the
> > > > > > > > > 0.7-release
> > > > > > > > > > > > branch. I personally vote to include the POJO support
> > (I
> > > > > think
> > > > > > > > Robert
> > > > > > > > > > is
> > > > > > > > > > > > sorting that one out and is close to completion),
> and I
> > > > want
> > > > > to
> > > > > > > add
> > > > > > > > > the
> > > > > > > > > > > > collection-based execution (today or tomorrow). Till
> > has
> > > > the
> > > > > > BLOB
> > > > > > > > > > manager
> > > > > > > > > > > > ready, which would be good to include (better support
> > > large
> > > > > > > > libraries
> > > > > > > > > > or
> > > > > > > > > > > > fat jars).
> > > > > > > > > > > >
> > > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > > gyfora@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey,
> > > > > > > > > > > > >
> > > > > > > > > > > > > So what is the current decision regarding the time
> of
> > > the
> > > > > > > > upcoming
> > > > > > > > > > > > release?
> > > > > > > > > > > > >
> > > > > > > > > > > > > As for the streaming component, we included all the
> > > > > features
> > > > > > we
> > > > > > > > > > wanted,
> > > > > > > > > > > > we
> > > > > > > > > > > > > will start to test everything tomorrow, making sure
> > > that
> > > > > all
> > > > > > > > works
> > > > > > > > > as
> > > > > > > > > > > > > intended.
> > > > > > > > > > > > > We are also almost finished with cleaning up the
> > > > connector
> > > > > > > > > > dependencies
> > > > > > > > > > > > > that Stephan pointed out, should be finished by
> > > tomorrow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Gyula
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > > fhueske@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > > > > aljoscha@apache.org
> > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > > > > > > sewen@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >>> I personally like the idea of SOFT time-based
> > > feature
> > > > > > > > freezes.
> > > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > > >>> releases will get delayed again and again,
> > because
> > > of
> > > > > > > > features
> > > > > > > > > > that
> > > > > > > > > > > > we
> > > > > > > > > > > > > >> try
> > > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Having reached the freeze point already, we
> could
> > > > still
> > > > > > > > include
> > > > > > > > > > the
> > > > > > > > > > > > > >>> features that are pending ready state in the
> next
> > > > days
> > > > > > > > > > (streaming,
> > > > > > > > > > > > blob
> > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a
> release
> > > > > state.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> We had a mail listing the issues to include
> into
> > > 0.7,
> > > > > > but a
> > > > > > > > > wiki
> > > > > > > > > > > page
> > > > > > > > > > > > > >> would
> > > > > > > > > > > > > >>> probably be better. In that sense, we could
> start
> > > > > > > collecting
> > > > > > > > > the
> > > > > > > > > > > > issues
> > > > > > > > > > > > > >> for
> > > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > > > > > > warneke@apache.org
> > > > > > > > > > >:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or
> > > > > something
> > > > > > > > > > similar)
> > > > > > > > > > > > > where
> > > > > > > > > > > > > >> we
> > > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline might
> not
> > > be
> > > > > the
> > > > > > > best
> > > > > > > > > > > > practice.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > > >>>>> We decide two (or three) weeks before a
> > targeted
> > > > > > release
> > > > > > > > date
> > > > > > > > > > > about
> > > > > > > > > > > > > >> which
> > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected
> for a
> > > > > release
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > > >>>>> or really close to completion (via progress
> > > > estimates
> > > > > > in
> > > > > > > > > JIRA).
> > > > > > > > > > > > > >>>>> After we decided which JIRAs to include in a
> > > > release,
> > > > > > we
> > > > > > > > can
> > > > > > > > > > use
> > > > > > > > > > > > JIRA
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >>>>> track the progress and dedicate another week
> > > > > > exclusively
> > > > > > > > for
> > > > > > > > > > > > testing
> > > > > > > > > > > > > >> after
> > > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> As for the streaming team we're also getting
> > > ready
> > > > > for
> > > > > > > the
> > > > > > > > > > > release,
> > > > > > > > > > > > > >> but a
> > > > > > > > > > > > > >>>>>> couple of days will be needed to finish the
> > > > features
> > > > > > > that
> > > > > > > > we
> > > > > > > > > > > would
> > > > > > > > > > > > > >> like
> > > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>    - A little work is still needed for
> reduce
> > > > > > operations
> > > > > > > > and
> > > > > > > > > > > > > >>>>>>    groups/connected streams (any comment on
> > > > Gyula's
> > > > > > > recent
> > > > > > > > > > > e-mail
> > > > > > > > > > > > is
> > > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > > >>>>>>    - The examples are being updated to match
> > the
> > > > > > > standard,
> > > > > > > > > > check
> > > > > > > > > > > > out
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>
> > > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > > >>>>>>
> > > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > > >>>>>>
> > > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk
> Celebi <
> > > > > > > > > uce@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > > > > > rmetzger@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a
> > > > feature
> > > > > > > freeze
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I
> would
> > > > > suggest
> > > > > > to
> > > > > > > > > ship
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > > > > > documentation.
> > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> > > generation,
> > > > > key
> > > > > > > > > > selection
> > > > > > > > > > > > > >> using a
> > > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala
> API
> > > in
> > > > > sync
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > >> latest
> > > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > > >>>>>>>>
> > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > > >>>>>>>>
> > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This
> feature
> > > has
> > > > > been
> > > > > > > > > > requested
> > > > > > > > > > > > by
> > > > > > > > > > > > > >> many
> > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> > > > > Collections
> > > > > > > > > inside
> > > > > > > > > > > > POJOs:
> > > > > > > > > > > > > >>>>>>>>
> > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> > > > > incremental
> > > > > > > > > program
> > > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory
> assignments
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> The following features are in the master,
> as
> > > of
> > > > > > today:
> > > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> We certainly need some days to test
> > everything
> > > > > until
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > > start
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>>> Based on our experience with the last
> major
> > > > > > release, I
> > > > > > > > > would
> > > > > > > > > > > > > really
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the
> > first
> > > > > > release
> > > > > > > > > > > candidate.
> > > > > > > > > > > > > For
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6
> candidates)
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we
> > > wait
> > > > a
> > > > > > few
> > > > > > > > more
> > > > > > > > > > > days
> > > > > > > > > > > > > for
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >>>>>>>> release so that a few more features make
> it
> > > into
> > > > > the
> > > > > > > > > > release?
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand,
> its
> > > > really
> > > > > > > nice
> > > > > > > > to
> > > > > > > > > > > > release
> > > > > > > > > > > > > >> on a
> > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some
> > > time
> > > > > and
> > > > > > > > causes
> > > > > > > > > > > > > overhead
> > > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > > >>>>>>>> I would really like to have the Java API
> > Pojo
> > > > > > > > improvements
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>>> I think I can finish it until end of this
> > > week.
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>> I agree that the finished features
> > (especially
> > > > the
> > > > > > > Scala
> > > > > > > > > API)
> > > > > > > > > > > are
> > > > > > > > > > > > > >> nice
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>> a new release, but still I would like to
> > wait a
> > > > few
> > > > > > > more
> > > > > > > > > > days.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> Some of the missing features are on the
> brink
> > > of
> > > > > > being
> > > > > > > > > > finished
> > > > > > > > > > > > > (e.g.
> > > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to
> > invest a
> > > > > week
> > > > > > in
> > > > > > > > bug
> > > > > > > > > > > > fixing
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >>>>>>> doing the release vote, when the new
> features
> > > are
> > > > > > > likely
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > >> finished
> > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the
> upcoming
> > > > > features
> > > > > > > > will
> > > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > > >>>>>>> worth a release, so users can work with
> them.
> > > ;)
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
We can add it and keep it initially undocumented (experimental) until
further tests...

On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <rm...@apache.org> wrote:

> I would add #142.
>
> On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > I agree, having a release candidate out would be nice.
> >
> > What is your opinion on issue #142? Fault tolerance is inactive by
> default,
> > but can be activated through the configuration.
> >
> > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Oh. I didn't know this. The last mail from Gyula sounded like the
> > Streaming
> > > part is ready2release.
> > >
> > > My personal goal was to prepare a release candiate today so that we
> have
> > a
> > > reference point against which we can test (which would also mean
> forking
> > of
> > > a 0.7-release branch and basically doing a feature freeze).
> > > I also though that it might be a good idea to have a release candidate
> by
> > > tomorrow because it could be of use for the Hackathon tomorrow in
> > > Stockholm.
> > > But since the Scala POJO changes and the streaming examples are both
> > > outstanding, we can probably scratch that.
> > >
> > > Doing a fork, release candidate and a feature freeze doesn't mean we
> can
> > > not supply bugfixes anymore for the 0.7-release ;)
> > >
> > > What is your best-case scenario for FLINK-1103? (as far as I can see
> the
> > > only outstanding streaming feature for the release)
> > >
> > > Best,
> > > Robert
> > >
> > >
> > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <
> balassi.marton@gmail.com
> > >
> > > wrote:
> > >
> > > > As for the streaming side we would like to push some bugfix commits
> and
> > > the
> > > > resolution of the FLINK-1103 JIRA issue.
> > > >
> > > > These are more or less ready, hopefully will be available at the end
> of
> > > > this week.
> > > >
> > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > I've merged the record api deprecation already.
> > > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > > >
> > > > > We certainly should merge #136 and #143 as well.
> > > > >
> > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org>
> > > > wrote:
> > > > >
> > > > > > So which PRs will be included in the candidate?
> > > > > >
> > > > > > #141 POJOs
> > > > > > #144 Deprecate Record API
> > > > > > #136 Fixed example quickstart
> > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> > > > (includes
> > > > > > PR #131)
> > > > > >
> > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > >
> > > > > > > Great news, looking forward to seeing this in the master!
> > > > > > >
> > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As an update for everyone: My POJO feature is finished,
> > including
> > > > > > > > documentation.
> > > > > > > > Aljoscha is currently adopting the Scala API to have support
> > for
> > > > > > (nested)
> > > > > > > > POJOs as well.
> > > > > > > > Once that is done, I'll merge everything and create a first
> > > > candidate
> > > > > > > that
> > > > > > > > we can use for testing.
> > > > > > > >
> > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > > fhueske@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I just checked the "Run Example" quickstart and it needs a
> > bit
> > > of
> > > > > > work.
> > > > > > > > >
> > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> > rmetzger@apache.org
> > > >:
> > > > > > > > >
> > > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > > >
> > > > > > > > > > We also should do a pass over our documentation, the
> > > > quickstarts
> > > > > > and
> > > > > > > > the
> > > > > > > > > > website to see if everything is in a good state (for
> > example
> > > > the
> > > > > > > > > > collection-based execution needs documentation as well).
> We
> > > > > should
> > > > > > > also
> > > > > > > > > > finally document the hadoop-input format wrappers (I
> think
> > > Timo
> > > > > is
> > > > > > > > > working
> > > > > > > > > > on a pull request for that).
> > > > > > > > > > This page mentions the LocalDistributedExecutor and
> > contains
> > > > some
> > > > > > > (most
> > > > > > > > > > likely outdated) scala code:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > > >
> > > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > > sewen@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > > Last issues are going in and we started working on
> > > > dependencies
> > > > > > and
> > > > > > > > > test
> > > > > > > > > > > platform diversity in order to make stabilizing phase.
> > > > > > > > > > >
> > > > > > > > > > > We should have an official feature freeze soon and fork
> > the
> > > > > > > > 0.7-release
> > > > > > > > > > > branch. I personally vote to include the POJO support
> (I
> > > > think
> > > > > > > Robert
> > > > > > > > > is
> > > > > > > > > > > sorting that one out and is close to completion), and I
> > > want
> > > > to
> > > > > > add
> > > > > > > > the
> > > > > > > > > > > collection-based execution (today or tomorrow). Till
> has
> > > the
> > > > > BLOB
> > > > > > > > > manager
> > > > > > > > > > > ready, which would be good to include (better support
> > large
> > > > > > > libraries
> > > > > > > > > or
> > > > > > > > > > > fat jars).
> > > > > > > > > > >
> > > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > > gyfora@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey,
> > > > > > > > > > > >
> > > > > > > > > > > > So what is the current decision regarding the time of
> > the
> > > > > > > upcoming
> > > > > > > > > > > release?
> > > > > > > > > > > >
> > > > > > > > > > > > As for the streaming component, we included all the
> > > > features
> > > > > we
> > > > > > > > > wanted,
> > > > > > > > > > > we
> > > > > > > > > > > > will start to test everything tomorrow, making sure
> > that
> > > > all
> > > > > > > works
> > > > > > > > as
> > > > > > > > > > > > intended.
> > > > > > > > > > > > We are also almost finished with cleaning up the
> > > connector
> > > > > > > > > dependencies
> > > > > > > > > > > > that Stephan pointed out, should be finished by
> > tomorrow.
> > > > > > > > > > > >
> > > > > > > > > > > > Gyula
> > > > > > > > > > > >
> > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > > fhueske@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > > > aljoscha@apache.org
> > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > > > > > sewen@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>> I personally like the idea of SOFT time-based
> > feature
> > > > > > > freezes.
> > > > > > > > > > > > Otherwise,
> > > > > > > > > > > > >>> releases will get delayed again and again,
> because
> > of
> > > > > > > features
> > > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > >> try
> > > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Having reached the freeze point already, we could
> > > still
> > > > > > > include
> > > > > > > > > the
> > > > > > > > > > > > >>> features that are pending ready state in the next
> > > days
> > > > > > > > > (streaming,
> > > > > > > > > > > blob
> > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a release
> > > > state.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> We had a mail listing the issues to include into
> > 0.7,
> > > > > but a
> > > > > > > > wiki
> > > > > > > > > > page
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >>> probably be better. In that sense, we could start
> > > > > > collecting
> > > > > > > > the
> > > > > > > > > > > issues
> > > > > > > > > > > > >> for
> > > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > > > > > warneke@apache.org
> > > > > > > > > >:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or
> > > > something
> > > > > > > > > similar)
> > > > > > > > > > > > where
> > > > > > > > > > > > >> we
> > > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline might not
> > be
> > > > the
> > > > > > best
> > > > > > > > > > > practice.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > > >>>>> We decide two (or three) weeks before a
> targeted
> > > > > release
> > > > > > > date
> > > > > > > > > > about
> > > > > > > > > > > > >> which
> > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a
> > > > release
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > >>>>> completed
> > > > > > > > > > > > >>>>> or really close to completion (via progress
> > > estimates
> > > > > in
> > > > > > > > JIRA).
> > > > > > > > > > > > >>>>> After we decided which JIRAs to include in a
> > > release,
> > > > > we
> > > > > > > can
> > > > > > > > > use
> > > > > > > > > > > JIRA
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >>>>> track the progress and dedicate another week
> > > > > exclusively
> > > > > > > for
> > > > > > > > > > > testing
> > > > > > > > > > > > >> after
> > > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> As for the streaming team we're also getting
> > ready
> > > > for
> > > > > > the
> > > > > > > > > > release,
> > > > > > > > > > > > >> but a
> > > > > > > > > > > > >>>>>> couple of days will be needed to finish the
> > > features
> > > > > > that
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > > > >> like
> > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>    - A little work is still needed for reduce
> > > > > operations
> > > > > > > and
> > > > > > > > > > > > >>>>>>    groups/connected streams (any comment on
> > > Gyula's
> > > > > > recent
> > > > > > > > > > e-mail
> > > > > > > > > > > is
> > > > > > > > > > > > >>>>>> really
> > > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > > >>>>>>    - The examples are being updated to match
> the
> > > > > > standard,
> > > > > > > > > check
> > > > > > > > > > > out
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>
> > https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > > >>>>>>
> > streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > > >>>>>>
> > streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > > >>>>>> )
> > > > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > > > > > uce@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > > > > rmetzger@apache.org
> > > > > > > > > >
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a
> > > feature
> > > > > > freeze
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would
> > > > suggest
> > > > > to
> > > > > > > > ship
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > > > > documentation.
> > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> > generation,
> > > > key
> > > > > > > > > selection
> > > > > > > > > > > > >> using a
> > > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API
> > in
> > > > sync
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > >> latest
> > > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > > >>>>>>>>
> > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > > >>>>>>>>
> > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature
> > has
> > > > been
> > > > > > > > > requested
> > > > > > > > > > > by
> > > > > > > > > > > > >> many
> > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> > > > Collections
> > > > > > > > inside
> > > > > > > > > > > POJOs:
> > > > > > > > > > > > >>>>>>>>
> > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> > > > incremental
> > > > > > > > program
> > > > > > > > > > > > >> rollout &
> > > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> The following features are in the master, as
> > of
> > > > > today:
> > > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> We certainly need some days to test
> everything
> > > > until
> > > > > > we
> > > > > > > > can
> > > > > > > > > > > start
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> Based on our experience with the last major
> > > > > release, I
> > > > > > > > would
> > > > > > > > > > > > really
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the
> first
> > > > > release
> > > > > > > > > > candidate.
> > > > > > > > > > > > For
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we
> > wait
> > > a
> > > > > few
> > > > > > > more
> > > > > > > > > > days
> > > > > > > > > > > > for
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>>>>>> release so that a few more features make it
> > into
> > > > the
> > > > > > > > > release?
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its
> > > really
> > > > > > nice
> > > > > > > to
> > > > > > > > > > > release
> > > > > > > > > > > > >> on a
> > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some
> > time
> > > > and
> > > > > > > causes
> > > > > > > > > > > > overhead
> > > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > > >>>>>>>> I would really like to have the Java API
> Pojo
> > > > > > > improvements
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> I think I can finish it until end of this
> > week.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>> I agree that the finished features
> (especially
> > > the
> > > > > > Scala
> > > > > > > > API)
> > > > > > > > > > are
> > > > > > > > > > > > >> nice
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> a new release, but still I would like to
> wait a
> > > few
> > > > > > more
> > > > > > > > > days.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> Some of the missing features are on the brink
> > of
> > > > > being
> > > > > > > > > finished
> > > > > > > > > > > > (e.g.
> > > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to
> invest a
> > > > week
> > > > > in
> > > > > > > bug
> > > > > > > > > > > fixing
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >>>>>>> doing the release vote, when the new features
> > are
> > > > > > likely
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > > >> finished
> > > > > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming
> > > > features
> > > > > > > will
> > > > > > > > > > > > >> definitely be
> > > > > > > > > > > > >>>>>>> worth a release, so users can work with them.
> > ;)
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
I would add #142.

On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <se...@apache.org> wrote:

> I agree, having a release candidate out would be nice.
>
> What is your opinion on issue #142? Fault tolerance is inactive by default,
> but can be activated through the configuration.
>
> On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Oh. I didn't know this. The last mail from Gyula sounded like the
> Streaming
> > part is ready2release.
> >
> > My personal goal was to prepare a release candiate today so that we have
> a
> > reference point against which we can test (which would also mean forking
> of
> > a 0.7-release branch and basically doing a feature freeze).
> > I also though that it might be a good idea to have a release candidate by
> > tomorrow because it could be of use for the Hackathon tomorrow in
> > Stockholm.
> > But since the Scala POJO changes and the streaming examples are both
> > outstanding, we can probably scratch that.
> >
> > Doing a fork, release candidate and a feature freeze doesn't mean we can
> > not supply bugfixes anymore for the 0.7-release ;)
> >
> > What is your best-case scenario for FLINK-1103? (as far as I can see the
> > only outstanding streaming feature for the release)
> >
> > Best,
> > Robert
> >
> >
> > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <balassi.marton@gmail.com
> >
> > wrote:
> >
> > > As for the streaming side we would like to push some bugfix commits and
> > the
> > > resolution of the FLINK-1103 JIRA issue.
> > >
> > > These are more or less ready, hopefully will be available at the end of
> > > this week.
> > >
> > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > I've merged the record api deprecation already.
> > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > >
> > > > We certainly should merge #136 and #143 as well.
> > > >
> > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org>
> > > wrote:
> > > >
> > > > > So which PRs will be included in the candidate?
> > > > >
> > > > > #141 POJOs
> > > > > #144 Deprecate Record API
> > > > > #136 Fixed example quickstart
> > > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> > > (includes
> > > > > PR #131)
> > > > >
> > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > >
> > > > > > Great news, looking forward to seeing this in the master!
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > As an update for everyone: My POJO feature is finished,
> including
> > > > > > > documentation.
> > > > > > > Aljoscha is currently adopting the Scala API to have support
> for
> > > > > (nested)
> > > > > > > POJOs as well.
> > > > > > > Once that is done, I'll merge everything and create a first
> > > candidate
> > > > > > that
> > > > > > > we can use for testing.
> > > > > > >
> > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > fhueske@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I just checked the "Run Example" quickstart and it needs a
> bit
> > of
> > > > > work.
> > > > > > > >
> > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> rmetzger@apache.org
> > >:
> > > > > > > >
> > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > >
> > > > > > > > > We also should do a pass over our documentation, the
> > > quickstarts
> > > > > and
> > > > > > > the
> > > > > > > > > website to see if everything is in a good state (for
> example
> > > the
> > > > > > > > > collection-based execution needs documentation as well). We
> > > > should
> > > > > > also
> > > > > > > > > finally document the hadoop-input format wrappers (I think
> > Timo
> > > > is
> > > > > > > > working
> > > > > > > > > on a pull request for that).
> > > > > > > > > This page mentions the LocalDistributedExecutor and
> contains
> > > some
> > > > > > (most
> > > > > > > > > likely outdated) scala code:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > >
> > > > > > > > > We also need to deprecate the old record api (
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > >
> > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > Last issues are going in and we started working on
> > > dependencies
> > > > > and
> > > > > > > > test
> > > > > > > > > > platform diversity in order to make stabilizing phase.
> > > > > > > > > >
> > > > > > > > > > We should have an official feature freeze soon and fork
> the
> > > > > > > 0.7-release
> > > > > > > > > > branch. I personally vote to include the POJO support (I
> > > think
> > > > > > Robert
> > > > > > > > is
> > > > > > > > > > sorting that one out and is close to completion), and I
> > want
> > > to
> > > > > add
> > > > > > > the
> > > > > > > > > > collection-based execution (today or tomorrow). Till has
> > the
> > > > BLOB
> > > > > > > > manager
> > > > > > > > > > ready, which would be good to include (better support
> large
> > > > > > libraries
> > > > > > > > or
> > > > > > > > > > fat jars).
> > > > > > > > > >
> > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > > gyfora@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey,
> > > > > > > > > > >
> > > > > > > > > > > So what is the current decision regarding the time of
> the
> > > > > > upcoming
> > > > > > > > > > release?
> > > > > > > > > > >
> > > > > > > > > > > As for the streaming component, we included all the
> > > features
> > > > we
> > > > > > > > wanted,
> > > > > > > > > > we
> > > > > > > > > > > will start to test everything tomorrow, making sure
> that
> > > all
> > > > > > works
> > > > > > > as
> > > > > > > > > > > intended.
> > > > > > > > > > > We are also almost finished with cleaning up the
> > connector
> > > > > > > > dependencies
> > > > > > > > > > > that Stephan pointed out, should be finished by
> tomorrow.
> > > > > > > > > > >
> > > > > > > > > > > Gyula
> > > > > > > > > > >
> > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > > fhueske@apache.org>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > > >
> > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > > aljoscha@apache.org
> > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > > > > sewen@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>> I personally like the idea of SOFT time-based
> feature
> > > > > > freezes.
> > > > > > > > > > > Otherwise,
> > > > > > > > > > > >>> releases will get delayed again and again, because
> of
> > > > > > features
> > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > >> try
> > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Having reached the freeze point already, we could
> > still
> > > > > > include
> > > > > > > > the
> > > > > > > > > > > >>> features that are pending ready state in the next
> > days
> > > > > > > > (streaming,
> > > > > > > > > > blob
> > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a release
> > > state.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> We had a mail listing the issues to include into
> 0.7,
> > > > but a
> > > > > > > wiki
> > > > > > > > > page
> > > > > > > > > > > >> would
> > > > > > > > > > > >>> probably be better. In that sense, we could start
> > > > > collecting
> > > > > > > the
> > > > > > > > > > issues
> > > > > > > > > > > >> for
> > > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > > > > warneke@apache.org
> > > > > > > > >:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or
> > > something
> > > > > > > > similar)
> > > > > > > > > > > where
> > > > > > > > > > > >> we
> > > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> I agree, a hard feature stop deadline might not
> be
> > > the
> > > > > best
> > > > > > > > > > practice.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > > >>>>> We decide two (or three) weeks before a targeted
> > > > release
> > > > > > date
> > > > > > > > > about
> > > > > > > > > > > >> which
> > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a
> > > release
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > >>>>> completed
> > > > > > > > > > > >>>>> or really close to completion (via progress
> > estimates
> > > > in
> > > > > > > JIRA).
> > > > > > > > > > > >>>>> After we decided which JIRAs to include in a
> > release,
> > > > we
> > > > > > can
> > > > > > > > use
> > > > > > > > > > JIRA
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>>> track the progress and dedicate another week
> > > > exclusively
> > > > > > for
> > > > > > > > > > testing
> > > > > > > > > > > >> after
> > > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> As for the streaming team we're also getting
> ready
> > > for
> > > > > the
> > > > > > > > > release,
> > > > > > > > > > > >> but a
> > > > > > > > > > > >>>>>> couple of days will be needed to finish the
> > features
> > > > > that
> > > > > > we
> > > > > > > > > would
> > > > > > > > > > > >> like
> > > > > > > > > > > >>>>>> to
> > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>    - A little work is still needed for reduce
> > > > operations
> > > > > > and
> > > > > > > > > > > >>>>>>    groups/connected streams (any comment on
> > Gyula's
> > > > > recent
> > > > > > > > > e-mail
> > > > > > > > > > is
> > > > > > > > > > > >>>>>> really
> > > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > > >>>>>>    - The examples are being updated to match the
> > > > > standard,
> > > > > > > > check
> > > > > > > > > > out
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>
> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > >>>>>>
> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > >>>>>>
> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > >>>>>> )
> > > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > > > > uce@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > > > rmetzger@apache.org
> > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a
> > feature
> > > > > freeze
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would
> > > suggest
> > > > to
> > > > > > > ship
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > > > documentation.
> > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code
> generation,
> > > key
> > > > > > > > selection
> > > > > > > > > > > >> using a
> > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API
> in
> > > sync
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > >> latest
> > > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > > >>>>>>>>
> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > > >>>>>>>>
> > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature
> has
> > > been
> > > > > > > > requested
> > > > > > > > > > by
> > > > > > > > > > > >> many
> > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> > > Collections
> > > > > > > inside
> > > > > > > > > > POJOs:
> > > > > > > > > > > >>>>>>>>
> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> > > incremental
> > > > > > > program
> > > > > > > > > > > >> rollout &
> > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> The following features are in the master, as
> of
> > > > today:
> > > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> We certainly need some days to test everything
> > > until
> > > > > we
> > > > > > > can
> > > > > > > > > > start
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> Based on our experience with the last major
> > > > release, I
> > > > > > > would
> > > > > > > > > > > really
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first
> > > > release
> > > > > > > > > candidate.
> > > > > > > > > > > For
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we
> wait
> > a
> > > > few
> > > > > > more
> > > > > > > > > days
> > > > > > > > > > > for
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>>> release so that a few more features make it
> into
> > > the
> > > > > > > > release?
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its
> > really
> > > > > nice
> > > > > > to
> > > > > > > > > > release
> > > > > > > > > > > >> on a
> > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some
> time
> > > and
> > > > > > causes
> > > > > > > > > > > overhead
> > > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > > >>>>>>>> I would really like to have the Java API Pojo
> > > > > > improvements
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> I think I can finish it until end of this
> week.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> I agree that the finished features (especially
> > the
> > > > > Scala
> > > > > > > API)
> > > > > > > > > are
> > > > > > > > > > > >> nice
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>> for
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> a new release, but still I would like to wait a
> > few
> > > > > more
> > > > > > > > days.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Some of the missing features are on the brink
> of
> > > > being
> > > > > > > > finished
> > > > > > > > > > > (e.g.
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a
> > > week
> > > > in
> > > > > > bug
> > > > > > > > > > fixing
> > > > > > > > > > > >> and
> > > > > > > > > > > >>>>>>> doing the release vote, when the new features
> are
> > > > > likely
> > > > > > to
> > > > > > > > be
> > > > > > > > > > > >> finished
> > > > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming
> > > features
> > > > > > will
> > > > > > > > > > > >> definitely be
> > > > > > > > > > > >>>>>>> worth a release, so users can work with them.
> ;)
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
I agree, having a release candidate out would be nice.

What is your opinion on issue #142? Fault tolerance is inactive by default,
but can be activated through the configuration.

On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rm...@apache.org> wrote:

> Oh. I didn't know this. The last mail from Gyula sounded like the Streaming
> part is ready2release.
>
> My personal goal was to prepare a release candiate today so that we have a
> reference point against which we can test (which would also mean forking of
> a 0.7-release branch and basically doing a feature freeze).
> I also though that it might be a good idea to have a release candidate by
> tomorrow because it could be of use for the Hackathon tomorrow in
> Stockholm.
> But since the Scala POJO changes and the streaming examples are both
> outstanding, we can probably scratch that.
>
> Doing a fork, release candidate and a feature freeze doesn't mean we can
> not supply bugfixes anymore for the 0.7-release ;)
>
> What is your best-case scenario for FLINK-1103? (as far as I can see the
> only outstanding streaming feature for the release)
>
> Best,
> Robert
>
>
> On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > As for the streaming side we would like to push some bugfix commits and
> the
> > resolution of the FLINK-1103 JIRA issue.
> >
> > These are more or less ready, hopefully will be available at the end of
> > this week.
> >
> > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > I've merged the record api deprecation already.
> > > I'll merge #141 once Aljoscha provided his Scala changes.
> > >
> > > We certainly should merge #136 and #143 as well.
> > >
> > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org>
> > wrote:
> > >
> > > > So which PRs will be included in the candidate?
> > > >
> > > > #141 POJOs
> > > > #144 Deprecate Record API
> > > > #136 Fixed example quickstart
> > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> > (includes
> > > > PR #131)
> > > >
> > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > >
> > > > > Great news, looking forward to seeing this in the master!
> > > > >
> > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > As an update for everyone: My POJO feature is finished, including
> > > > > > documentation.
> > > > > > Aljoscha is currently adopting the Scala API to have support for
> > > > (nested)
> > > > > > POJOs as well.
> > > > > > Once that is done, I'll merge everything and create a first
> > candidate
> > > > > that
> > > > > > we can use for testing.
> > > > > >
> > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > fhueske@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > I just checked the "Run Example" quickstart and it needs a bit
> of
> > > > work.
> > > > > > >
> > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rmetzger@apache.org
> >:
> > > > > > >
> > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > >
> > > > > > > > We also should do a pass over our documentation, the
> > quickstarts
> > > > and
> > > > > > the
> > > > > > > > website to see if everything is in a good state (for example
> > the
> > > > > > > > collection-based execution needs documentation as well). We
> > > should
> > > > > also
> > > > > > > > finally document the hadoop-input format wrappers (I think
> Timo
> > > is
> > > > > > > working
> > > > > > > > on a pull request for that).
> > > > > > > > This page mentions the LocalDistributedExecutor and contains
> > some
> > > > > (most
> > > > > > > > likely outdated) scala code:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > >
> > > > > > > > We also need to deprecate the old record api (
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > >
> > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> > sewen@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think we are approaching ready state.
> > > > > > > > > Last issues are going in and we started working on
> > dependencies
> > > > and
> > > > > > > test
> > > > > > > > > platform diversity in order to make stabilizing phase.
> > > > > > > > >
> > > > > > > > > We should have an official feature freeze soon and fork the
> > > > > > 0.7-release
> > > > > > > > > branch. I personally vote to include the POJO support (I
> > think
> > > > > Robert
> > > > > > > is
> > > > > > > > > sorting that one out and is close to completion), and I
> want
> > to
> > > > add
> > > > > > the
> > > > > > > > > collection-based execution (today or tomorrow). Till has
> the
> > > BLOB
> > > > > > > manager
> > > > > > > > > ready, which would be good to include (better support large
> > > > > libraries
> > > > > > > or
> > > > > > > > > fat jars).
> > > > > > > > >
> > > > > > > > > After that, I vote to freeze.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> > gyfora@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey,
> > > > > > > > > >
> > > > > > > > > > So what is the current decision regarding the time of the
> > > > > upcoming
> > > > > > > > > release?
> > > > > > > > > >
> > > > > > > > > > As for the streaming component, we included all the
> > features
> > > we
> > > > > > > wanted,
> > > > > > > > > we
> > > > > > > > > > will start to test everything tomorrow, making sure that
> > all
> > > > > works
> > > > > > as
> > > > > > > > > > intended.
> > > > > > > > > > We are also almost finished with cleaning up the
> connector
> > > > > > > dependencies
> > > > > > > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > > > > > > >
> > > > > > > > > > Gyula
> > > > > > > > > >
> > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> > fhueske@apache.org>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > > >
> > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > > aljoscha@apache.org
> > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > > > sewen@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >>> I personally like the idea of SOFT time-based feature
> > > > > freezes.
> > > > > > > > > > Otherwise,
> > > > > > > > > > >>> releases will get delayed again and again, because of
> > > > > features
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > > >> try
> > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Having reached the freeze point already, we could
> still
> > > > > include
> > > > > > > the
> > > > > > > > > > >>> features that are pending ready state in the next
> days
> > > > > > > (streaming,
> > > > > > > > > blob
> > > > > > > > > > >>> Manager, POJOs), but otherwise head for a release
> > state.
> > > > > > > > > > >>>
> > > > > > > > > > >>> We had a mail listing the issues to include into 0.7,
> > > but a
> > > > > > wiki
> > > > > > > > page
> > > > > > > > > > >> would
> > > > > > > > > > >>> probably be better. In that sense, we could start
> > > > collecting
> > > > > > the
> > > > > > > > > issues
> > > > > > > > > > >> for
> > > > > > > > > > >>> the next release from now on.
> > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > > > warneke@apache.org
> > > > > > > >:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Hi,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or
> > something
> > > > > > > similar)
> > > > > > > > > > where
> > > > > > > > > > >> we
> > > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Best regards,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>    Daniel
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> I agree, a hard feature stop deadline might not be
> > the
> > > > best
> > > > > > > > > practice.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > > >>>>> We decide two (or three) weeks before a targeted
> > > release
> > > > > date
> > > > > > > > about
> > > > > > > > > > >> which
> > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a
> > release
> > > > > > should
> > > > > > > be
> > > > > > > > > > >>>>> completed
> > > > > > > > > > >>>>> or really close to completion (via progress
> estimates
> > > in
> > > > > > JIRA).
> > > > > > > > > > >>>>> After we decided which JIRAs to include in a
> release,
> > > we
> > > > > can
> > > > > > > use
> > > > > > > > > JIRA
> > > > > > > > > > >> to
> > > > > > > > > > >>>>> track the progress and dedicate another week
> > > exclusively
> > > > > for
> > > > > > > > > testing
> > > > > > > > > > >> after
> > > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> As for the streaming team we're also getting ready
> > for
> > > > the
> > > > > > > > release,
> > > > > > > > > > >> but a
> > > > > > > > > > >>>>>> couple of days will be needed to finish the
> features
> > > > that
> > > > > we
> > > > > > > > would
> > > > > > > > > > >> like
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>> include.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>    - A little work is still needed for reduce
> > > operations
> > > > > and
> > > > > > > > > > >>>>>>    groups/connected streams (any comment on
> Gyula's
> > > > recent
> > > > > > > > e-mail
> > > > > > > > > is
> > > > > > > > > > >>>>>> really
> > > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > > >>>>>>    - The examples are being updated to match the
> > > > standard,
> > > > > > > check
> > > > > > > > > out
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > >>>>>> )
> > > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > > > uce@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > > rmetzger@apache.org
> > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a
> feature
> > > > freeze
> > > > > > for
> > > > > > > > the
> > > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would
> > suggest
> > > to
> > > > > > ship
> > > > > > > > the
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > > documentation.
> > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation,
> > key
> > > > > > > selection
> > > > > > > > > > >> using a
> > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in
> > sync
> > > > > with
> > > > > > > the
> > > > > > > > > > >> latest
> > > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > > >>>>>>>>
> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has
> > been
> > > > > > > requested
> > > > > > > > > by
> > > > > > > > > > >> many
> > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> > Collections
> > > > > > inside
> > > > > > > > > POJOs:
> > > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> > incremental
> > > > > > program
> > > > > > > > > > >> rollout &
> > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> The following features are in the master, as of
> > > today:
> > > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> We certainly need some days to test everything
> > until
> > > > we
> > > > > > can
> > > > > > > > > start
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> Based on our experience with the last major
> > > release, I
> > > > > > would
> > > > > > > > > > really
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> like
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first
> > > release
> > > > > > > > candidate.
> > > > > > > > > > For
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we wait
> a
> > > few
> > > > > more
> > > > > > > > days
> > > > > > > > > > for
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>>>> release so that a few more features make it into
> > the
> > > > > > > release?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its
> really
> > > > nice
> > > > > to
> > > > > > > > > release
> > > > > > > > > > >> on a
> > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some time
> > and
> > > > > causes
> > > > > > > > > > overhead
> > > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > > >>>>>>>> I would really like to have the Java API Pojo
> > > > > improvements
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> release.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>> I agree that the finished features (especially
> the
> > > > Scala
> > > > > > API)
> > > > > > > > are
> > > > > > > > > > >> nice
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>> for
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> a new release, but still I would like to wait a
> few
> > > > more
> > > > > > > days.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Some of the missing features are on the brink of
> > > being
> > > > > > > finished
> > > > > > > > > > (e.g.
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a
> > week
> > > in
> > > > > bug
> > > > > > > > > fixing
> > > > > > > > > > >> and
> > > > > > > > > > >>>>>>> doing the release vote, when the new features are
> > > > likely
> > > > > to
> > > > > > > be
> > > > > > > > > > >> finished
> > > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming
> > features
> > > > > will
> > > > > > > > > > >> definitely be
> > > > > > > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
Oh. I didn't know this. The last mail from Gyula sounded like the Streaming
part is ready2release.

My personal goal was to prepare a release candiate today so that we have a
reference point against which we can test (which would also mean forking of
a 0.7-release branch and basically doing a feature freeze).
I also though that it might be a good idea to have a release candidate by
tomorrow because it could be of use for the Hackathon tomorrow in Stockholm.
But since the Scala POJO changes and the streaming examples are both
outstanding, we can probably scratch that.

Doing a fork, release candidate and a feature freeze doesn't mean we can
not supply bugfixes anymore for the 0.7-release ;)

What is your best-case scenario for FLINK-1103? (as far as I can see the
only outstanding streaming feature for the release)

Best,
Robert


On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <ba...@gmail.com>
wrote:

> As for the streaming side we would like to push some bugfix commits and the
> resolution of the FLINK-1103 JIRA issue.
>
> These are more or less ready, hopefully will be available at the end of
> this week.
>
> On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > I've merged the record api deprecation already.
> > I'll merge #141 once Aljoscha provided his Scala changes.
> >
> > We certainly should merge #136 and #143 as well.
> >
> > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org>
> wrote:
> >
> > > So which PRs will be included in the candidate?
> > >
> > > #141 POJOs
> > > #144 Deprecate Record API
> > > #136 Fixed example quickstart
> > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> (includes
> > > PR #131)
> > >
> > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > >
> > > > Great news, looking forward to seeing this in the master!
> > > >
> > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > As an update for everyone: My POJO feature is finished, including
> > > > > documentation.
> > > > > Aljoscha is currently adopting the Scala API to have support for
> > > (nested)
> > > > > POJOs as well.
> > > > > Once that is done, I'll merge everything and create a first
> candidate
> > > > that
> > > > > we can use for testing.
> > > > >
> > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> fhueske@apache.org>
> > > > > wrote:
> > > > >
> > > > > > I just checked the "Run Example" quickstart and it needs a bit of
> > > work.
> > > > > >
> > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
> > > > > >
> > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > >
> > > > > > > We also should do a pass over our documentation, the
> quickstarts
> > > and
> > > > > the
> > > > > > > website to see if everything is in a good state (for example
> the
> > > > > > > collection-based execution needs documentation as well). We
> > should
> > > > also
> > > > > > > finally document the hadoop-input format wrappers (I think Timo
> > is
> > > > > > working
> > > > > > > on a pull request for that).
> > > > > > > This page mentions the LocalDistributedExecutor and contains
> some
> > > > (most
> > > > > > > likely outdated) scala code:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > >
> > > > > > > We also need to deprecate the old record api (
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > >
> > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <
> sewen@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > I think we are approaching ready state.
> > > > > > > > Last issues are going in and we started working on
> dependencies
> > > and
> > > > > > test
> > > > > > > > platform diversity in order to make stabilizing phase.
> > > > > > > >
> > > > > > > > We should have an official feature freeze soon and fork the
> > > > > 0.7-release
> > > > > > > > branch. I personally vote to include the POJO support (I
> think
> > > > Robert
> > > > > > is
> > > > > > > > sorting that one out and is close to completion), and I want
> to
> > > add
> > > > > the
> > > > > > > > collection-based execution (today or tomorrow). Till has the
> > BLOB
> > > > > > manager
> > > > > > > > ready, which would be good to include (better support large
> > > > libraries
> > > > > > or
> > > > > > > > fat jars).
> > > > > > > >
> > > > > > > > After that, I vote to freeze.
> > > > > > > >
> > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <
> gyfora@apache.org
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey,
> > > > > > > > >
> > > > > > > > > So what is the current decision regarding the time of the
> > > > upcoming
> > > > > > > > release?
> > > > > > > > >
> > > > > > > > > As for the streaming component, we included all the
> features
> > we
> > > > > > wanted,
> > > > > > > > we
> > > > > > > > > will start to test everything tomorrow, making sure that
> all
> > > > works
> > > > > as
> > > > > > > > > intended.
> > > > > > > > > We are also almost finished with cleaning up the connector
> > > > > > dependencies
> > > > > > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > > > > > >
> > > > > > > > > Gyula
> > > > > > > > >
> > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <
> fhueske@apache.org>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > > >
> > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > > aljoscha@apache.org
> > > > > >:
> > > > > > > > > >
> > > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > > >>
> > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > > sewen@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >>> I personally like the idea of SOFT time-based feature
> > > > freezes.
> > > > > > > > > Otherwise,
> > > > > > > > > >>> releases will get delayed again and again, because of
> > > > features
> > > > > > that
> > > > > > > > we
> > > > > > > > > >> try
> > > > > > > > > >>> to squeeze in.
> > > > > > > > > >>>
> > > > > > > > > >>> Having reached the freeze point already, we could still
> > > > include
> > > > > > the
> > > > > > > > > >>> features that are pending ready state in the next days
> > > > > > (streaming,
> > > > > > > > blob
> > > > > > > > > >>> Manager, POJOs), but otherwise head for a release
> state.
> > > > > > > > > >>>
> > > > > > > > > >>> We had a mail listing the issues to include into 0.7,
> > but a
> > > > > wiki
> > > > > > > page
> > > > > > > > > >> would
> > > > > > > > > >>> probably be better. In that sense, we could start
> > > collecting
> > > > > the
> > > > > > > > issues
> > > > > > > > > >> for
> > > > > > > > > >>> the next release from now on.
> > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > > warneke@apache.org
> > > > > > >:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi,
> > > > > > > > > >>>>
> > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or
> something
> > > > > > similar)
> > > > > > > > > where
> > > > > > > > > >> we
> > > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best regards,
> > > > > > > > > >>>>
> > > > > > > > > >>>>    Daniel
> > > > > > > > > >>>>
> > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I agree, a hard feature stop deadline might not be
> the
> > > best
> > > > > > > > practice.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> How about the following procedure:
> > > > > > > > > >>>>> We decide two (or three) weeks before a targeted
> > release
> > > > date
> > > > > > > about
> > > > > > > > > >> which
> > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a
> release
> > > > > should
> > > > > > be
> > > > > > > > > >>>>> completed
> > > > > > > > > >>>>> or really close to completion (via progress estimates
> > in
> > > > > JIRA).
> > > > > > > > > >>>>> After we decided which JIRAs to include in a release,
> > we
> > > > can
> > > > > > use
> > > > > > > > JIRA
> > > > > > > > > >> to
> > > > > > > > > >>>>> track the progress and dedicate another week
> > exclusively
> > > > for
> > > > > > > > testing
> > > > > > > > > >> after
> > > > > > > > > >>>>> the last feature was completed.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > > balassi.marton@gmail.com
> > > > > > > > > >:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> As for the streaming team we're also getting ready
> for
> > > the
> > > > > > > release,
> > > > > > > > > >> but a
> > > > > > > > > >>>>>> couple of days will be needed to finish the features
> > > that
> > > > we
> > > > > > > would
> > > > > > > > > >> like
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>> include.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>    - A little work is still needed for reduce
> > operations
> > > > and
> > > > > > > > > >>>>>>    groups/connected streams (any comment on Gyula's
> > > recent
> > > > > > > e-mail
> > > > > > > > is
> > > > > > > > > >>>>>> really
> > > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > > >>>>>>    - The examples are being updated to match the
> > > standard,
> > > > > > check
> > > > > > > > out
> > > > > > > > > >> the
> > > > > > > > > >>>>>>    WordCount. (
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > >>>>>> )
> > > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > > uce@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > > rmetzger@apache.org
> > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature
> > > freeze
> > > > > for
> > > > > > > the
> > > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would
> suggest
> > to
> > > > > ship
> > > > > > > the
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> streaming,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> > documentation.
> > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation,
> key
> > > > > > selection
> > > > > > > > > >> using a
> > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in
> sync
> > > > with
> > > > > > the
> > > > > > > > > >> latest
> > > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has
> been
> > > > > > requested
> > > > > > > > by
> > > > > > > > > >> many
> > > > > > > > > >>>>>>>> users. Mostly because they wanted to use
> Collections
> > > > > inside
> > > > > > > > POJOs:
> > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > >>>>>>>> - Rework JobManager internals to support
> incremental
> > > > > program
> > > > > > > > > >> rollout &
> > > > > > > > > >>>>>>>> execution
> > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> The following features are in the master, as of
> > today:
> > > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> We certainly need some days to test everything
> until
> > > we
> > > > > can
> > > > > > > > start
> > > > > > > > > >> the
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> vote.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Based on our experience with the last major
> > release, I
> > > > > would
> > > > > > > > > really
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first
> > release
> > > > > > > candidate.
> > > > > > > > > For
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> How do you guys feel about this? Should we wait a
> > few
> > > > more
> > > > > > > days
> > > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >>>>>>>> release so that a few more features make it into
> the
> > > > > > release?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its really
> > > nice
> > > > to
> > > > > > > > release
> > > > > > > > > >> on a
> > > > > > > > > >>>>>>>> regular schedule, but it also eats up some time
> and
> > > > causes
> > > > > > > > > overhead
> > > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > > >>>>>>>> I would really like to have the Java API Pojo
> > > > improvements
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> release.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>> I agree that the finished features (especially the
> > > Scala
> > > > > API)
> > > > > > > are
> > > > > > > > > >> nice
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> a new release, but still I would like to wait a few
> > > more
> > > > > > days.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Some of the missing features are on the brink of
> > being
> > > > > > finished
> > > > > > > > > (e.g.
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a
> week
> > in
> > > > bug
> > > > > > > > fixing
> > > > > > > > > >> and
> > > > > > > > > >>>>>>> doing the release vote, when the new features are
> > > likely
> > > > to
> > > > > > be
> > > > > > > > > >> finished
> > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming
> features
> > > > will
> > > > > > > > > >> definitely be
> > > > > > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Márton Balassi <ba...@gmail.com>.
As for the streaming side we would like to push some bugfix commits and the
resolution of the FLINK-1103 JIRA issue.

These are more or less ready, hopefully will be available at the end of
this week.

On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rm...@apache.org> wrote:

> I've merged the record api deprecation already.
> I'll merge #141 once Aljoscha provided his Scala changes.
>
> We certainly should merge #136 and #143 as well.
>
> On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org> wrote:
>
> > So which PRs will be included in the candidate?
> >
> > #141 POJOs
> > #144 Deprecate Record API
> > #136 Fixed example quickstart
> > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers (includes
> > PR #131)
> >
> > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
> >
> > > Great news, looking forward to seeing this in the master!
> > >
> > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > As an update for everyone: My POJO feature is finished, including
> > > > documentation.
> > > > Aljoscha is currently adopting the Scala API to have support for
> > (nested)
> > > > POJOs as well.
> > > > Once that is done, I'll merge everything and create a first candidate
> > > that
> > > > we can use for testing.
> > > >
> > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <fh...@apache.org>
> > > > wrote:
> > > >
> > > > > I just checked the "Run Example" quickstart and it needs a bit of
> > work.
> > > > >
> > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
> > > > >
> > > > > > I'm working hard on getting the POJOs ready.
> > > > > >
> > > > > > We also should do a pass over our documentation, the quickstarts
> > and
> > > > the
> > > > > > website to see if everything is in a good state (for example the
> > > > > > collection-based execution needs documentation as well). We
> should
> > > also
> > > > > > finally document the hadoop-input format wrappers (I think Timo
> is
> > > > > working
> > > > > > on a pull request for that).
> > > > > > This page mentions the LocalDistributedExecutor and contains some
> > > (most
> > > > > > likely outdated) scala code:
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > >
> > > > > > We also need to deprecate the old record api (
> > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > >
> > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > I think we are approaching ready state.
> > > > > > > Last issues are going in and we started working on dependencies
> > and
> > > > > test
> > > > > > > platform diversity in order to make stabilizing phase.
> > > > > > >
> > > > > > > We should have an official feature freeze soon and fork the
> > > > 0.7-release
> > > > > > > branch. I personally vote to include the POJO support (I think
> > > Robert
> > > > > is
> > > > > > > sorting that one out and is close to completion), and I want to
> > add
> > > > the
> > > > > > > collection-based execution (today or tomorrow). Till has the
> BLOB
> > > > > manager
> > > > > > > ready, which would be good to include (better support large
> > > libraries
> > > > > or
> > > > > > > fat jars).
> > > > > > >
> > > > > > > After that, I vote to freeze.
> > > > > > >
> > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gyfora@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hey,
> > > > > > > >
> > > > > > > > So what is the current decision regarding the time of the
> > > upcoming
> > > > > > > release?
> > > > > > > >
> > > > > > > > As for the streaming component, we included all the features
> we
> > > > > wanted,
> > > > > > > we
> > > > > > > > will start to test everything tomorrow, making sure that all
> > > works
> > > > as
> > > > > > > > intended.
> > > > > > > > We are also almost finished with cleaning up the connector
> > > > > dependencies
> > > > > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > > > > >
> > > > > > > > Gyula
> > > > > > > >
> > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > > >
> > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > > aljoscha@apache.org
> > > > >:
> > > > > > > > >
> > > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > > >>
> > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > > sewen@apache.org>
> > > > > > > > wrote:
> > > > > > > > >>> I personally like the idea of SOFT time-based feature
> > > freezes.
> > > > > > > > Otherwise,
> > > > > > > > >>> releases will get delayed again and again, because of
> > > features
> > > > > that
> > > > > > > we
> > > > > > > > >> try
> > > > > > > > >>> to squeeze in.
> > > > > > > > >>>
> > > > > > > > >>> Having reached the freeze point already, we could still
> > > include
> > > > > the
> > > > > > > > >>> features that are pending ready state in the next days
> > > > > (streaming,
> > > > > > > blob
> > > > > > > > >>> Manager, POJOs), but otherwise head for a release state.
> > > > > > > > >>>
> > > > > > > > >>> We had a mail listing the issues to include into 0.7,
> but a
> > > > wiki
> > > > > > page
> > > > > > > > >> would
> > > > > > > > >>> probably be better. In that sense, we could start
> > collecting
> > > > the
> > > > > > > issues
> > > > > > > > >> for
> > > > > > > > >>> the next release from now on.
> > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > > warneke@apache.org
> > > > > >:
> > > > > > > > >>>
> > > > > > > > >>>> Hi,
> > > > > > > > >>>>
> > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or something
> > > > > similar)
> > > > > > > > where
> > > > > > > > >> we
> > > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > > >>>>
> > > > > > > > >>>> Best regards,
> > > > > > > > >>>>
> > > > > > > > >>>>    Daniel
> > > > > > > > >>>>
> > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > > >>>>
> > > > > > > > >>>>> I agree, a hard feature stop deadline might not be the
> > best
> > > > > > > practice.
> > > > > > > > >>>>>
> > > > > > > > >>>>> How about the following procedure:
> > > > > > > > >>>>> We decide two (or three) weeks before a targeted
> release
> > > date
> > > > > > about
> > > > > > > > >> which
> > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a release
> > > > should
> > > > > be
> > > > > > > > >>>>> completed
> > > > > > > > >>>>> or really close to completion (via progress estimates
> in
> > > > JIRA).
> > > > > > > > >>>>> After we decided which JIRAs to include in a release,
> we
> > > can
> > > > > use
> > > > > > > JIRA
> > > > > > > > >> to
> > > > > > > > >>>>> track the progress and dedicate another week
> exclusively
> > > for
> > > > > > > testing
> > > > > > > > >> after
> > > > > > > > >>>>> the last feature was completed.
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > > balassi.marton@gmail.com
> > > > > > > > >:
> > > > > > > > >>>>>
> > > > > > > > >>>>> As for the streaming team we're also getting ready for
> > the
> > > > > > release,
> > > > > > > > >> but a
> > > > > > > > >>>>>> couple of days will be needed to finish the features
> > that
> > > we
> > > > > > would
> > > > > > > > >> like
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>> include.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>    - A little work is still needed for reduce
> operations
> > > and
> > > > > > > > >>>>>>    groups/connected streams (any comment on Gyula's
> > recent
> > > > > > e-mail
> > > > > > > is
> > > > > > > > >>>>>> really
> > > > > > > > >>>>>>    appreciated :) )
> > > > > > > > >>>>>>    - The examples are being updated to match the
> > standard,
> > > > > check
> > > > > > > out
> > > > > > > > >> the
> > > > > > > > >>>>>>    WordCount. (
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > >>>>>> )
> > > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > uce@apache.org>
> > > > > > > > wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > > rmetzger@apache.org
> > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Hey guys,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature
> > freeze
> > > > for
> > > > > > the
> > > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest
> to
> > > > ship
> > > > > > the
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> streaming,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> but clearly mark it as a preview in the
> documentation.
> > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key
> > > > > selection
> > > > > > > > >> using a
> > > > > > > > >>>>>>>> string-expression:
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync
> > > with
> > > > > the
> > > > > > > > >> latest
> > > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has been
> > > > > requested
> > > > > > > by
> > > > > > > > >> many
> > > > > > > > >>>>>>>> users. Mostly because they wanted to use Collections
> > > > inside
> > > > > > > POJOs:
> > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > >>>>>>>> - Rework JobManager internals to support incremental
> > > > program
> > > > > > > > >> rollout &
> > > > > > > > >>>>>>>> execution
> > > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> The following features are in the master, as of
> today:
> > > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> We certainly need some days to test everything until
> > we
> > > > can
> > > > > > > start
> > > > > > > > >> the
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> vote.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Based on our experience with the last major
> release, I
> > > > would
> > > > > > > > really
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> like
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first
> release
> > > > > > candidate.
> > > > > > > > For
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> How do you guys feel about this? Should we wait a
> few
> > > more
> > > > > > days
> > > > > > > > for
> > > > > > > > >> the
> > > > > > > > >>>>>>>> release so that a few more features make it into the
> > > > > release?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its really
> > nice
> > > to
> > > > > > > release
> > > > > > > > >> on a
> > > > > > > > >>>>>>>> regular schedule, but it also eats up some time and
> > > causes
> > > > > > > > overhead
> > > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > > >>>>>>>> I would really like to have the Java API Pojo
> > > improvements
> > > > > in
> > > > > > > the
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> release.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Opinions?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>> I agree that the finished features (especially the
> > Scala
> > > > API)
> > > > > > are
> > > > > > > > >> nice
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> a new release, but still I would like to wait a few
> > more
> > > > > days.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Some of the missing features are on the brink of
> being
> > > > > finished
> > > > > > > > (e.g.
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a week
> in
> > > bug
> > > > > > > fixing
> > > > > > > > >> and
> > > > > > > > >>>>>>> doing the release vote, when the new features are
> > likely
> > > to
> > > > > be
> > > > > > > > >> finished
> > > > > > > > >>>>>>> just a few days afterwards. And the upcoming features
> > > will
> > > > > > > > >> definitely be
> > > > > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
I've merged the record api deprecation already.
I'll merge #141 once Aljoscha provided his Scala changes.

We certainly should merge #136 and #143 as well.

On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fh...@apache.org> wrote:

> So which PRs will be included in the candidate?
>
> #141 POJOs
> #144 Deprecate Record API
> #136 Fixed example quickstart
> #143(?) Hadoop Compat: Documentation + Hadoop function wrappers (includes
> PR #131)
>
> 2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:
>
> > Great news, looking forward to seeing this in the master!
> >
> > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > As an update for everyone: My POJO feature is finished, including
> > > documentation.
> > > Aljoscha is currently adopting the Scala API to have support for
> (nested)
> > > POJOs as well.
> > > Once that is done, I'll merge everything and create a first candidate
> > that
> > > we can use for testing.
> > >
> > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <fh...@apache.org>
> > > wrote:
> > >
> > > > I just checked the "Run Example" quickstart and it needs a bit of
> work.
> > > >
> > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
> > > >
> > > > > I'm working hard on getting the POJOs ready.
> > > > >
> > > > > We also should do a pass over our documentation, the quickstarts
> and
> > > the
> > > > > website to see if everything is in a good state (for example the
> > > > > collection-based execution needs documentation as well). We should
> > also
> > > > > finally document the hadoop-input format wrappers (I think Timo is
> > > > working
> > > > > on a pull request for that).
> > > > > This page mentions the LocalDistributedExecutor and contains some
> > (most
> > > > > likely outdated) scala code:
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > >
> > > > > We also need to deprecate the old record api (
> > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > >
> > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > I think we are approaching ready state.
> > > > > > Last issues are going in and we started working on dependencies
> and
> > > > test
> > > > > > platform diversity in order to make stabilizing phase.
> > > > > >
> > > > > > We should have an official feature freeze soon and fork the
> > > 0.7-release
> > > > > > branch. I personally vote to include the POJO support (I think
> > Robert
> > > > is
> > > > > > sorting that one out and is close to completion), and I want to
> add
> > > the
> > > > > > collection-based execution (today or tomorrow). Till has the BLOB
> > > > manager
> > > > > > ready, which would be good to include (better support large
> > libraries
> > > > or
> > > > > > fat jars).
> > > > > >
> > > > > > After that, I vote to freeze.
> > > > > >
> > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hey,
> > > > > > >
> > > > > > > So what is the current decision regarding the time of the
> > upcoming
> > > > > > release?
> > > > > > >
> > > > > > > As for the streaming component, we included all the features we
> > > > wanted,
> > > > > > we
> > > > > > > will start to test everything tomorrow, making sure that all
> > works
> > > as
> > > > > > > intended.
> > > > > > > We are also almost finished with cleaning up the connector
> > > > dependencies
> > > > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > > > >
> > > > > > > Gyula
> > > > > > >
> > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > > >
> > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> > aljoscha@apache.org
> > > >:
> > > > > > > >
> > > > > > > >> Can we not manage this stuff on Jira?
> > > > > > > >>
> > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > >>> I personally like the idea of SOFT time-based feature
> > freezes.
> > > > > > > Otherwise,
> > > > > > > >>> releases will get delayed again and again, because of
> > features
> > > > that
> > > > > > we
> > > > > > > >> try
> > > > > > > >>> to squeeze in.
> > > > > > > >>>
> > > > > > > >>> Having reached the freeze point already, we could still
> > include
> > > > the
> > > > > > > >>> features that are pending ready state in the next days
> > > > (streaming,
> > > > > > blob
> > > > > > > >>> Manager, POJOs), but otherwise head for a release state.
> > > > > > > >>>
> > > > > > > >>> We had a mail listing the issues to include into 0.7, but a
> > > wiki
> > > > > page
> > > > > > > >> would
> > > > > > > >>> probably be better. In that sense, we could start
> collecting
> > > the
> > > > > > issues
> > > > > > > >> for
> > > > > > > >>> the next release from now on.
> > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > > warneke@apache.org
> > > > >:
> > > > > > > >>>
> > > > > > > >>>> Hi,
> > > > > > > >>>>
> > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or something
> > > > similar)
> > > > > > > where
> > > > > > > >> we
> > > > > > > >>>> can collect the proposed JIRAs?
> > > > > > > >>>>
> > > > > > > >>>> Best regards,
> > > > > > > >>>>
> > > > > > > >>>>    Daniel
> > > > > > > >>>>
> > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > > >>>>
> > > > > > > >>>>> I agree, a hard feature stop deadline might not be the
> best
> > > > > > practice.
> > > > > > > >>>>>
> > > > > > > >>>>> How about the following procedure:
> > > > > > > >>>>> We decide two (or three) weeks before a targeted release
> > date
> > > > > about
> > > > > > > >> which
> > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a release
> > > should
> > > > be
> > > > > > > >>>>> completed
> > > > > > > >>>>> or really close to completion (via progress estimates in
> > > JIRA).
> > > > > > > >>>>> After we decided which JIRAs to include in a release, we
> > can
> > > > use
> > > > > > JIRA
> > > > > > > >> to
> > > > > > > >>>>> track the progress and dedicate another week exclusively
> > for
> > > > > > testing
> > > > > > > >> after
> > > > > > > >>>>> the last feature was completed.
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > > balassi.marton@gmail.com
> > > > > > > >:
> > > > > > > >>>>>
> > > > > > > >>>>> As for the streaming team we're also getting ready for
> the
> > > > > release,
> > > > > > > >> but a
> > > > > > > >>>>>> couple of days will be needed to finish the features
> that
> > we
> > > > > would
> > > > > > > >> like
> > > > > > > >>>>>> to
> > > > > > > >>>>>> include.
> > > > > > > >>>>>>
> > > > > > > >>>>>>    - A little work is still needed for reduce operations
> > and
> > > > > > > >>>>>>    groups/connected streams (any comment on Gyula's
> recent
> > > > > e-mail
> > > > > > is
> > > > > > > >>>>>> really
> > > > > > > >>>>>>    appreciated :) )
> > > > > > > >>>>>>    - The examples are being updated to match the
> standard,
> > > > check
> > > > > > out
> > > > > > > >> the
> > > > > > > >>>>>>    WordCount. (
> > > > > > > >>>>>>
> > > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > >>>>>> )
> > > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > > uce@apache.org>
> > > > > > > wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > > rmetzger@apache.org
> > > > >
> > > > > > > >> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Hey guys,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature
> freeze
> > > for
> > > > > the
> > > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> From our initial feature list:
> > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to
> > > ship
> > > > > the
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> streaming,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> but clearly mark it as a preview in the documentation.
> > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key
> > > > selection
> > > > > > > >> using a
> > > > > > > >>>>>>>> string-expression:
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync
> > with
> > > > the
> > > > > > > >> latest
> > > > > > > >>>>>>>> developments in the Java API:
> > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has been
> > > > requested
> > > > > > by
> > > > > > > >> many
> > > > > > > >>>>>>>> users. Mostly because they wanted to use Collections
> > > inside
> > > > > > POJOs:
> > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > >>>>>>>> - Rework JobManager internals to support incremental
> > > program
> > > > > > > >> rollout &
> > > > > > > >>>>>>>> execution
> > > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> The following features are in the master, as of today:
> > > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > > >>>>>>>> -* New Scheduler*
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> We certainly need some days to test everything until
> we
> > > can
> > > > > > start
> > > > > > > >> the
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> vote.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Based on our experience with the last major release, I
> > > would
> > > > > > > really
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> like
> > > > > > > >>>>>>
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first release
> > > > > candidate.
> > > > > > > For
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>
> > > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> How do you guys feel about this? Should we wait a few
> > more
> > > > > days
> > > > > > > for
> > > > > > > >> the
> > > > > > > >>>>>>>> release so that a few more features make it into the
> > > > release?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its really
> nice
> > to
> > > > > > release
> > > > > > > >> on a
> > > > > > > >>>>>>>> regular schedule, but it also eats up some time and
> > causes
> > > > > > > overhead
> > > > > > > >>>>>>>> (different branches etc.).
> > > > > > > >>>>>>>> I would really like to have the Java API Pojo
> > improvements
> > > > in
> > > > > > the
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> release.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Opinions?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>> I agree that the finished features (especially the
> Scala
> > > API)
> > > > > are
> > > > > > > >> nice
> > > > > > > >>>>>>>
> > > > > > > >>>>>> for
> > > > > > > >>>>>>
> > > > > > > >>>>>>> a new release, but still I would like to wait a few
> more
> > > > days.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Some of the missing features are on the brink of being
> > > > finished
> > > > > > > (e.g.
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a week in
> > bug
> > > > > > fixing
> > > > > > > >> and
> > > > > > > >>>>>>> doing the release vote, when the new features are
> likely
> > to
> > > > be
> > > > > > > >> finished
> > > > > > > >>>>>>> just a few days afterwards. And the upcoming features
> > will
> > > > > > > >> definitely be
> > > > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
So which PRs will be included in the candidate?

#141 POJOs
#144 Deprecate Record API
#136 Fixed example quickstart
#143(?) Hadoop Compat: Documentation + Hadoop function wrappers (includes
PR #131)

2014-10-07 16:26 GMT+02:00 Stephan Ewen <se...@apache.org>:

> Great news, looking forward to seeing this in the master!
>
> On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > As an update for everyone: My POJO feature is finished, including
> > documentation.
> > Aljoscha is currently adopting the Scala API to have support for (nested)
> > POJOs as well.
> > Once that is done, I'll merge everything and create a first candidate
> that
> > we can use for testing.
> >
> > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <fh...@apache.org>
> > wrote:
> >
> > > I just checked the "Run Example" quickstart and it needs a bit of work.
> > >
> > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
> > >
> > > > I'm working hard on getting the POJOs ready.
> > > >
> > > > We also should do a pass over our documentation, the quickstarts and
> > the
> > > > website to see if everything is in a good state (for example the
> > > > collection-based execution needs documentation as well). We should
> also
> > > > finally document the hadoop-input format wrappers (I think Timo is
> > > working
> > > > on a pull request for that).
> > > > This page mentions the LocalDistributedExecutor and contains some
> (most
> > > > likely outdated) scala code:
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > >
> > > > We also need to deprecate the old record api (
> > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > >
> > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > I think we are approaching ready state.
> > > > > Last issues are going in and we started working on dependencies and
> > > test
> > > > > platform diversity in order to make stabilizing phase.
> > > > >
> > > > > We should have an official feature freeze soon and fork the
> > 0.7-release
> > > > > branch. I personally vote to include the POJO support (I think
> Robert
> > > is
> > > > > sorting that one out and is close to completion), and I want to add
> > the
> > > > > collection-based execution (today or tomorrow). Till has the BLOB
> > > manager
> > > > > ready, which would be good to include (better support large
> libraries
> > > or
> > > > > fat jars).
> > > > >
> > > > > After that, I vote to freeze.
> > > > >
> > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org>
> > wrote:
> > > > >
> > > > > > Hey,
> > > > > >
> > > > > > So what is the current decision regarding the time of the
> upcoming
> > > > > release?
> > > > > >
> > > > > > As for the streaming component, we included all the features we
> > > wanted,
> > > > > we
> > > > > > will start to test everything tomorrow, making sure that all
> works
> > as
> > > > > > intended.
> > > > > > We are also almost finished with cleaning up the connector
> > > dependencies
> > > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > > >
> > > > > > Gyula
> > > > > >
> > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org>
> > wrote:
> > > > > >
> > > > > > > +1 to manage this on JIRA (if possible)
> > > > > > >
> > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <
> aljoscha@apache.org
> > >:
> > > > > > >
> > > > > > >> Can we not manage this stuff on Jira?
> > > > > > >>
> > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> > sewen@apache.org>
> > > > > > wrote:
> > > > > > >>> I personally like the idea of SOFT time-based feature
> freezes.
> > > > > > Otherwise,
> > > > > > >>> releases will get delayed again and again, because of
> features
> > > that
> > > > > we
> > > > > > >> try
> > > > > > >>> to squeeze in.
> > > > > > >>>
> > > > > > >>> Having reached the freeze point already, we could still
> include
> > > the
> > > > > > >>> features that are pending ready state in the next days
> > > (streaming,
> > > > > blob
> > > > > > >>> Manager, POJOs), but otherwise head for a release state.
> > > > > > >>>
> > > > > > >>> We had a mail listing the issues to include into 0.7, but a
> > wiki
> > > > page
> > > > > > >> would
> > > > > > >>> probably be better. In that sense, we could start collecting
> > the
> > > > > issues
> > > > > > >> for
> > > > > > >>> the next release from now on.
> > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> > warneke@apache.org
> > > >:
> > > > > > >>>
> > > > > > >>>> Hi,
> > > > > > >>>>
> > > > > > >>>> I like Fabian's idea. Is there a wiki page (or something
> > > similar)
> > > > > > where
> > > > > > >> we
> > > > > > >>>> can collect the proposed JIRAs?
> > > > > > >>>>
> > > > > > >>>> Best regards,
> > > > > > >>>>
> > > > > > >>>>    Daniel
> > > > > > >>>>
> > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > > >>>>
> > > > > > >>>>> I agree, a hard feature stop deadline might not be the best
> > > > > practice.
> > > > > > >>>>>
> > > > > > >>>>> How about the following procedure:
> > > > > > >>>>> We decide two (or three) weeks before a targeted release
> date
> > > > about
> > > > > > >> which
> > > > > > >>>>> JIRAs to include. JIRAs that are selected for a release
> > should
> > > be
> > > > > > >>>>> completed
> > > > > > >>>>> or really close to completion (via progress estimates in
> > JIRA).
> > > > > > >>>>> After we decided which JIRAs to include in a release, we
> can
> > > use
> > > > > JIRA
> > > > > > >> to
> > > > > > >>>>> track the progress and dedicate another week exclusively
> for
> > > > > testing
> > > > > > >> after
> > > > > > >>>>> the last feature was completed.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > > balassi.marton@gmail.com
> > > > > > >:
> > > > > > >>>>>
> > > > > > >>>>> As for the streaming team we're also getting ready for the
> > > > release,
> > > > > > >> but a
> > > > > > >>>>>> couple of days will be needed to finish the features that
> we
> > > > would
> > > > > > >> like
> > > > > > >>>>>> to
> > > > > > >>>>>> include.
> > > > > > >>>>>>
> > > > > > >>>>>>    - A little work is still needed for reduce operations
> and
> > > > > > >>>>>>    groups/connected streams (any comment on Gyula's recent
> > > > e-mail
> > > > > is
> > > > > > >>>>>> really
> > > > > > >>>>>>    appreciated :) )
> > > > > > >>>>>>    - The examples are being updated to match the standard,
> > > check
> > > > > out
> > > > > > >> the
> > > > > > >>>>>>    WordCount. (
> > > > > > >>>>>>
> > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > >>>>>> )
> > > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> > uce@apache.org>
> > > > > > wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > > > > >> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>> Hey guys,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze
> > for
> > > > the
> > > > > > >>>>>>>> 0.7-incubating release today.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> From our initial feature list:
> > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to
> > ship
> > > > the
> > > > > > >>>>>>>>
> > > > > > >>>>>>> streaming,
> > > > > > >>>>>>>
> > > > > > >>>>>>>> but clearly mark it as a preview in the documentation.
> > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key
> > > selection
> > > > > > >> using a
> > > > > > >>>>>>>> string-expression:
> > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync
> with
> > > the
> > > > > > >> latest
> > > > > > >>>>>>>> developments in the Java API:
> > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has been
> > > requested
> > > > > by
> > > > > > >> many
> > > > > > >>>>>>>> users. Mostly because they wanted to use Collections
> > inside
> > > > > POJOs:
> > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > >>>>>>>> - Rework JobManager internals to support incremental
> > program
> > > > > > >> rollout &
> > > > > > >>>>>>>> execution
> > > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The following features are in the master, as of today:
> > > > > > >>>>>>>> - *Flink Streaming*
> > > > > > >>>>>>>> - *Reworked Scala API*
> > > > > > >>>>>>>> -* New Scheduler*
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> We certainly need some days to test everything until we
> > can
> > > > > start
> > > > > > >> the
> > > > > > >>>>>>>>
> > > > > > >>>>>>> vote.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Based on our experience with the last major release, I
> > would
> > > > > > really
> > > > > > >>>>>>>>
> > > > > > >>>>>>> like
> > > > > > >>>>>>
> > > > > > >>>>>>> to
> > > > > > >>>>>>>
> > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first release
> > > > candidate.
> > > > > > For
> > > > > > >>>>>>>>
> > > > > > >>>>>>> the
> > > > > > >>>>>>
> > > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> How do you guys feel about this? Should we wait a few
> more
> > > > days
> > > > > > for
> > > > > > >> the
> > > > > > >>>>>>>> release so that a few more features make it into the
> > > release?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I'm undecided on this. On the one hand, its really nice
> to
> > > > > release
> > > > > > >> on a
> > > > > > >>>>>>>> regular schedule, but it also eats up some time and
> causes
> > > > > > overhead
> > > > > > >>>>>>>> (different branches etc.).
> > > > > > >>>>>>>> I would really like to have the Java API Pojo
> improvements
> > > in
> > > > > the
> > > > > > >>>>>>>>
> > > > > > >>>>>>> release.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Opinions?
> > > > > > >>>>>>>>
> > > > > > >>>>>>> I agree that the finished features (especially the Scala
> > API)
> > > > are
> > > > > > >> nice
> > > > > > >>>>>>>
> > > > > > >>>>>> for
> > > > > > >>>>>>
> > > > > > >>>>>>> a new release, but still I would like to wait a few more
> > > days.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Some of the missing features are on the brink of being
> > > finished
> > > > > > (e.g.
> > > > > > >>>>>>> the
> > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a week in
> bug
> > > > > fixing
> > > > > > >> and
> > > > > > >>>>>>> doing the release vote, when the new features are likely
> to
> > > be
> > > > > > >> finished
> > > > > > >>>>>>> just a few days afterwards. And the upcoming features
> will
> > > > > > >> definitely be
> > > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
Great news, looking forward to seeing this in the master!

On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <rm...@apache.org> wrote:

> As an update for everyone: My POJO feature is finished, including
> documentation.
> Aljoscha is currently adopting the Scala API to have support for (nested)
> POJOs as well.
> Once that is done, I'll merge everything and create a first candidate that
> we can use for testing.
>
> On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <fh...@apache.org>
> wrote:
>
> > I just checked the "Run Example" quickstart and it needs a bit of work.
> >
> > 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
> >
> > > I'm working hard on getting the POJOs ready.
> > >
> > > We also should do a pass over our documentation, the quickstarts and
> the
> > > website to see if everything is in a good state (for example the
> > > collection-based execution needs documentation as well). We should also
> > > finally document the hadoop-input format wrappers (I think Timo is
> > working
> > > on a pull request for that).
> > > This page mentions the LocalDistributedExecutor and contains some (most
> > > likely outdated) scala code:
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > >
> > > We also need to deprecate the old record api (
> > > https://issues.apache.org/jira/browse/FLINK-1106).
> > >
> > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > I think we are approaching ready state.
> > > > Last issues are going in and we started working on dependencies and
> > test
> > > > platform diversity in order to make stabilizing phase.
> > > >
> > > > We should have an official feature freeze soon and fork the
> 0.7-release
> > > > branch. I personally vote to include the POJO support (I think Robert
> > is
> > > > sorting that one out and is close to completion), and I want to add
> the
> > > > collection-based execution (today or tomorrow). Till has the BLOB
> > manager
> > > > ready, which would be good to include (better support large libraries
> > or
> > > > fat jars).
> > > >
> > > > After that, I vote to freeze.
> > > >
> > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org>
> wrote:
> > > >
> > > > > Hey,
> > > > >
> > > > > So what is the current decision regarding the time of the upcoming
> > > > release?
> > > > >
> > > > > As for the streaming component, we included all the features we
> > wanted,
> > > > we
> > > > > will start to test everything tomorrow, making sure that all works
> as
> > > > > intended.
> > > > > We are also almost finished with cleaning up the connector
> > dependencies
> > > > > that Stephan pointed out, should be finished by tomorrow.
> > > > >
> > > > > Gyula
> > > > >
> > > > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org>
> wrote:
> > > > >
> > > > > > +1 to manage this on JIRA (if possible)
> > > > > >
> > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <aljoscha@apache.org
> >:
> > > > > >
> > > > > >> Can we not manage this stuff on Jira?
> > > > > >>
> > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <
> sewen@apache.org>
> > > > > wrote:
> > > > > >>> I personally like the idea of SOFT time-based feature freezes.
> > > > > Otherwise,
> > > > > >>> releases will get delayed again and again, because of features
> > that
> > > > we
> > > > > >> try
> > > > > >>> to squeeze in.
> > > > > >>>
> > > > > >>> Having reached the freeze point already, we could still include
> > the
> > > > > >>> features that are pending ready state in the next days
> > (streaming,
> > > > blob
> > > > > >>> Manager, POJOs), but otherwise head for a release state.
> > > > > >>>
> > > > > >>> We had a mail listing the issues to include into 0.7, but a
> wiki
> > > page
> > > > > >> would
> > > > > >>> probably be better. In that sense, we could start collecting
> the
> > > > issues
> > > > > >> for
> > > > > >>> the next release from now on.
> > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <
> warneke@apache.org
> > >:
> > > > > >>>
> > > > > >>>> Hi,
> > > > > >>>>
> > > > > >>>> I like Fabian's idea. Is there a wiki page (or something
> > similar)
> > > > > where
> > > > > >> we
> > > > > >>>> can collect the proposed JIRAs?
> > > > > >>>>
> > > > > >>>> Best regards,
> > > > > >>>>
> > > > > >>>>    Daniel
> > > > > >>>>
> > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > > >>>>
> > > > > >>>>> I agree, a hard feature stop deadline might not be the best
> > > > practice.
> > > > > >>>>>
> > > > > >>>>> How about the following procedure:
> > > > > >>>>> We decide two (or three) weeks before a targeted release date
> > > about
> > > > > >> which
> > > > > >>>>> JIRAs to include. JIRAs that are selected for a release
> should
> > be
> > > > > >>>>> completed
> > > > > >>>>> or really close to completion (via progress estimates in
> JIRA).
> > > > > >>>>> After we decided which JIRAs to include in a release, we can
> > use
> > > > JIRA
> > > > > >> to
> > > > > >>>>> track the progress and dedicate another week exclusively for
> > > > testing
> > > > > >> after
> > > > > >>>>> the last feature was completed.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > > balassi.marton@gmail.com
> > > > > >:
> > > > > >>>>>
> > > > > >>>>> As for the streaming team we're also getting ready for the
> > > release,
> > > > > >> but a
> > > > > >>>>>> couple of days will be needed to finish the features that we
> > > would
> > > > > >> like
> > > > > >>>>>> to
> > > > > >>>>>> include.
> > > > > >>>>>>
> > > > > >>>>>>    - A little work is still needed for reduce operations and
> > > > > >>>>>>    groups/connected streams (any comment on Gyula's recent
> > > e-mail
> > > > is
> > > > > >>>>>> really
> > > > > >>>>>>    appreciated :) )
> > > > > >>>>>>    - The examples are being updated to match the standard,
> > check
> > > > out
> > > > > >> the
> > > > > >>>>>>    WordCount. (
> > > > > >>>>>>
> > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > >>>>>> )
> > > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <
> uce@apache.org>
> > > > > wrote:
> > > > > >>>>>>
> > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <
> rmetzger@apache.org
> > >
> > > > > >> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> Hey guys,
> > > > > >>>>>>>>
> > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze
> for
> > > the
> > > > > >>>>>>>> 0.7-incubating release today.
> > > > > >>>>>>>>
> > > > > >>>>>>>> From our initial feature list:
> > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to
> ship
> > > the
> > > > > >>>>>>>>
> > > > > >>>>>>> streaming,
> > > > > >>>>>>>
> > > > > >>>>>>>> but clearly mark it as a preview in the documentation.
> > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key
> > selection
> > > > > >> using a
> > > > > >>>>>>>> string-expression:
> > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with
> > the
> > > > > >> latest
> > > > > >>>>>>>> developments in the Java API:
> > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > > >>>>>>>>  - *Kryo-based serialization*. This feature has been
> > requested
> > > > by
> > > > > >> many
> > > > > >>>>>>>> users. Mostly because they wanted to use Collections
> inside
> > > > POJOs:
> > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > > >>>>>>>> - Rework JobManager internals to support incremental
> program
> > > > > >> rollout &
> > > > > >>>>>>>> execution
> > > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > > >>>>>>>>
> > > > > >>>>>>>> The following features are in the master, as of today:
> > > > > >>>>>>>> - *Flink Streaming*
> > > > > >>>>>>>> - *Reworked Scala API*
> > > > > >>>>>>>> -* New Scheduler*
> > > > > >>>>>>>>
> > > > > >>>>>>>> We certainly need some days to test everything until we
> can
> > > > start
> > > > > >> the
> > > > > >>>>>>>>
> > > > > >>>>>>> vote.
> > > > > >>>>>>>
> > > > > >>>>>>>> Based on our experience with the last major release, I
> would
> > > > > really
> > > > > >>>>>>>>
> > > > > >>>>>>> like
> > > > > >>>>>>
> > > > > >>>>>>> to
> > > > > >>>>>>>
> > > > > >>>>>>>> do the testing and bugfixing BEFORE the first release
> > > candidate.
> > > > > For
> > > > > >>>>>>>>
> > > > > >>>>>>> the
> > > > > >>>>>>
> > > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > > >>>>>>>>
> > > > > >>>>>>>> How do you guys feel about this? Should we wait a few more
> > > days
> > > > > for
> > > > > >> the
> > > > > >>>>>>>> release so that a few more features make it into the
> > release?
> > > > > >>>>>>>>
> > > > > >>>>>>>> I'm undecided on this. On the one hand, its really nice to
> > > > release
> > > > > >> on a
> > > > > >>>>>>>> regular schedule, but it also eats up some time and causes
> > > > > overhead
> > > > > >>>>>>>> (different branches etc.).
> > > > > >>>>>>>> I would really like to have the Java API Pojo improvements
> > in
> > > > the
> > > > > >>>>>>>>
> > > > > >>>>>>> release.
> > > > > >>>>>>>
> > > > > >>>>>>>> I think I can finish it until end of this week.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Opinions?
> > > > > >>>>>>>>
> > > > > >>>>>>> I agree that the finished features (especially the Scala
> API)
> > > are
> > > > > >> nice
> > > > > >>>>>>>
> > > > > >>>>>> for
> > > > > >>>>>>
> > > > > >>>>>>> a new release, but still I would like to wait a few more
> > days.
> > > > > >>>>>>>
> > > > > >>>>>>> Some of the missing features are on the brink of being
> > finished
> > > > > (e.g.
> > > > > >>>>>>> the
> > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a week in bug
> > > > fixing
> > > > > >> and
> > > > > >>>>>>> doing the release vote, when the new features are likely to
> > be
> > > > > >> finished
> > > > > >>>>>>> just a few days afterwards. And the upcoming features will
> > > > > >> definitely be
> > > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
As an update for everyone: My POJO feature is finished, including
documentation.
Aljoscha is currently adopting the Scala API to have support for (nested)
POJOs as well.
Once that is done, I'll merge everything and create a first candidate that
we can use for testing.

On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <fh...@apache.org> wrote:

> I just checked the "Run Example" quickstart and it needs a bit of work.
>
> 2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:
>
> > I'm working hard on getting the POJOs ready.
> >
> > We also should do a pass over our documentation, the quickstarts and the
> > website to see if everything is in a good state (for example the
> > collection-based execution needs documentation as well). We should also
> > finally document the hadoop-input format wrappers (I think Timo is
> working
> > on a pull request for that).
> > This page mentions the LocalDistributedExecutor and contains some (most
> > likely outdated) scala code:
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> >
> > We also need to deprecate the old record api (
> > https://issues.apache.org/jira/browse/FLINK-1106).
> >
> > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > I think we are approaching ready state.
> > > Last issues are going in and we started working on dependencies and
> test
> > > platform diversity in order to make stabilizing phase.
> > >
> > > We should have an official feature freeze soon and fork the 0.7-release
> > > branch. I personally vote to include the POJO support (I think Robert
> is
> > > sorting that one out and is close to completion), and I want to add the
> > > collection-based execution (today or tomorrow). Till has the BLOB
> manager
> > > ready, which would be good to include (better support large libraries
> or
> > > fat jars).
> > >
> > > After that, I vote to freeze.
> > >
> > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org> wrote:
> > >
> > > > Hey,
> > > >
> > > > So what is the current decision regarding the time of the upcoming
> > > release?
> > > >
> > > > As for the streaming component, we included all the features we
> wanted,
> > > we
> > > > will start to test everything tomorrow, making sure that all works as
> > > > intended.
> > > > We are also almost finished with cleaning up the connector
> dependencies
> > > > that Stephan pointed out, should be finished by tomorrow.
> > > >
> > > > Gyula
> > > >
> > > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org> wrote:
> > > >
> > > > > +1 to manage this on JIRA (if possible)
> > > > >
> > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:
> > > > >
> > > > >> Can we not manage this stuff on Jira?
> > > > >>
> > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > >>> I personally like the idea of SOFT time-based feature freezes.
> > > > Otherwise,
> > > > >>> releases will get delayed again and again, because of features
> that
> > > we
> > > > >> try
> > > > >>> to squeeze in.
> > > > >>>
> > > > >>> Having reached the freeze point already, we could still include
> the
> > > > >>> features that are pending ready state in the next days
> (streaming,
> > > blob
> > > > >>> Manager, POJOs), but otherwise head for a release state.
> > > > >>>
> > > > >>> We had a mail listing the issues to include into 0.7, but a wiki
> > page
> > > > >> would
> > > > >>> probably be better. In that sense, we could start collecting the
> > > issues
> > > > >> for
> > > > >>> the next release from now on.
> > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <warneke@apache.org
> >:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I like Fabian's idea. Is there a wiki page (or something
> similar)
> > > > where
> > > > >> we
> > > > >>>> can collect the proposed JIRAs?
> > > > >>>>
> > > > >>>> Best regards,
> > > > >>>>
> > > > >>>>    Daniel
> > > > >>>>
> > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > > >>>>
> > > > >>>>> I agree, a hard feature stop deadline might not be the best
> > > practice.
> > > > >>>>>
> > > > >>>>> How about the following procedure:
> > > > >>>>> We decide two (or three) weeks before a targeted release date
> > about
> > > > >> which
> > > > >>>>> JIRAs to include. JIRAs that are selected for a release should
> be
> > > > >>>>> completed
> > > > >>>>> or really close to completion (via progress estimates in JIRA).
> > > > >>>>> After we decided which JIRAs to include in a release, we can
> use
> > > JIRA
> > > > >> to
> > > > >>>>> track the progress and dedicate another week exclusively for
> > > testing
> > > > >> after
> > > > >>>>> the last feature was completed.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >:
> > > > >>>>>
> > > > >>>>> As for the streaming team we're also getting ready for the
> > release,
> > > > >> but a
> > > > >>>>>> couple of days will be needed to finish the features that we
> > would
> > > > >> like
> > > > >>>>>> to
> > > > >>>>>> include.
> > > > >>>>>>
> > > > >>>>>>    - A little work is still needed for reduce operations and
> > > > >>>>>>    groups/connected streams (any comment on Gyula's recent
> > e-mail
> > > is
> > > > >>>>>> really
> > > > >>>>>>    appreciated :) )
> > > > >>>>>>    - The examples are being updated to match the standard,
> check
> > > out
> > > > >> the
> > > > >>>>>>    WordCount. (
> > > > >>>>>>
> > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > >>>>>> )
> > > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org>
> > > > wrote:
> > > > >>>>>>
> > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <rmetzger@apache.org
> >
> > > > >> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Hey guys,
> > > > >>>>>>>>
> > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for
> > the
> > > > >>>>>>>> 0.7-incubating release today.
> > > > >>>>>>>>
> > > > >>>>>>>> From our initial feature list:
> > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship
> > the
> > > > >>>>>>>>
> > > > >>>>>>> streaming,
> > > > >>>>>>>
> > > > >>>>>>>> but clearly mark it as a preview in the documentation.
> > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key
> selection
> > > > >> using a
> > > > >>>>>>>> string-expression:
> > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with
> the
> > > > >> latest
> > > > >>>>>>>> developments in the Java API:
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > > >>>>>>>>  -* Akka-based RPC service*:
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > > >>>>>>>>  - *Kryo-based serialization*. This feature has been
> requested
> > > by
> > > > >> many
> > > > >>>>>>>> users. Mostly because they wanted to use Collections inside
> > > POJOs:
> > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > > >>>>>>>> - Rework JobManager internals to support incremental program
> > > > >> rollout &
> > > > >>>>>>>> execution
> > > > >>>>>>>> - First parts of dynamic memory assignments
> > > > >>>>>>>>
> > > > >>>>>>>> The following features are in the master, as of today:
> > > > >>>>>>>> - *Flink Streaming*
> > > > >>>>>>>> - *Reworked Scala API*
> > > > >>>>>>>> -* New Scheduler*
> > > > >>>>>>>>
> > > > >>>>>>>> We certainly need some days to test everything until we can
> > > start
> > > > >> the
> > > > >>>>>>>>
> > > > >>>>>>> vote.
> > > > >>>>>>>
> > > > >>>>>>>> Based on our experience with the last major release, I would
> > > > really
> > > > >>>>>>>>
> > > > >>>>>>> like
> > > > >>>>>>
> > > > >>>>>>> to
> > > > >>>>>>>
> > > > >>>>>>>> do the testing and bugfixing BEFORE the first release
> > candidate.
> > > > For
> > > > >>>>>>>>
> > > > >>>>>>> the
> > > > >>>>>>
> > > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > > >>>>>>>>
> > > > >>>>>>>> How do you guys feel about this? Should we wait a few more
> > days
> > > > for
> > > > >> the
> > > > >>>>>>>> release so that a few more features make it into the
> release?
> > > > >>>>>>>>
> > > > >>>>>>>> I'm undecided on this. On the one hand, its really nice to
> > > release
> > > > >> on a
> > > > >>>>>>>> regular schedule, but it also eats up some time and causes
> > > > overhead
> > > > >>>>>>>> (different branches etc.).
> > > > >>>>>>>> I would really like to have the Java API Pojo improvements
> in
> > > the
> > > > >>>>>>>>
> > > > >>>>>>> release.
> > > > >>>>>>>
> > > > >>>>>>>> I think I can finish it until end of this week.
> > > > >>>>>>>>
> > > > >>>>>>>> Opinions?
> > > > >>>>>>>>
> > > > >>>>>>> I agree that the finished features (especially the Scala API)
> > are
> > > > >> nice
> > > > >>>>>>>
> > > > >>>>>> for
> > > > >>>>>>
> > > > >>>>>>> a new release, but still I would like to wait a few more
> days.
> > > > >>>>>>>
> > > > >>>>>>> Some of the missing features are on the brink of being
> finished
> > > > (e.g.
> > > > >>>>>>> the
> > > > >>>>>>> Pojo improvements). I wouldn't want to invest a week in bug
> > > fixing
> > > > >> and
> > > > >>>>>>> doing the release vote, when the new features are likely to
> be
> > > > >> finished
> > > > >>>>>>> just a few days afterwards. And the upcoming features will
> > > > >> definitely be
> > > > >>>>>>> worth a release, so users can work with them. ;)
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
I just checked the "Run Example" quickstart and it needs a bit of work.

2014-09-30 22:41 GMT+02:00 Robert Metzger <rm...@apache.org>:

> I'm working hard on getting the POJOs ready.
>
> We also should do a pass over our documentation, the quickstarts and the
> website to see if everything is in a good state (for example the
> collection-based execution needs documentation as well). We should also
> finally document the hadoop-input format wrappers (I think Timo is working
> on a pull request for that).
> This page mentions the LocalDistributedExecutor and contains some (most
> likely outdated) scala code:
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
>
> We also need to deprecate the old record api (
> https://issues.apache.org/jira/browse/FLINK-1106).
>
> On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > I think we are approaching ready state.
> > Last issues are going in and we started working on dependencies and test
> > platform diversity in order to make stabilizing phase.
> >
> > We should have an official feature freeze soon and fork the 0.7-release
> > branch. I personally vote to include the POJO support (I think Robert is
> > sorting that one out and is close to completion), and I want to add the
> > collection-based execution (today or tomorrow). Till has the BLOB manager
> > ready, which would be good to include (better support large libraries or
> > fat jars).
> >
> > After that, I vote to freeze.
> >
> > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org> wrote:
> >
> > > Hey,
> > >
> > > So what is the current decision regarding the time of the upcoming
> > release?
> > >
> > > As for the streaming component, we included all the features we wanted,
> > we
> > > will start to test everything tomorrow, making sure that all works as
> > > intended.
> > > We are also almost finished with cleaning up the connector dependencies
> > > that Stephan pointed out, should be finished by tomorrow.
> > >
> > > Gyula
> > >
> > > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org> wrote:
> > >
> > > > +1 to manage this on JIRA (if possible)
> > > >
> > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:
> > > >
> > > >> Can we not manage this stuff on Jira?
> > > >>
> > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > >>> I personally like the idea of SOFT time-based feature freezes.
> > > Otherwise,
> > > >>> releases will get delayed again and again, because of features that
> > we
> > > >> try
> > > >>> to squeeze in.
> > > >>>
> > > >>> Having reached the freeze point already, we could still include the
> > > >>> features that are pending ready state in the next days (streaming,
> > blob
> > > >>> Manager, POJOs), but otherwise head for a release state.
> > > >>>
> > > >>> We had a mail listing the issues to include into 0.7, but a wiki
> page
> > > >> would
> > > >>> probably be better. In that sense, we could start collecting the
> > issues
> > > >> for
> > > >>> the next release from now on.
> > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> I like Fabian's idea. Is there a wiki page (or something similar)
> > > where
> > > >> we
> > > >>>> can collect the proposed JIRAs?
> > > >>>>
> > > >>>> Best regards,
> > > >>>>
> > > >>>>    Daniel
> > > >>>>
> > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > > >>>>
> > > >>>>> I agree, a hard feature stop deadline might not be the best
> > practice.
> > > >>>>>
> > > >>>>> How about the following procedure:
> > > >>>>> We decide two (or three) weeks before a targeted release date
> about
> > > >> which
> > > >>>>> JIRAs to include. JIRAs that are selected for a release should be
> > > >>>>> completed
> > > >>>>> or really close to completion (via progress estimates in JIRA).
> > > >>>>> After we decided which JIRAs to include in a release, we can use
> > JIRA
> > > >> to
> > > >>>>> track the progress and dedicate another week exclusively for
> > testing
> > > >> after
> > > >>>>> the last feature was completed.
> > > >>>>>
> > > >>>>>
> > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> > balassi.marton@gmail.com
> > > >:
> > > >>>>>
> > > >>>>> As for the streaming team we're also getting ready for the
> release,
> > > >> but a
> > > >>>>>> couple of days will be needed to finish the features that we
> would
> > > >> like
> > > >>>>>> to
> > > >>>>>> include.
> > > >>>>>>
> > > >>>>>>    - A little work is still needed for reduce operations and
> > > >>>>>>    groups/connected streams (any comment on Gyula's recent
> e-mail
> > is
> > > >>>>>> really
> > > >>>>>>    appreciated :) )
> > > >>>>>>    - The examples are being updated to match the standard, check
> > out
> > > >> the
> > > >>>>>>    WordCount. (
> > > >>>>>>
> > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > >>>>>> )
> > > >>>>>>    Hopefully it gives you some deja vu. :)
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org>
> > > >> wrote:
> > > >>>>>>>
> > > >>>>>>> Hey guys,
> > > >>>>>>>>
> > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for
> the
> > > >>>>>>>> 0.7-incubating release today.
> > > >>>>>>>>
> > > >>>>>>>> From our initial feature list:
> > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship
> the
> > > >>>>>>>>
> > > >>>>>>> streaming,
> > > >>>>>>>
> > > >>>>>>>> but clearly mark it as a preview in the documentation.
> > > >>>>>>>> -* Java API Pojo improvements*: Code generation, key selection
> > > >> using a
> > > >>>>>>>> string-expression:
> > > https://issues.apache.org/jira/browse/FLINK-1032
> > > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with the
> > > >> latest
> > > >>>>>>>> developments in the Java API:
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > > >>>>>>>>  -* Akka-based RPC service*:
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > > >>>>>>>>  - *Kryo-based serialization*. This feature has been requested
> > by
> > > >> many
> > > >>>>>>>> users. Mostly because they wanted to use Collections inside
> > POJOs:
> > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > > >>>>>>>> - Rework JobManager internals to support incremental program
> > > >> rollout &
> > > >>>>>>>> execution
> > > >>>>>>>> - First parts of dynamic memory assignments
> > > >>>>>>>>
> > > >>>>>>>> The following features are in the master, as of today:
> > > >>>>>>>> - *Flink Streaming*
> > > >>>>>>>> - *Reworked Scala API*
> > > >>>>>>>> -* New Scheduler*
> > > >>>>>>>>
> > > >>>>>>>> We certainly need some days to test everything until we can
> > start
> > > >> the
> > > >>>>>>>>
> > > >>>>>>> vote.
> > > >>>>>>>
> > > >>>>>>>> Based on our experience with the last major release, I would
> > > really
> > > >>>>>>>>
> > > >>>>>>> like
> > > >>>>>>
> > > >>>>>>> to
> > > >>>>>>>
> > > >>>>>>>> do the testing and bugfixing BEFORE the first release
> candidate.
> > > For
> > > >>>>>>>>
> > > >>>>>>> the
> > > >>>>>>
> > > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > > >>>>>>>>
> > > >>>>>>>> How do you guys feel about this? Should we wait a few more
> days
> > > for
> > > >> the
> > > >>>>>>>> release so that a few more features make it into the release?
> > > >>>>>>>>
> > > >>>>>>>> I'm undecided on this. On the one hand, its really nice to
> > release
> > > >> on a
> > > >>>>>>>> regular schedule, but it also eats up some time and causes
> > > overhead
> > > >>>>>>>> (different branches etc.).
> > > >>>>>>>> I would really like to have the Java API Pojo improvements in
> > the
> > > >>>>>>>>
> > > >>>>>>> release.
> > > >>>>>>>
> > > >>>>>>>> I think I can finish it until end of this week.
> > > >>>>>>>>
> > > >>>>>>>> Opinions?
> > > >>>>>>>>
> > > >>>>>>> I agree that the finished features (especially the Scala API)
> are
> > > >> nice
> > > >>>>>>>
> > > >>>>>> for
> > > >>>>>>
> > > >>>>>>> a new release, but still I would like to wait a few more days.
> > > >>>>>>>
> > > >>>>>>> Some of the missing features are on the brink of being finished
> > > (e.g.
> > > >>>>>>> the
> > > >>>>>>> Pojo improvements). I wouldn't want to invest a week in bug
> > fixing
> > > >> and
> > > >>>>>>> doing the release vote, when the new features are likely to be
> > > >> finished
> > > >>>>>>> just a few days afterwards. And the upcoming features will
> > > >> definitely be
> > > >>>>>>> worth a release, so users can work with them. ;)
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
I'm working hard on getting the POJOs ready.

We also should do a pass over our documentation, the quickstarts and the
website to see if everything is in a good state (for example the
collection-based execution needs documentation as well). We should also
finally document the hadoop-input format wrappers (I think Timo is working
on a pull request for that).
This page mentions the LocalDistributedExecutor and contains some (most
likely outdated) scala code:
http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html

We also need to deprecate the old record api (
https://issues.apache.org/jira/browse/FLINK-1106).

On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen <se...@apache.org> wrote:

> I think we are approaching ready state.
> Last issues are going in and we started working on dependencies and test
> platform diversity in order to make stabilizing phase.
>
> We should have an official feature freeze soon and fork the 0.7-release
> branch. I personally vote to include the POJO support (I think Robert is
> sorting that one out and is close to completion), and I want to add the
> collection-based execution (today or tomorrow). Till has the BLOB manager
> ready, which would be good to include (better support large libraries or
> fat jars).
>
> After that, I vote to freeze.
>
> On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org> wrote:
>
> > Hey,
> >
> > So what is the current decision regarding the time of the upcoming
> release?
> >
> > As for the streaming component, we included all the features we wanted,
> we
> > will start to test everything tomorrow, making sure that all works as
> > intended.
> > We are also almost finished with cleaning up the connector dependencies
> > that Stephan pointed out, should be finished by tomorrow.
> >
> > Gyula
> >
> > On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org> wrote:
> >
> > > +1 to manage this on JIRA (if possible)
> > >
> > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:
> > >
> > >> Can we not manage this stuff on Jira?
> > >>
> > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org>
> > wrote:
> > >>> I personally like the idea of SOFT time-based feature freezes.
> > Otherwise,
> > >>> releases will get delayed again and again, because of features that
> we
> > >> try
> > >>> to squeeze in.
> > >>>
> > >>> Having reached the freeze point already, we could still include the
> > >>> features that are pending ready state in the next days (streaming,
> blob
> > >>> Manager, POJOs), but otherwise head for a release state.
> > >>>
> > >>> We had a mail listing the issues to include into 0.7, but a wiki page
> > >> would
> > >>> probably be better. In that sense, we could start collecting the
> issues
> > >> for
> > >>> the next release from now on.
> > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> I like Fabian's idea. Is there a wiki page (or something similar)
> > where
> > >> we
> > >>>> can collect the proposed JIRAs?
> > >>>>
> > >>>> Best regards,
> > >>>>
> > >>>>    Daniel
> > >>>>
> > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> > >>>>
> > >>>>> I agree, a hard feature stop deadline might not be the best
> practice.
> > >>>>>
> > >>>>> How about the following procedure:
> > >>>>> We decide two (or three) weeks before a targeted release date about
> > >> which
> > >>>>> JIRAs to include. JIRAs that are selected for a release should be
> > >>>>> completed
> > >>>>> or really close to completion (via progress estimates in JIRA).
> > >>>>> After we decided which JIRAs to include in a release, we can use
> JIRA
> > >> to
> > >>>>> track the progress and dedicate another week exclusively for
> testing
> > >> after
> > >>>>> the last feature was completed.
> > >>>>>
> > >>>>>
> > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <
> balassi.marton@gmail.com
> > >:
> > >>>>>
> > >>>>> As for the streaming team we're also getting ready for the release,
> > >> but a
> > >>>>>> couple of days will be needed to finish the features that we would
> > >> like
> > >>>>>> to
> > >>>>>> include.
> > >>>>>>
> > >>>>>>    - A little work is still needed for reduce operations and
> > >>>>>>    groups/connected streams (any comment on Gyula's recent e-mail
> is
> > >>>>>> really
> > >>>>>>    appreciated :) )
> > >>>>>>    - The examples are being updated to match the standard, check
> out
> > >> the
> > >>>>>>    WordCount. (
> > >>>>>>
> > >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> > >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> > >>>>>> streaming-examples/src/main/java/org/apache/flink/
> > >>>>>> streaming/examples/wordcount/WordCount.java
> > >>>>>> )
> > >>>>>>    Hopefully it gives you some deja vu. :)
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> Hey guys,
> > >>>>>>>>
> > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
> > >>>>>>>> 0.7-incubating release today.
> > >>>>>>>>
> > >>>>>>>> From our initial feature list:
> > >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > >>>>>>>>
> > >>>>>>> streaming,
> > >>>>>>>
> > >>>>>>>> but clearly mark it as a preview in the documentation.
> > >>>>>>>> -* Java API Pojo improvements*: Code generation, key selection
> > >> using a
> > >>>>>>>> string-expression:
> > https://issues.apache.org/jira/browse/FLINK-1032
> > >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with the
> > >> latest
> > >>>>>>>> developments in the Java API:
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> > >>>>>>>>  -* Akka-based RPC service*:
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> > >>>>>>>>  - *Kryo-based serialization*. This feature has been requested
> by
> > >> many
> > >>>>>>>> users. Mostly because they wanted to use Collections inside
> POJOs:
> > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> > >>>>>>>> - Rework JobManager internals to support incremental program
> > >> rollout &
> > >>>>>>>> execution
> > >>>>>>>> - First parts of dynamic memory assignments
> > >>>>>>>>
> > >>>>>>>> The following features are in the master, as of today:
> > >>>>>>>> - *Flink Streaming*
> > >>>>>>>> - *Reworked Scala API*
> > >>>>>>>> -* New Scheduler*
> > >>>>>>>>
> > >>>>>>>> We certainly need some days to test everything until we can
> start
> > >> the
> > >>>>>>>>
> > >>>>>>> vote.
> > >>>>>>>
> > >>>>>>>> Based on our experience with the last major release, I would
> > really
> > >>>>>>>>
> > >>>>>>> like
> > >>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>>> do the testing and bugfixing BEFORE the first release candidate.
> > For
> > >>>>>>>>
> > >>>>>>> the
> > >>>>>>
> > >>>>>>> 0.6-incubating release, we had 6  candidates)
> > >>>>>>>>
> > >>>>>>>> How do you guys feel about this? Should we wait a few more days
> > for
> > >> the
> > >>>>>>>> release so that a few more features make it into the release?
> > >>>>>>>>
> > >>>>>>>> I'm undecided on this. On the one hand, its really nice to
> release
> > >> on a
> > >>>>>>>> regular schedule, but it also eats up some time and causes
> > overhead
> > >>>>>>>> (different branches etc.).
> > >>>>>>>> I would really like to have the Java API Pojo improvements in
> the
> > >>>>>>>>
> > >>>>>>> release.
> > >>>>>>>
> > >>>>>>>> I think I can finish it until end of this week.
> > >>>>>>>>
> > >>>>>>>> Opinions?
> > >>>>>>>>
> > >>>>>>> I agree that the finished features (especially the Scala API) are
> > >> nice
> > >>>>>>>
> > >>>>>> for
> > >>>>>>
> > >>>>>>> a new release, but still I would like to wait a few more days.
> > >>>>>>>
> > >>>>>>> Some of the missing features are on the brink of being finished
> > (e.g.
> > >>>>>>> the
> > >>>>>>> Pojo improvements). I wouldn't want to invest a week in bug
> fixing
> > >> and
> > >>>>>>> doing the release vote, when the new features are likely to be
> > >> finished
> > >>>>>>> just a few days afterwards. And the upcoming features will
> > >> definitely be
> > >>>>>>> worth a release, so users can work with them. ;)
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
I think we are approaching ready state.
Last issues are going in and we started working on dependencies and test
platform diversity in order to make stabilizing phase.

We should have an official feature freeze soon and fork the 0.7-release
branch. I personally vote to include the POJO support (I think Robert is
sorting that one out and is close to completion), and I want to add the
collection-based execution (today or tomorrow). Till has the BLOB manager
ready, which would be good to include (better support large libraries or
fat jars).

After that, I vote to freeze.

On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora <gy...@apache.org> wrote:

> Hey,
>
> So what is the current decision regarding the time of the upcoming release?
>
> As for the streaming component, we included all the features we wanted, we
> will start to test everything tomorrow, making sure that all works as
> intended.
> We are also almost finished with cleaning up the connector dependencies
> that Stephan pointed out, should be finished by tomorrow.
>
> Gyula
>
> On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org> wrote:
>
> > +1 to manage this on JIRA (if possible)
> >
> > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:
> >
> >> Can we not manage this stuff on Jira?
> >>
> >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >>> I personally like the idea of SOFT time-based feature freezes.
> Otherwise,
> >>> releases will get delayed again and again, because of features that we
> >> try
> >>> to squeeze in.
> >>>
> >>> Having reached the freeze point already, we could still include the
> >>> features that are pending ready state in the next days (streaming, blob
> >>> Manager, POJOs), but otherwise head for a release state.
> >>>
> >>> We had a mail listing the issues to include into 0.7, but a wiki page
> >> would
> >>> probably be better. In that sense, we could start collecting the issues
> >> for
> >>> the next release from now on.
> >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I like Fabian's idea. Is there a wiki page (or something similar)
> where
> >> we
> >>>> can collect the proposed JIRAs?
> >>>>
> >>>> Best regards,
> >>>>
> >>>>    Daniel
> >>>>
> >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> >>>>
> >>>>> I agree, a hard feature stop deadline might not be the best practice.
> >>>>>
> >>>>> How about the following procedure:
> >>>>> We decide two (or three) weeks before a targeted release date about
> >> which
> >>>>> JIRAs to include. JIRAs that are selected for a release should be
> >>>>> completed
> >>>>> or really close to completion (via progress estimates in JIRA).
> >>>>> After we decided which JIRAs to include in a release, we can use JIRA
> >> to
> >>>>> track the progress and dedicate another week exclusively for testing
> >> after
> >>>>> the last feature was completed.
> >>>>>
> >>>>>
> >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <balassi.marton@gmail.com
> >:
> >>>>>
> >>>>> As for the streaming team we're also getting ready for the release,
> >> but a
> >>>>>> couple of days will be needed to finish the features that we would
> >> like
> >>>>>> to
> >>>>>> include.
> >>>>>>
> >>>>>>    - A little work is still needed for reduce operations and
> >>>>>>    groups/connected streams (any comment on Gyula's recent e-mail is
> >>>>>> really
> >>>>>>    appreciated :) )
> >>>>>>    - The examples are being updated to match the standard, check out
> >> the
> >>>>>>    WordCount. (
> >>>>>>
> >>>>>> https://github.com/mbalassi/incubator-flink/blob/
> >>>>>> streaming-new/flink-addons/flink-streaming/flink-
> >>>>>> streaming-examples/src/main/java/org/apache/flink/
> >>>>>> streaming/examples/wordcount/WordCount.java
> >>>>>> )
> >>>>>>    Hopefully it gives you some deja vu. :)
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org>
> wrote:
> >>>>>>
> >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> Hey guys,
> >>>>>>>>
> >>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
> >>>>>>>> 0.7-incubating release today.
> >>>>>>>>
> >>>>>>>> From our initial feature list:
> >>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
> >>>>>>>>
> >>>>>>> streaming,
> >>>>>>>
> >>>>>>>> but clearly mark it as a preview in the documentation.
> >>>>>>>> -* Java API Pojo improvements*: Code generation, key selection
> >> using a
> >>>>>>>> string-expression:
> https://issues.apache.org/jira/browse/FLINK-1032
> >>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with the
> >> latest
> >>>>>>>> developments in the Java API:
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
> >>>>>>>>  -* Akka-based RPC service*:
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> >>>>>>>>  - *Kryo-based serialization*. This feature has been requested by
> >> many
> >>>>>>>> users. Mostly because they wanted to use Collections inside POJOs:
> >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
> >>>>>>>> - Rework JobManager internals to support incremental program
> >> rollout &
> >>>>>>>> execution
> >>>>>>>> - First parts of dynamic memory assignments
> >>>>>>>>
> >>>>>>>> The following features are in the master, as of today:
> >>>>>>>> - *Flink Streaming*
> >>>>>>>> - *Reworked Scala API*
> >>>>>>>> -* New Scheduler*
> >>>>>>>>
> >>>>>>>> We certainly need some days to test everything until we can start
> >> the
> >>>>>>>>
> >>>>>>> vote.
> >>>>>>>
> >>>>>>>> Based on our experience with the last major release, I would
> really
> >>>>>>>>
> >>>>>>> like
> >>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>>> do the testing and bugfixing BEFORE the first release candidate.
> For
> >>>>>>>>
> >>>>>>> the
> >>>>>>
> >>>>>>> 0.6-incubating release, we had 6  candidates)
> >>>>>>>>
> >>>>>>>> How do you guys feel about this? Should we wait a few more days
> for
> >> the
> >>>>>>>> release so that a few more features make it into the release?
> >>>>>>>>
> >>>>>>>> I'm undecided on this. On the one hand, its really nice to release
> >> on a
> >>>>>>>> regular schedule, but it also eats up some time and causes
> overhead
> >>>>>>>> (different branches etc.).
> >>>>>>>> I would really like to have the Java API Pojo improvements in the
> >>>>>>>>
> >>>>>>> release.
> >>>>>>>
> >>>>>>>> I think I can finish it until end of this week.
> >>>>>>>>
> >>>>>>>> Opinions?
> >>>>>>>>
> >>>>>>> I agree that the finished features (especially the Scala API) are
> >> nice
> >>>>>>>
> >>>>>> for
> >>>>>>
> >>>>>>> a new release, but still I would like to wait a few more days.
> >>>>>>>
> >>>>>>> Some of the missing features are on the brink of being finished
> (e.g.
> >>>>>>> the
> >>>>>>> Pojo improvements). I wouldn't want to invest a week in bug fixing
> >> and
> >>>>>>> doing the release vote, when the new features are likely to be
> >> finished
> >>>>>>> just a few days afterwards. And the upcoming features will
> >> definitely be
> >>>>>>> worth a release, so users can work with them. ;)
> >>>>>>>
> >>>>>>
> >>>>
> >>
>
>

Re: Planning Flink release 0.7-incubating

Posted by Gyula Fora <gy...@apache.org>.
Hey,

So what is the current decision regarding the time of the upcoming release?

As for the streaming component, we included all the features we wanted, we will start to test everything tomorrow, making sure that all works as intended. 
We are also almost finished with cleaning up the connector dependencies that Stephan pointed out, should be finished by tomorrow.

Gyula

On 26 Sep 2014, at 10:49, Fabian Hueske <fh...@apache.org> wrote:

> +1 to manage this on JIRA (if possible)
> 
> 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:
> 
>> Can we not manage this stuff on Jira?
>> 
>> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org> wrote:
>>> I personally like the idea of SOFT time-based feature freezes. Otherwise,
>>> releases will get delayed again and again, because of features that we
>> try
>>> to squeeze in.
>>> 
>>> Having reached the freeze point already, we could still include the
>>> features that are pending ready state in the next days (streaming, blob
>>> Manager, POJOs), but otherwise head for a release state.
>>> 
>>> We had a mail listing the issues to include into 0.7, but a wiki page
>> would
>>> probably be better. In that sense, we could start collecting the issues
>> for
>>> the next release from now on.
>>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
>>> 
>>>> Hi,
>>>> 
>>>> I like Fabian's idea. Is there a wiki page (or something similar) where
>> we
>>>> can collect the proposed JIRAs?
>>>> 
>>>> Best regards,
>>>> 
>>>>    Daniel
>>>> 
>>>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
>>>> 
>>>>> I agree, a hard feature stop deadline might not be the best practice.
>>>>> 
>>>>> How about the following procedure:
>>>>> We decide two (or three) weeks before a targeted release date about
>> which
>>>>> JIRAs to include. JIRAs that are selected for a release should be
>>>>> completed
>>>>> or really close to completion (via progress estimates in JIRA).
>>>>> After we decided which JIRAs to include in a release, we can use JIRA
>> to
>>>>> track the progress and dedicate another week exclusively for testing
>> after
>>>>> the last feature was completed.
>>>>> 
>>>>> 
>>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:
>>>>> 
>>>>> As for the streaming team we're also getting ready for the release,
>> but a
>>>>>> couple of days will be needed to finish the features that we would
>> like
>>>>>> to
>>>>>> include.
>>>>>> 
>>>>>>    - A little work is still needed for reduce operations and
>>>>>>    groups/connected streams (any comment on Gyula's recent e-mail is
>>>>>> really
>>>>>>    appreciated :) )
>>>>>>    - The examples are being updated to match the standard, check out
>> the
>>>>>>    WordCount. (
>>>>>> 
>>>>>> https://github.com/mbalassi/incubator-flink/blob/
>>>>>> streaming-new/flink-addons/flink-streaming/flink-
>>>>>> streaming-examples/src/main/java/org/apache/flink/
>>>>>> streaming/examples/wordcount/WordCount.java
>>>>>> )
>>>>>>    Hopefully it gives you some deja vu. :)
>>>>>> 
>>>>>> 
>>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>>> 
>>>>>> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> Hey guys,
>>>>>>>> 
>>>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
>>>>>>>> 0.7-incubating release today.
>>>>>>>> 
>>>>>>>> From our initial feature list:
>>>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
>>>>>>>> 
>>>>>>> streaming,
>>>>>>> 
>>>>>>>> but clearly mark it as a preview in the documentation.
>>>>>>>> -* Java API Pojo improvements*: Code generation, key selection
>> using a
>>>>>>>> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>>>>>>>>  - *Reworked Scala API*. Bring the Scala API in sync with the
>> latest
>>>>>>>> developments in the Java API:
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-641
>>>>>>>>  -* Akka-based RPC service*:
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
>>>>>>>>  - *Kryo-based serialization*. This feature has been requested by
>> many
>>>>>>>> users. Mostly because they wanted to use Collections inside POJOs:
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-610
>>>>>>>> - Rework JobManager internals to support incremental program
>> rollout &
>>>>>>>> execution
>>>>>>>> - First parts of dynamic memory assignments
>>>>>>>> 
>>>>>>>> The following features are in the master, as of today:
>>>>>>>> - *Flink Streaming*
>>>>>>>> - *Reworked Scala API*
>>>>>>>> -* New Scheduler*
>>>>>>>> 
>>>>>>>> We certainly need some days to test everything until we can start
>> the
>>>>>>>> 
>>>>>>> vote.
>>>>>>> 
>>>>>>>> Based on our experience with the last major release, I would really
>>>>>>>> 
>>>>>>> like
>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>>> do the testing and bugfixing BEFORE the first release candidate. For
>>>>>>>> 
>>>>>>> the
>>>>>> 
>>>>>>> 0.6-incubating release, we had 6  candidates)
>>>>>>>> 
>>>>>>>> How do you guys feel about this? Should we wait a few more days for
>> the
>>>>>>>> release so that a few more features make it into the release?
>>>>>>>> 
>>>>>>>> I'm undecided on this. On the one hand, its really nice to release
>> on a
>>>>>>>> regular schedule, but it also eats up some time and causes overhead
>>>>>>>> (different branches etc.).
>>>>>>>> I would really like to have the Java API Pojo improvements in the
>>>>>>>> 
>>>>>>> release.
>>>>>>> 
>>>>>>>> I think I can finish it until end of this week.
>>>>>>>> 
>>>>>>>> Opinions?
>>>>>>>> 
>>>>>>> I agree that the finished features (especially the Scala API) are
>> nice
>>>>>>> 
>>>>>> for
>>>>>> 
>>>>>>> a new release, but still I would like to wait a few more days.
>>>>>>> 
>>>>>>> Some of the missing features are on the brink of being finished (e.g.
>>>>>>> the
>>>>>>> Pojo improvements). I wouldn't want to invest a week in bug fixing
>> and
>>>>>>> doing the release vote, when the new features are likely to be
>> finished
>>>>>>> just a few days afterwards. And the upcoming features will
>> definitely be
>>>>>>> worth a release, so users can work with them. ;)
>>>>>>> 
>>>>>> 
>>>> 
>>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
+1 to manage this on JIRA (if possible)

2014-09-26 10:40 GMT+02:00 Aljoscha Krettek <al...@apache.org>:

> Can we not manage this stuff on Jira?
>
> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org> wrote:
> > I personally like the idea of SOFT time-based feature freezes. Otherwise,
> > releases will get delayed again and again, because of features that we
> try
> > to squeeze in.
> >
> > Having reached the freeze point already, we could still include the
> > features that are pending ready state in the next days (streaming, blob
> > Manager, POJOs), but otherwise head for a release state.
> >
> > We had a mail listing the issues to include into 0.7, but a wiki page
> would
> > probably be better. In that sense, we could start collecting the issues
> for
> > the next release from now on.
> >  Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
> >
> >> Hi,
> >>
> >> I like Fabian's idea. Is there a wiki page (or something similar) where
> we
> >> can collect the proposed JIRAs?
> >>
> >> Best regards,
> >>
> >>     Daniel
> >>
> >> Am 24.09.2014 23:03, schrieb Fabian Hueske:
> >>
> >>> I agree, a hard feature stop deadline might not be the best practice.
> >>>
> >>> How about the following procedure:
> >>> We decide two (or three) weeks before a targeted release date about
> which
> >>> JIRAs to include. JIRAs that are selected for a release should be
> >>> completed
> >>> or really close to completion (via progress estimates in JIRA).
> >>> After we decided which JIRAs to include in a release, we can use JIRA
> to
> >>> track the progress and dedicate another week exclusively for testing
> after
> >>> the last feature was completed.
> >>>
> >>>
> >>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:
> >>>
> >>>  As for the streaming team we're also getting ready for the release,
> but a
> >>>> couple of days will be needed to finish the features that we would
> like
> >>>> to
> >>>> include.
> >>>>
> >>>>     - A little work is still needed for reduce operations and
> >>>>     groups/connected streams (any comment on Gyula's recent e-mail is
> >>>> really
> >>>>     appreciated :) )
> >>>>     - The examples are being updated to match the standard, check out
> the
> >>>>     WordCount. (
> >>>>
> >>>> https://github.com/mbalassi/incubator-flink/blob/
> >>>> streaming-new/flink-addons/flink-streaming/flink-
> >>>> streaming-examples/src/main/java/org/apache/flink/
> >>>> streaming/examples/wordcount/WordCount.java
> >>>> )
> >>>>     Hopefully it gives you some deja vu. :)
> >>>>
> >>>>
> >>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >>>>
> >>>>  On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org>
> wrote:
> >>>>>
> >>>>>  Hey guys,
> >>>>>>
> >>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
> >>>>>> 0.7-incubating release today.
> >>>>>>
> >>>>>>  From our initial feature list:
> >>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
> >>>>>>
> >>>>> streaming,
> >>>>>
> >>>>>> but clearly mark it as a preview in the documentation.
> >>>>>> -* Java API Pojo improvements*: Code generation, key selection
> using a
> >>>>>> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> >>>>>>   - *Reworked Scala API*. Bring the Scala API in sync with the
> latest
> >>>>>> developments in the Java API:
> >>>>>> https://issues.apache.org/jira/browse/FLINK-641
> >>>>>>   -* Akka-based RPC service*:
> >>>>>> https://issues.apache.org/jira/browse/FLINK-1019
> >>>>>>   - *Kryo-based serialization*. This feature has been requested by
> many
> >>>>>> users. Mostly because they wanted to use Collections inside POJOs:
> >>>>>> https://issues.apache.org/jira/browse/FLINK-610
> >>>>>> - Rework JobManager internals to support incremental program
> rollout &
> >>>>>> execution
> >>>>>> - First parts of dynamic memory assignments
> >>>>>>
> >>>>>> The following features are in the master, as of today:
> >>>>>> - *Flink Streaming*
> >>>>>> - *Reworked Scala API*
> >>>>>> -* New Scheduler*
> >>>>>>
> >>>>>> We certainly need some days to test everything until we can start
> the
> >>>>>>
> >>>>> vote.
> >>>>>
> >>>>>> Based on our experience with the last major release, I would really
> >>>>>>
> >>>>> like
> >>>>
> >>>>> to
> >>>>>
> >>>>>> do the testing and bugfixing BEFORE the first release candidate. For
> >>>>>>
> >>>>> the
> >>>>
> >>>>> 0.6-incubating release, we had 6  candidates)
> >>>>>>
> >>>>>> How do you guys feel about this? Should we wait a few more days for
> the
> >>>>>> release so that a few more features make it into the release?
> >>>>>>
> >>>>>> I'm undecided on this. On the one hand, its really nice to release
> on a
> >>>>>> regular schedule, but it also eats up some time and causes overhead
> >>>>>> (different branches etc.).
> >>>>>> I would really like to have the Java API Pojo improvements in the
> >>>>>>
> >>>>> release.
> >>>>>
> >>>>>> I think I can finish it until end of this week.
> >>>>>>
> >>>>>> Opinions?
> >>>>>>
> >>>>> I agree that the finished features (especially the Scala API) are
> nice
> >>>>>
> >>>> for
> >>>>
> >>>>> a new release, but still I would like to wait a few more days.
> >>>>>
> >>>>> Some of the missing features are on the brink of being finished (e.g.
> >>>>> the
> >>>>> Pojo improvements). I wouldn't want to invest a week in bug fixing
> and
> >>>>> doing the release vote, when the new features are likely to be
> finished
> >>>>> just a few days afterwards. And the upcoming features will
> definitely be
> >>>>> worth a release, so users can work with them. ;)
> >>>>>
> >>>>
> >>
>

Re: Planning Flink release 0.7-incubating

Posted by Aljoscha Krettek <al...@apache.org>.
Can we not manage this stuff on Jira?

On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen <se...@apache.org> wrote:
> I personally like the idea of SOFT time-based feature freezes. Otherwise,
> releases will get delayed again and again, because of features that we try
> to squeeze in.
>
> Having reached the freeze point already, we could still include the
> features that are pending ready state in the next days (streaming, blob
> Manager, POJOs), but otherwise head for a release state.
>
> We had a mail listing the issues to include into 0.7, but a wiki page would
> probably be better. In that sense, we could start collecting the issues for
> the next release from now on.
>  Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:
>
>> Hi,
>>
>> I like Fabian's idea. Is there a wiki page (or something similar) where we
>> can collect the proposed JIRAs?
>>
>> Best regards,
>>
>>     Daniel
>>
>> Am 24.09.2014 23:03, schrieb Fabian Hueske:
>>
>>> I agree, a hard feature stop deadline might not be the best practice.
>>>
>>> How about the following procedure:
>>> We decide two (or three) weeks before a targeted release date about which
>>> JIRAs to include. JIRAs that are selected for a release should be
>>> completed
>>> or really close to completion (via progress estimates in JIRA).
>>> After we decided which JIRAs to include in a release, we can use JIRA to
>>> track the progress and dedicate another week exclusively for testing after
>>> the last feature was completed.
>>>
>>>
>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:
>>>
>>>  As for the streaming team we're also getting ready for the release, but a
>>>> couple of days will be needed to finish the features that we would like
>>>> to
>>>> include.
>>>>
>>>>     - A little work is still needed for reduce operations and
>>>>     groups/connected streams (any comment on Gyula's recent e-mail is
>>>> really
>>>>     appreciated :) )
>>>>     - The examples are being updated to match the standard, check out the
>>>>     WordCount. (
>>>>
>>>> https://github.com/mbalassi/incubator-flink/blob/
>>>> streaming-new/flink-addons/flink-streaming/flink-
>>>> streaming-examples/src/main/java/org/apache/flink/
>>>> streaming/examples/wordcount/WordCount.java
>>>> )
>>>>     Hopefully it gives you some deja vu. :)
>>>>
>>>>
>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>
>>>>  On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:
>>>>>
>>>>>  Hey guys,
>>>>>>
>>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
>>>>>> 0.7-incubating release today.
>>>>>>
>>>>>>  From our initial feature list:
>>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
>>>>>>
>>>>> streaming,
>>>>>
>>>>>> but clearly mark it as a preview in the documentation.
>>>>>> -* Java API Pojo improvements*: Code generation, key selection using a
>>>>>> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>>>>>>   - *Reworked Scala API*. Bring the Scala API in sync with the latest
>>>>>> developments in the Java API:
>>>>>> https://issues.apache.org/jira/browse/FLINK-641
>>>>>>   -* Akka-based RPC service*:
>>>>>> https://issues.apache.org/jira/browse/FLINK-1019
>>>>>>   - *Kryo-based serialization*. This feature has been requested by many
>>>>>> users. Mostly because they wanted to use Collections inside POJOs:
>>>>>> https://issues.apache.org/jira/browse/FLINK-610
>>>>>> - Rework JobManager internals to support incremental program rollout &
>>>>>> execution
>>>>>> - First parts of dynamic memory assignments
>>>>>>
>>>>>> The following features are in the master, as of today:
>>>>>> - *Flink Streaming*
>>>>>> - *Reworked Scala API*
>>>>>> -* New Scheduler*
>>>>>>
>>>>>> We certainly need some days to test everything until we can start the
>>>>>>
>>>>> vote.
>>>>>
>>>>>> Based on our experience with the last major release, I would really
>>>>>>
>>>>> like
>>>>
>>>>> to
>>>>>
>>>>>> do the testing and bugfixing BEFORE the first release candidate. For
>>>>>>
>>>>> the
>>>>
>>>>> 0.6-incubating release, we had 6  candidates)
>>>>>>
>>>>>> How do you guys feel about this? Should we wait a few more days for the
>>>>>> release so that a few more features make it into the release?
>>>>>>
>>>>>> I'm undecided on this. On the one hand, its really nice to release on a
>>>>>> regular schedule, but it also eats up some time and causes overhead
>>>>>> (different branches etc.).
>>>>>> I would really like to have the Java API Pojo improvements in the
>>>>>>
>>>>> release.
>>>>>
>>>>>> I think I can finish it until end of this week.
>>>>>>
>>>>>> Opinions?
>>>>>>
>>>>> I agree that the finished features (especially the Scala API) are nice
>>>>>
>>>> for
>>>>
>>>>> a new release, but still I would like to wait a few more days.
>>>>>
>>>>> Some of the missing features are on the brink of being finished (e.g.
>>>>> the
>>>>> Pojo improvements). I wouldn't want to invest a week in bug fixing and
>>>>> doing the release vote, when the new features are likely to be finished
>>>>> just a few days afterwards. And the upcoming features will definitely be
>>>>> worth a release, so users can work with them. ;)
>>>>>
>>>>
>>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
I personally like the idea of SOFT time-based feature freezes. Otherwise,
releases will get delayed again and again, because of features that we try
to squeeze in.

Having reached the freeze point already, we could still include the
features that are pending ready state in the next days (streaming, blob
Manager, POJOs), but otherwise head for a release state.

We had a mail listing the issues to include into 0.7, but a wiki page would
probably be better. In that sense, we could start collecting the issues for
the next release from now on.
 Am 26.09.2014 09:17 schrieb "Daniel Warneke" <wa...@apache.org>:

> Hi,
>
> I like Fabian's idea. Is there a wiki page (or something similar) where we
> can collect the proposed JIRAs?
>
> Best regards,
>
>     Daniel
>
> Am 24.09.2014 23:03, schrieb Fabian Hueske:
>
>> I agree, a hard feature stop deadline might not be the best practice.
>>
>> How about the following procedure:
>> We decide two (or three) weeks before a targeted release date about which
>> JIRAs to include. JIRAs that are selected for a release should be
>> completed
>> or really close to completion (via progress estimates in JIRA).
>> After we decided which JIRAs to include in a release, we can use JIRA to
>> track the progress and dedicate another week exclusively for testing after
>> the last feature was completed.
>>
>>
>> 2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:
>>
>>  As for the streaming team we're also getting ready for the release, but a
>>> couple of days will be needed to finish the features that we would like
>>> to
>>> include.
>>>
>>>     - A little work is still needed for reduce operations and
>>>     groups/connected streams (any comment on Gyula's recent e-mail is
>>> really
>>>     appreciated :) )
>>>     - The examples are being updated to match the standard, check out the
>>>     WordCount. (
>>>
>>> https://github.com/mbalassi/incubator-flink/blob/
>>> streaming-new/flink-addons/flink-streaming/flink-
>>> streaming-examples/src/main/java/org/apache/flink/
>>> streaming/examples/wordcount/WordCount.java
>>> )
>>>     Hopefully it gives you some deja vu. :)
>>>
>>>
>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>
>>>  On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:
>>>>
>>>>  Hey guys,
>>>>>
>>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
>>>>> 0.7-incubating release today.
>>>>>
>>>>>  From our initial feature list:
>>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
>>>>>
>>>> streaming,
>>>>
>>>>> but clearly mark it as a preview in the documentation.
>>>>> -* Java API Pojo improvements*: Code generation, key selection using a
>>>>> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>>>>>   - *Reworked Scala API*. Bring the Scala API in sync with the latest
>>>>> developments in the Java API:
>>>>> https://issues.apache.org/jira/browse/FLINK-641
>>>>>   -* Akka-based RPC service*:
>>>>> https://issues.apache.org/jira/browse/FLINK-1019
>>>>>   - *Kryo-based serialization*. This feature has been requested by many
>>>>> users. Mostly because they wanted to use Collections inside POJOs:
>>>>> https://issues.apache.org/jira/browse/FLINK-610
>>>>> - Rework JobManager internals to support incremental program rollout &
>>>>> execution
>>>>> - First parts of dynamic memory assignments
>>>>>
>>>>> The following features are in the master, as of today:
>>>>> - *Flink Streaming*
>>>>> - *Reworked Scala API*
>>>>> -* New Scheduler*
>>>>>
>>>>> We certainly need some days to test everything until we can start the
>>>>>
>>>> vote.
>>>>
>>>>> Based on our experience with the last major release, I would really
>>>>>
>>>> like
>>>
>>>> to
>>>>
>>>>> do the testing and bugfixing BEFORE the first release candidate. For
>>>>>
>>>> the
>>>
>>>> 0.6-incubating release, we had 6  candidates)
>>>>>
>>>>> How do you guys feel about this? Should we wait a few more days for the
>>>>> release so that a few more features make it into the release?
>>>>>
>>>>> I'm undecided on this. On the one hand, its really nice to release on a
>>>>> regular schedule, but it also eats up some time and causes overhead
>>>>> (different branches etc.).
>>>>> I would really like to have the Java API Pojo improvements in the
>>>>>
>>>> release.
>>>>
>>>>> I think I can finish it until end of this week.
>>>>>
>>>>> Opinions?
>>>>>
>>>> I agree that the finished features (especially the Scala API) are nice
>>>>
>>> for
>>>
>>>> a new release, but still I would like to wait a few more days.
>>>>
>>>> Some of the missing features are on the brink of being finished (e.g.
>>>> the
>>>> Pojo improvements). I wouldn't want to invest a week in bug fixing and
>>>> doing the release vote, when the new features are likely to be finished
>>>> just a few days afterwards. And the upcoming features will definitely be
>>>> worth a release, so users can work with them. ;)
>>>>
>>>
>

Re: Planning Flink release 0.7-incubating

Posted by Daniel Warneke <wa...@apache.org>.
Hi,

I like Fabian's idea. Is there a wiki page (or something similar) where 
we can collect the proposed JIRAs?

Best regards,

     Daniel

Am 24.09.2014 23:03, schrieb Fabian Hueske:
> I agree, a hard feature stop deadline might not be the best practice.
>
> How about the following procedure:
> We decide two (or three) weeks before a targeted release date about which
> JIRAs to include. JIRAs that are selected for a release should be completed
> or really close to completion (via progress estimates in JIRA).
> After we decided which JIRAs to include in a release, we can use JIRA to
> track the progress and dedicate another week exclusively for testing after
> the last feature was completed.
>
>
> 2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:
>
>> As for the streaming team we're also getting ready for the release, but a
>> couple of days will be needed to finish the features that we would like to
>> include.
>>
>>     - A little work is still needed for reduce operations and
>>     groups/connected streams (any comment on Gyula's recent e-mail is really
>>     appreciated :) )
>>     - The examples are being updated to match the standard, check out the
>>     WordCount. (
>>
>> https://github.com/mbalassi/incubator-flink/blob/streaming-new/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java
>> )
>>     Hopefully it gives you some deja vu. :)
>>
>>
>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:
>>>
>>>> Hey guys,
>>>>
>>>> exactly 3 weeks ago, we discussed to do a feature freeze for the
>>>> 0.7-incubating release today.
>>>>
>>>>  From our initial feature list:
>>>> - *Flink Streaming* "Beta Preview". I would suggest to ship the
>>> streaming,
>>>> but clearly mark it as a preview in the documentation.
>>>> -* Java API Pojo improvements*: Code generation, key selection using a
>>>> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>>>>   - *Reworked Scala API*. Bring the Scala API in sync with the latest
>>>> developments in the Java API:
>>>> https://issues.apache.org/jira/browse/FLINK-641
>>>>   -* Akka-based RPC service*:
>>>> https://issues.apache.org/jira/browse/FLINK-1019
>>>>   - *Kryo-based serialization*. This feature has been requested by many
>>>> users. Mostly because they wanted to use Collections inside POJOs:
>>>> https://issues.apache.org/jira/browse/FLINK-610
>>>> - Rework JobManager internals to support incremental program rollout &
>>>> execution
>>>> - First parts of dynamic memory assignments
>>>>
>>>> The following features are in the master, as of today:
>>>> - *Flink Streaming*
>>>> - *Reworked Scala API*
>>>> -* New Scheduler*
>>>>
>>>> We certainly need some days to test everything until we can start the
>>> vote.
>>>> Based on our experience with the last major release, I would really
>> like
>>> to
>>>> do the testing and bugfixing BEFORE the first release candidate. For
>> the
>>>> 0.6-incubating release, we had 6  candidates)
>>>>
>>>> How do you guys feel about this? Should we wait a few more days for the
>>>> release so that a few more features make it into the release?
>>>>
>>>> I'm undecided on this. On the one hand, its really nice to release on a
>>>> regular schedule, but it also eats up some time and causes overhead
>>>> (different branches etc.).
>>>> I would really like to have the Java API Pojo improvements in the
>>> release.
>>>> I think I can finish it until end of this week.
>>>>
>>>> Opinions?
>>> I agree that the finished features (especially the Scala API) are nice
>> for
>>> a new release, but still I would like to wait a few more days.
>>>
>>> Some of the missing features are on the brink of being finished (e.g. the
>>> Pojo improvements). I wouldn't want to invest a week in bug fixing and
>>> doing the release vote, when the new features are likely to be finished
>>> just a few days afterwards. And the upcoming features will definitely be
>>> worth a release, so users can work with them. ;)

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
I agree, a hard feature stop deadline might not be the best practice.

How about the following procedure:
We decide two (or three) weeks before a targeted release date about which
JIRAs to include. JIRAs that are selected for a release should be completed
or really close to completion (via progress estimates in JIRA).
After we decided which JIRAs to include in a release, we can use JIRA to
track the progress and dedicate another week exclusively for testing after
the last feature was completed.


2014-09-24 19:10 GMT+02:00 Márton Balassi <ba...@gmail.com>:

> As for the streaming team we're also getting ready for the release, but a
> couple of days will be needed to finish the features that we would like to
> include.
>
>    - A little work is still needed for reduce operations and
>    groups/connected streams (any comment on Gyula's recent e-mail is really
>    appreciated :) )
>    - The examples are being updated to match the standard, check out the
>    WordCount. (
>
> https://github.com/mbalassi/incubator-flink/blob/streaming-new/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java
> )
>    Hopefully it gives you some deja vu. :)
>
>
> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:
> >
> > > Hey guys,
> > >
> > > exactly 3 weeks ago, we discussed to do a feature freeze for the
> > > 0.7-incubating release today.
> > >
> > > From our initial feature list:
> > > - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > streaming,
> > > but clearly mark it as a preview in the documentation.
> > > -* Java API Pojo improvements*: Code generation, key selection using a
> > > string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> > >  - *Reworked Scala API*. Bring the Scala API in sync with the latest
> > > developments in the Java API:
> > > https://issues.apache.org/jira/browse/FLINK-641
> > >  -* Akka-based RPC service*:
> > > https://issues.apache.org/jira/browse/FLINK-1019
> > >  - *Kryo-based serialization*. This feature has been requested by many
> > > users. Mostly because they wanted to use Collections inside POJOs:
> > > https://issues.apache.org/jira/browse/FLINK-610
> > > - Rework JobManager internals to support incremental program rollout &
> > > execution
> > > - First parts of dynamic memory assignments
> > >
> > > The following features are in the master, as of today:
> > > - *Flink Streaming*
> > > - *Reworked Scala API*
> > > -* New Scheduler*
> > >
> > > We certainly need some days to test everything until we can start the
> > vote.
> > > Based on our experience with the last major release, I would really
> like
> > to
> > > do the testing and bugfixing BEFORE the first release candidate. For
> the
> > > 0.6-incubating release, we had 6  candidates)
> > >
> > > How do you guys feel about this? Should we wait a few more days for the
> > > release so that a few more features make it into the release?
> > >
> > > I'm undecided on this. On the one hand, its really nice to release on a
> > > regular schedule, but it also eats up some time and causes overhead
> > > (different branches etc.).
> > > I would really like to have the Java API Pojo improvements in the
> > release.
> > > I think I can finish it until end of this week.
> > >
> > > Opinions?
> >
> > I agree that the finished features (especially the Scala API) are nice
> for
> > a new release, but still I would like to wait a few more days.
> >
> > Some of the missing features are on the brink of being finished (e.g. the
> > Pojo improvements). I wouldn't want to invest a week in bug fixing and
> > doing the release vote, when the new features are likely to be finished
> > just a few days afterwards. And the upcoming features will definitely be
> > worth a release, so users can work with them. ;)
>

Re: Planning Flink release 0.7-incubating

Posted by Márton Balassi <ba...@gmail.com>.
As for the streaming team we're also getting ready for the release, but a
couple of days will be needed to finish the features that we would like to
include.

   - A little work is still needed for reduce operations and
   groups/connected streams (any comment on Gyula's recent e-mail is really
   appreciated :) )
   - The examples are being updated to match the standard, check out the
   WordCount. (
   https://github.com/mbalassi/incubator-flink/blob/streaming-new/flink-addons/flink-streaming/flink-streaming-examples/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java)
   Hopefully it gives you some deja vu. :)


On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi <uc...@apache.org> wrote:

>
> On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:
>
> > Hey guys,
> >
> > exactly 3 weeks ago, we discussed to do a feature freeze for the
> > 0.7-incubating release today.
> >
> > From our initial feature list:
> > - *Flink Streaming* "Beta Preview". I would suggest to ship the
> streaming,
> > but clearly mark it as a preview in the documentation.
> > -* Java API Pojo improvements*: Code generation, key selection using a
> > string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> >  - *Reworked Scala API*. Bring the Scala API in sync with the latest
> > developments in the Java API:
> > https://issues.apache.org/jira/browse/FLINK-641
> >  -* Akka-based RPC service*:
> > https://issues.apache.org/jira/browse/FLINK-1019
> >  - *Kryo-based serialization*. This feature has been requested by many
> > users. Mostly because they wanted to use Collections inside POJOs:
> > https://issues.apache.org/jira/browse/FLINK-610
> > - Rework JobManager internals to support incremental program rollout &
> > execution
> > - First parts of dynamic memory assignments
> >
> > The following features are in the master, as of today:
> > - *Flink Streaming*
> > - *Reworked Scala API*
> > -* New Scheduler*
> >
> > We certainly need some days to test everything until we can start the
> vote.
> > Based on our experience with the last major release, I would really like
> to
> > do the testing and bugfixing BEFORE the first release candidate. For the
> > 0.6-incubating release, we had 6  candidates)
> >
> > How do you guys feel about this? Should we wait a few more days for the
> > release so that a few more features make it into the release?
> >
> > I'm undecided on this. On the one hand, its really nice to release on a
> > regular schedule, but it also eats up some time and causes overhead
> > (different branches etc.).
> > I would really like to have the Java API Pojo improvements in the
> release.
> > I think I can finish it until end of this week.
> >
> > Opinions?
>
> I agree that the finished features (especially the Scala API) are nice for
> a new release, but still I would like to wait a few more days.
>
> Some of the missing features are on the brink of being finished (e.g. the
> Pojo improvements). I wouldn't want to invest a week in bug fixing and
> doing the release vote, when the new features are likely to be finished
> just a few days afterwards. And the upcoming features will definitely be
> worth a release, so users can work with them. ;)

Re: Planning Flink release 0.7-incubating

Posted by Ufuk Celebi <uc...@apache.org>.
On 24 Sep 2014, at 18:37, Robert Metzger <rm...@apache.org> wrote:

> Hey guys,
> 
> exactly 3 weeks ago, we discussed to do a feature freeze for the
> 0.7-incubating release today.
> 
> From our initial feature list:
> - *Flink Streaming* "Beta Preview". I would suggest to ship the streaming,
> but clearly mark it as a preview in the documentation.
> -* Java API Pojo improvements*: Code generation, key selection using a
> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>  - *Reworked Scala API*. Bring the Scala API in sync with the latest
> developments in the Java API:
> https://issues.apache.org/jira/browse/FLINK-641
>  -* Akka-based RPC service*:
> https://issues.apache.org/jira/browse/FLINK-1019
>  - *Kryo-based serialization*. This feature has been requested by many
> users. Mostly because they wanted to use Collections inside POJOs:
> https://issues.apache.org/jira/browse/FLINK-610
> - Rework JobManager internals to support incremental program rollout &
> execution
> - First parts of dynamic memory assignments
> 
> The following features are in the master, as of today:
> - *Flink Streaming*
> - *Reworked Scala API*
> -* New Scheduler*
> 
> We certainly need some days to test everything until we can start the vote.
> Based on our experience with the last major release, I would really like to
> do the testing and bugfixing BEFORE the first release candidate. For the
> 0.6-incubating release, we had 6  candidates)
> 
> How do you guys feel about this? Should we wait a few more days for the
> release so that a few more features make it into the release?
> 
> I'm undecided on this. On the one hand, its really nice to release on a
> regular schedule, but it also eats up some time and causes overhead
> (different branches etc.).
> I would really like to have the Java API Pojo improvements in the release.
> I think I can finish it until end of this week.
> 
> Opinions?

I agree that the finished features (especially the Scala API) are nice for a new release, but still I would like to wait a few more days.

Some of the missing features are on the brink of being finished (e.g. the Pojo improvements). I wouldn't want to invest a week in bug fixing and doing the release vote, when the new features are likely to be finished just a few days afterwards. And the upcoming features will definitely be worth a release, so users can work with them. ;)

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
Hey guys,

exactly 3 weeks ago, we discussed to do a feature freeze for the
0.7-incubating release today.

>From our initial feature list:
 - *Flink Streaming* "Beta Preview". I would suggest to ship the streaming,
but clearly mark it as a preview in the documentation.
 -* Java API Pojo improvements*: Code generation, key selection using a
string-expression: https://issues.apache.org/jira/browse/FLINK-1032
  - *Reworked Scala API*. Bring the Scala API in sync with the latest
developments in the Java API:
https://issues.apache.org/jira/browse/FLINK-641
  -* Akka-based RPC service*:
https://issues.apache.org/jira/browse/FLINK-1019
  - *Kryo-based serialization*. This feature has been requested by many
users. Mostly because they wanted to use Collections inside POJOs:
https://issues.apache.org/jira/browse/FLINK-610
 - Rework JobManager internals to support incremental program rollout &
execution
 - First parts of dynamic memory assignments

The following features are in the master, as of today:
 - *Flink Streaming*
 - *Reworked Scala API*
 -* New Scheduler*

We certainly need some days to test everything until we can start the vote.
Based on our experience with the last major release, I would really like to
do the testing and bugfixing BEFORE the first release candidate. For the
0.6-incubating release, we had 6  candidates)

How do you guys feel about this? Should we wait a few more days for the
release so that a few more features make it into the release?

I'm undecided on this. On the one hand, its really nice to release on a
regular schedule, but it also eats up some time and causes overhead
(different branches etc.).
I would really like to have the Java API Pojo improvements in the release.
I think I can finish it until end of this week.

Opinions?


Cheers,
Robert




On Fri, Sep 5, 2014 at 8:55 AM, Kostas Tzoumas <kt...@apache.org> wrote:

> +1
>
>
> On Thu, Sep 4, 2014 at 12:00 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > +1
> > 2014.09.04. 11:52 ezt írta ("Stephan Ewen" <se...@apache.org>):
> >
> > > +1
> > > Am 04.09.2014 00:05 schrieb "Fabian Hueske" <fh...@apache.org>:
> > >
> > > > +1
> > > >
> > > >
> > > > 2014-09-03 22:39 GMT+02:00 Kostas Tzoumas <kt...@apache.org>:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <
> > > > balassi.marton@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <
> > trohrmann@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 Sounds good to me.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > sorry, my previous message was confusing. I suggest to
> release
> > a
> > > > new
> > > > > > > Flink
> > > > > > > > version all 3 months.
> > > > > > > > BUT, the 0.7-incubating release is going to be feature freeze
> > in
> > > 3
> > > > > > weeks
> > > > > > > > because 0.6-incubating was more about getting the release
> infra
> > > set
> > > > > up
> > > > > > > and
> > > > > > > > the apache rename out.
> > > > > > > > The last release that contained a lot of features was 0.5 and
> > it
> > > > was
> > > > > > > > released on May 29. So aiming for end of September for the
> next
> > > > > feature
> > > > > > > > release should be somehow in the schedule.
> > > > > > > >
> > > > > > > >
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uce@apache.org
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Robert,
> > > > > > > > >
> > > > > > > > > +1 to frequent regular frequent major releases (I guess you
> > > > meant 3
> > > > > > > weeks
> > > > > > > > > and not 3 months, right?).
> > > > > > > > >
> > > > > > > > > Ufuk
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <
> > > > > rmetzger@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > > I agree with Fabian that the list of features is a lot of
> > > > work. I
> > > > > > > would
> > > > > > > > > > prefer to have frequent regular major releases (say 3
> > months
> > > > > > > schedule).
> > > > > > > > > > This way, users can quickly access the latest features
> and
> > we
> > > > > don't
> > > > > > > > force
> > > > > > > > > > them to use SNAPSHOT versions.
> > > > > > > > > > Also, releases draw attention to our project.
> > > > > > > > > >
> > > > > > > > > > I would suggest to do a feature freeze on September 24 (3
> > > weeks
> > > > > > from
> > > > > > > > now)
> > > > > > > > > > and start the vote a few days afterwards. I assume that
> at
> > > > least
> > > > > > some
> > > > > > > > of
> > > > > > > > > > the suggested 7 features of the release are ready.
> > > > > > > > > >
> > > > > > > > > > Robert
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <
> > > > > fhueske@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I agree that these should be features to add soon, but
> > I'm
> > > a
> > > > > bit
> > > > > > > > > doubtful
> > > > > > > > > > > that we will have the next release in 5 weeks if we
> want
> > to
> > > > > > include
> > > > > > > > all
> > > > > > > > > > of
> > > > > > > > > > > this.
> > > > > > > > > > >
> > > > > > > > > > > I think we should either have a feature- or
> time-oriented
> > > > > release
> > > > > > > > plan.
> > > > > > > > > > > If we want to have a fixed release date, we could make
> a
> > > > > feature
> > > > > > > > stop 1
> > > > > > > > > > > week (or so) in advance, include what's in until then
> and
> > > > > > continue
> > > > > > > to
> > > > > > > > > > > release every n weeks.
> > > > > > > > > > > Or we make a list of what should be in the next release
> > and
> > > > try
> > > > > > to
> > > > > > > > > give a
> > > > > > > > > > > reasonable time estimate for that (which didn't work
> out
> > so
> > > > > well
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > past.... ;-) )
> > > > > > > > > > >
> > > > > > > > > > > But having both, a close deadline and a long list of
> > > complex
> > > > > > > features
> > > > > > > > > > will
> > > > > > > > > > > not work out well, IMO.
> > > > > > > > > > >
> > > > > > > > > > > Just my 2 cents, Fabian
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <
> > sewen@apache.org
> > > >:
> > > > > > > > > > >
> > > > > > > > > > > > This is a nice list.
> > > > > > > > > > > >
> > > > > > > > > > > > I would like to add:
> > > > > > > > > > > >
> > > > > > > > > > > >  - Rework JobManager internals to support incremental
> > > > program
> > > > > > > > > rollout &
> > > > > > > > > > > > execution
> > > > > > > > > > > >  - First parts of dynamic memory assignments
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > > > > > > rmetzger@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > since we have our release infrastructure in place
> > now,
> > > I
> > > > > > would
> > > > > > > > > > suggest
> > > > > > > > > > > to
> > > > > > > > > > > > > aim for a 0.7-incubating release in the near future
> > > (say
> > > > > 3-5
> > > > > > > > > weeks).
> > > > > > > > > > > > > While 0.6-incubating was mainly about getting the
> > > release
> > > > > > > infra /
> > > > > > > > > > legal
> > > > > > > > > > > > > stuff sorted out, I think it would be nice to have
> a
> > > > > > "feature"
> > > > > > > > > > release
> > > > > > > > > > > > > soon.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The following new features would make a great
> > > > > 0.7-incubating
> > > > > > > > > release:
> > > > > > > > > > > > >  - *Flink Streaming* "Beta Preview". I would
> suggest
> > to
> > > > > ship
> > > > > > > the
> > > > > > > > > > > > streaming,
> > > > > > > > > > > > > but clearly mark it as a preview in the
> > documentation.
> > > > > > > > > > > > >  -* Java API Pojo improvements*: Code generation,
> key
> > > > > > selection
> > > > > > > > > > using a
> > > > > > > > > > > > > string-expression:
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > > >   - *Reworked Scala API*. Bring the Scala API in
> sync
> > > > with
> > > > > > the
> > > > > > > > > latest
> > > > > > > > > > > > > developments in the Java API:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > > >   - *Kryo-based serialization*. This feature has
> been
> > > > > > requested
> > > > > > > > by
> > > > > > > > > > many
> > > > > > > > > > > > > users. Mostly because they wanted to use
> Collections
> > > > inside
> > > > > > > > POJOs:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Opinions?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Kostas Tzoumas <kt...@apache.org>.
+1


On Thu, Sep 4, 2014 at 12:00 PM, Gyula Fóra <gy...@gmail.com> wrote:

> +1
> 2014.09.04. 11:52 ezt írta ("Stephan Ewen" <se...@apache.org>):
>
> > +1
> > Am 04.09.2014 00:05 schrieb "Fabian Hueske" <fh...@apache.org>:
> >
> > > +1
> > >
> > >
> > > 2014-09-03 22:39 GMT+02:00 Kostas Tzoumas <kt...@apache.org>:
> > >
> > > > +1
> > > >
> > > >
> > > > On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <
> > > balassi.marton@gmail.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <
> trohrmann@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 Sounds good to me.
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > sorry, my previous message was confusing. I suggest to release
> a
> > > new
> > > > > > Flink
> > > > > > > version all 3 months.
> > > > > > > BUT, the 0.7-incubating release is going to be feature freeze
> in
> > 3
> > > > > weeks
> > > > > > > because 0.6-incubating was more about getting the release infra
> > set
> > > > up
> > > > > > and
> > > > > > > the apache rename out.
> > > > > > > The last release that contained a lot of features was 0.5 and
> it
> > > was
> > > > > > > released on May 29. So aiming for end of September for the next
> > > > feature
> > > > > > > release should be somehow in the schedule.
> > > > > > >
> > > > > > >
> > > > > > > Robert
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > Hey Robert,
> > > > > > > >
> > > > > > > > +1 to frequent regular frequent major releases (I guess you
> > > meant 3
> > > > > > weeks
> > > > > > > > and not 3 months, right?).
> > > > > > > >
> > > > > > > > Ufuk
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <
> > > > rmetzger@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I agree with Fabian that the list of features is a lot of
> > > work. I
> > > > > > would
> > > > > > > > > prefer to have frequent regular major releases (say 3
> months
> > > > > > schedule).
> > > > > > > > > This way, users can quickly access the latest features and
> we
> > > > don't
> > > > > > > force
> > > > > > > > > them to use SNAPSHOT versions.
> > > > > > > > > Also, releases draw attention to our project.
> > > > > > > > >
> > > > > > > > > I would suggest to do a feature freeze on September 24 (3
> > weeks
> > > > > from
> > > > > > > now)
> > > > > > > > > and start the vote a few days afterwards. I assume that at
> > > least
> > > > > some
> > > > > > > of
> > > > > > > > > the suggested 7 features of the release are ready.
> > > > > > > > >
> > > > > > > > > Robert
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <
> > > > fhueske@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree that these should be features to add soon, but
> I'm
> > a
> > > > bit
> > > > > > > > doubtful
> > > > > > > > > > that we will have the next release in 5 weeks if we want
> to
> > > > > include
> > > > > > > all
> > > > > > > > > of
> > > > > > > > > > this.
> > > > > > > > > >
> > > > > > > > > > I think we should either have a feature- or time-oriented
> > > > release
> > > > > > > plan.
> > > > > > > > > > If we want to have a fixed release date, we could make a
> > > > feature
> > > > > > > stop 1
> > > > > > > > > > week (or so) in advance, include what's in until then and
> > > > > continue
> > > > > > to
> > > > > > > > > > release every n weeks.
> > > > > > > > > > Or we make a list of what should be in the next release
> and
> > > try
> > > > > to
> > > > > > > > give a
> > > > > > > > > > reasonable time estimate for that (which didn't work out
> so
> > > > well
> > > > > in
> > > > > > > the
> > > > > > > > > > past.... ;-) )
> > > > > > > > > >
> > > > > > > > > > But having both, a close deadline and a long list of
> > complex
> > > > > > features
> > > > > > > > > will
> > > > > > > > > > not work out well, IMO.
> > > > > > > > > >
> > > > > > > > > > Just my 2 cents, Fabian
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <
> sewen@apache.org
> > >:
> > > > > > > > > >
> > > > > > > > > > > This is a nice list.
> > > > > > > > > > >
> > > > > > > > > > > I would like to add:
> > > > > > > > > > >
> > > > > > > > > > >  - Rework JobManager internals to support incremental
> > > program
> > > > > > > > rollout &
> > > > > > > > > > > execution
> > > > > > > > > > >  - First parts of dynamic memory assignments
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > > > > > rmetzger@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > since we have our release infrastructure in place
> now,
> > I
> > > > > would
> > > > > > > > > suggest
> > > > > > > > > > to
> > > > > > > > > > > > aim for a 0.7-incubating release in the near future
> > (say
> > > > 3-5
> > > > > > > > weeks).
> > > > > > > > > > > > While 0.6-incubating was mainly about getting the
> > release
> > > > > > infra /
> > > > > > > > > legal
> > > > > > > > > > > > stuff sorted out, I think it would be nice to have a
> > > > > "feature"
> > > > > > > > > release
> > > > > > > > > > > > soon.
> > > > > > > > > > > >
> > > > > > > > > > > > The following new features would make a great
> > > > 0.7-incubating
> > > > > > > > release:
> > > > > > > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest
> to
> > > > ship
> > > > > > the
> > > > > > > > > > > streaming,
> > > > > > > > > > > > but clearly mark it as a preview in the
> documentation.
> > > > > > > > > > > >  -* Java API Pojo improvements*: Code generation, key
> > > > > selection
> > > > > > > > > using a
> > > > > > > > > > > > string-expression:
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync
> > > with
> > > > > the
> > > > > > > > latest
> > > > > > > > > > > > developments in the Java API:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > >   - *Kryo-based serialization*. This feature has been
> > > > > requested
> > > > > > > by
> > > > > > > > > many
> > > > > > > > > > > > users. Mostly because they wanted to use Collections
> > > inside
> > > > > > > POJOs:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Opinions?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Gyula Fóra <gy...@gmail.com>.
+1
2014.09.04. 11:52 ezt írta ("Stephan Ewen" <se...@apache.org>):

> +1
> Am 04.09.2014 00:05 schrieb "Fabian Hueske" <fh...@apache.org>:
>
> > +1
> >
> >
> > 2014-09-03 22:39 GMT+02:00 Kostas Tzoumas <kt...@apache.org>:
> >
> > > +1
> > >
> > >
> > > On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > >
> > > > On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <tr...@apache.org>
> > > > wrote:
> > > >
> > > > > +1 Sounds good to me.
> > > > >
> > > > >
> > > > > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > sorry, my previous message was confusing. I suggest to release a
> > new
> > > > > Flink
> > > > > > version all 3 months.
> > > > > > BUT, the 0.7-incubating release is going to be feature freeze in
> 3
> > > > weeks
> > > > > > because 0.6-incubating was more about getting the release infra
> set
> > > up
> > > > > and
> > > > > > the apache rename out.
> > > > > > The last release that contained a lot of features was 0.5 and it
> > was
> > > > > > released on May 29. So aiming for end of September for the next
> > > feature
> > > > > > release should be somehow in the schedule.
> > > > > >
> > > > > >
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Hey Robert,
> > > > > > >
> > > > > > > +1 to frequent regular frequent major releases (I guess you
> > meant 3
> > > > > weeks
> > > > > > > and not 3 months, right?).
> > > > > > >
> > > > > > > Ufuk
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <
> > > rmetzger@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > I agree with Fabian that the list of features is a lot of
> > work. I
> > > > > would
> > > > > > > > prefer to have frequent regular major releases (say 3 months
> > > > > schedule).
> > > > > > > > This way, users can quickly access the latest features and we
> > > don't
> > > > > > force
> > > > > > > > them to use SNAPSHOT versions.
> > > > > > > > Also, releases draw attention to our project.
> > > > > > > >
> > > > > > > > I would suggest to do a feature freeze on September 24 (3
> weeks
> > > > from
> > > > > > now)
> > > > > > > > and start the vote a few days afterwards. I assume that at
> > least
> > > > some
> > > > > > of
> > > > > > > > the suggested 7 features of the release are ready.
> > > > > > > >
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <
> > > fhueske@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree that these should be features to add soon, but I'm
> a
> > > bit
> > > > > > > doubtful
> > > > > > > > > that we will have the next release in 5 weeks if we want to
> > > > include
> > > > > > all
> > > > > > > > of
> > > > > > > > > this.
> > > > > > > > >
> > > > > > > > > I think we should either have a feature- or time-oriented
> > > release
> > > > > > plan.
> > > > > > > > > If we want to have a fixed release date, we could make a
> > > feature
> > > > > > stop 1
> > > > > > > > > week (or so) in advance, include what's in until then and
> > > > continue
> > > > > to
> > > > > > > > > release every n weeks.
> > > > > > > > > Or we make a list of what should be in the next release and
> > try
> > > > to
> > > > > > > give a
> > > > > > > > > reasonable time estimate for that (which didn't work out so
> > > well
> > > > in
> > > > > > the
> > > > > > > > > past.... ;-) )
> > > > > > > > >
> > > > > > > > > But having both, a close deadline and a long list of
> complex
> > > > > features
> > > > > > > > will
> > > > > > > > > not work out well, IMO.
> > > > > > > > >
> > > > > > > > > Just my 2 cents, Fabian
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <sewen@apache.org
> >:
> > > > > > > > >
> > > > > > > > > > This is a nice list.
> > > > > > > > > >
> > > > > > > > > > I would like to add:
> > > > > > > > > >
> > > > > > > > > >  - Rework JobManager internals to support incremental
> > program
> > > > > > > rollout &
> > > > > > > > > > execution
> > > > > > > > > >  - First parts of dynamic memory assignments
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > > > > rmetzger@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > since we have our release infrastructure in place now,
> I
> > > > would
> > > > > > > > suggest
> > > > > > > > > to
> > > > > > > > > > > aim for a 0.7-incubating release in the near future
> (say
> > > 3-5
> > > > > > > weeks).
> > > > > > > > > > > While 0.6-incubating was mainly about getting the
> release
> > > > > infra /
> > > > > > > > legal
> > > > > > > > > > > stuff sorted out, I think it would be nice to have a
> > > > "feature"
> > > > > > > > release
> > > > > > > > > > > soon.
> > > > > > > > > > >
> > > > > > > > > > > The following new features would make a great
> > > 0.7-incubating
> > > > > > > release:
> > > > > > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to
> > > ship
> > > > > the
> > > > > > > > > > streaming,
> > > > > > > > > > > but clearly mark it as a preview in the documentation.
> > > > > > > > > > >  -* Java API Pojo improvements*: Code generation, key
> > > > selection
> > > > > > > > using a
> > > > > > > > > > > string-expression:
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync
> > with
> > > > the
> > > > > > > latest
> > > > > > > > > > > developments in the Java API:
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > >   - *Kryo-based serialization*. This feature has been
> > > > requested
> > > > > > by
> > > > > > > > many
> > > > > > > > > > > users. Mostly because they wanted to use Collections
> > inside
> > > > > > POJOs:
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Opinions?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
+1
Am 04.09.2014 00:05 schrieb "Fabian Hueske" <fh...@apache.org>:

> +1
>
>
> 2014-09-03 22:39 GMT+02:00 Kostas Tzoumas <kt...@apache.org>:
>
> > +1
> >
> >
> > On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > >
> > > On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > +1 Sounds good to me.
> > > >
> > > >
> > > > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > sorry, my previous message was confusing. I suggest to release a
> new
> > > > Flink
> > > > > version all 3 months.
> > > > > BUT, the 0.7-incubating release is going to be feature freeze in 3
> > > weeks
> > > > > because 0.6-incubating was more about getting the release infra set
> > up
> > > > and
> > > > > the apache rename out.
> > > > > The last release that contained a lot of features was 0.5 and it
> was
> > > > > released on May 29. So aiming for end of September for the next
> > feature
> > > > > release should be somehow in the schedule.
> > > > >
> > > > >
> > > > > Robert
> > > > >
> > > > >
> > > > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org>
> wrote:
> > > > >
> > > > > > Hey Robert,
> > > > > >
> > > > > > +1 to frequent regular frequent major releases (I guess you
> meant 3
> > > > weeks
> > > > > > and not 3 months, right?).
> > > > > >
> > > > > > Ufuk
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I agree with Fabian that the list of features is a lot of
> work. I
> > > > would
> > > > > > > prefer to have frequent regular major releases (say 3 months
> > > > schedule).
> > > > > > > This way, users can quickly access the latest features and we
> > don't
> > > > > force
> > > > > > > them to use SNAPSHOT versions.
> > > > > > > Also, releases draw attention to our project.
> > > > > > >
> > > > > > > I would suggest to do a feature freeze on September 24 (3 weeks
> > > from
> > > > > now)
> > > > > > > and start the vote a few days afterwards. I assume that at
> least
> > > some
> > > > > of
> > > > > > > the suggested 7 features of the release are ready.
> > > > > > >
> > > > > > > Robert
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <
> > fhueske@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that these should be features to add soon, but I'm a
> > bit
> > > > > > doubtful
> > > > > > > > that we will have the next release in 5 weeks if we want to
> > > include
> > > > > all
> > > > > > > of
> > > > > > > > this.
> > > > > > > >
> > > > > > > > I think we should either have a feature- or time-oriented
> > release
> > > > > plan.
> > > > > > > > If we want to have a fixed release date, we could make a
> > feature
> > > > > stop 1
> > > > > > > > week (or so) in advance, include what's in until then and
> > > continue
> > > > to
> > > > > > > > release every n weeks.
> > > > > > > > Or we make a list of what should be in the next release and
> try
> > > to
> > > > > > give a
> > > > > > > > reasonable time estimate for that (which didn't work out so
> > well
> > > in
> > > > > the
> > > > > > > > past.... ;-) )
> > > > > > > >
> > > > > > > > But having both, a close deadline and a long list of complex
> > > > features
> > > > > > > will
> > > > > > > > not work out well, IMO.
> > > > > > > >
> > > > > > > > Just my 2 cents, Fabian
> > > > > > > >
> > > > > > > >
> > > > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > > > >
> > > > > > > > > This is a nice list.
> > > > > > > > >
> > > > > > > > > I would like to add:
> > > > > > > > >
> > > > > > > > >  - Rework JobManager internals to support incremental
> program
> > > > > > rollout &
> > > > > > > > > execution
> > > > > > > > >  - First parts of dynamic memory assignments
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > > > rmetzger@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > since we have our release infrastructure in place now, I
> > > would
> > > > > > > suggest
> > > > > > > > to
> > > > > > > > > > aim for a 0.7-incubating release in the near future (say
> > 3-5
> > > > > > weeks).
> > > > > > > > > > While 0.6-incubating was mainly about getting the release
> > > > infra /
> > > > > > > legal
> > > > > > > > > > stuff sorted out, I think it would be nice to have a
> > > "feature"
> > > > > > > release
> > > > > > > > > > soon.
> > > > > > > > > >
> > > > > > > > > > The following new features would make a great
> > 0.7-incubating
> > > > > > release:
> > > > > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to
> > ship
> > > > the
> > > > > > > > > streaming,
> > > > > > > > > > but clearly mark it as a preview in the documentation.
> > > > > > > > > >  -* Java API Pojo improvements*: Code generation, key
> > > selection
> > > > > > > using a
> > > > > > > > > > string-expression:
> > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync
> with
> > > the
> > > > > > latest
> > > > > > > > > > developments in the Java API:
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > >   - *Kryo-based serialization*. This feature has been
> > > requested
> > > > > by
> > > > > > > many
> > > > > > > > > > users. Mostly because they wanted to use Collections
> inside
> > > > > POJOs:
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Opinions?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
+1


2014-09-03 22:39 GMT+02:00 Kostas Tzoumas <kt...@apache.org>:

> +1
>
>
> On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > +1
> >
> >
> > On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > +1 Sounds good to me.
> > >
> > >
> > > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > sorry, my previous message was confusing. I suggest to release a new
> > > Flink
> > > > version all 3 months.
> > > > BUT, the 0.7-incubating release is going to be feature freeze in 3
> > weeks
> > > > because 0.6-incubating was more about getting the release infra set
> up
> > > and
> > > > the apache rename out.
> > > > The last release that contained a lot of features was 0.5 and it was
> > > > released on May 29. So aiming for end of September for the next
> feature
> > > > release should be somehow in the schedule.
> > > >
> > > >
> > > > Robert
> > > >
> > > >
> > > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > > Hey Robert,
> > > > >
> > > > > +1 to frequent regular frequent major releases (I guess you meant 3
> > > weeks
> > > > > and not 3 months, right?).
> > > > >
> > > > > Ufuk
> > > > >
> > > > >
> > > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <
> rmetzger@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I agree with Fabian that the list of features is a lot of work. I
> > > would
> > > > > > prefer to have frequent regular major releases (say 3 months
> > > schedule).
> > > > > > This way, users can quickly access the latest features and we
> don't
> > > > force
> > > > > > them to use SNAPSHOT versions.
> > > > > > Also, releases draw attention to our project.
> > > > > >
> > > > > > I would suggest to do a feature freeze on September 24 (3 weeks
> > from
> > > > now)
> > > > > > and start the vote a few days afterwards. I assume that at least
> > some
> > > > of
> > > > > > the suggested 7 features of the release are ready.
> > > > > >
> > > > > > Robert
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <
> fhueske@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I agree that these should be features to add soon, but I'm a
> bit
> > > > > doubtful
> > > > > > > that we will have the next release in 5 weeks if we want to
> > include
> > > > all
> > > > > > of
> > > > > > > this.
> > > > > > >
> > > > > > > I think we should either have a feature- or time-oriented
> release
> > > > plan.
> > > > > > > If we want to have a fixed release date, we could make a
> feature
> > > > stop 1
> > > > > > > week (or so) in advance, include what's in until then and
> > continue
> > > to
> > > > > > > release every n weeks.
> > > > > > > Or we make a list of what should be in the next release and try
> > to
> > > > > give a
> > > > > > > reasonable time estimate for that (which didn't work out so
> well
> > in
> > > > the
> > > > > > > past.... ;-) )
> > > > > > >
> > > > > > > But having both, a close deadline and a long list of complex
> > > features
> > > > > > will
> > > > > > > not work out well, IMO.
> > > > > > >
> > > > > > > Just my 2 cents, Fabian
> > > > > > >
> > > > > > >
> > > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > > >
> > > > > > > > This is a nice list.
> > > > > > > >
> > > > > > > > I would like to add:
> > > > > > > >
> > > > > > > >  - Rework JobManager internals to support incremental program
> > > > > rollout &
> > > > > > > > execution
> > > > > > > >  - First parts of dynamic memory assignments
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > > rmetzger@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > since we have our release infrastructure in place now, I
> > would
> > > > > > suggest
> > > > > > > to
> > > > > > > > > aim for a 0.7-incubating release in the near future (say
> 3-5
> > > > > weeks).
> > > > > > > > > While 0.6-incubating was mainly about getting the release
> > > infra /
> > > > > > legal
> > > > > > > > > stuff sorted out, I think it would be nice to have a
> > "feature"
> > > > > > release
> > > > > > > > > soon.
> > > > > > > > >
> > > > > > > > > The following new features would make a great
> 0.7-incubating
> > > > > release:
> > > > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to
> ship
> > > the
> > > > > > > > streaming,
> > > > > > > > > but clearly mark it as a preview in the documentation.
> > > > > > > > >  -* Java API Pojo improvements*: Code generation, key
> > selection
> > > > > > using a
> > > > > > > > > string-expression:
> > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync with
> > the
> > > > > latest
> > > > > > > > > developments in the Java API:
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > >   - *Kryo-based serialization*. This feature has been
> > requested
> > > > by
> > > > > > many
> > > > > > > > > users. Mostly because they wanted to use Collections inside
> > > > POJOs:
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Opinions?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Kostas Tzoumas <kt...@apache.org>.
+1


On Wed, Sep 3, 2014 at 10:12 PM, Márton Balassi <ba...@gmail.com>
wrote:

> +1
>
>
> On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > +1 Sounds good to me.
> >
> >
> > On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > sorry, my previous message was confusing. I suggest to release a new
> > Flink
> > > version all 3 months.
> > > BUT, the 0.7-incubating release is going to be feature freeze in 3
> weeks
> > > because 0.6-incubating was more about getting the release infra set up
> > and
> > > the apache rename out.
> > > The last release that contained a lot of features was 0.5 and it was
> > > released on May 29. So aiming for end of September for the next feature
> > > release should be somehow in the schedule.
> > >
> > >
> > > Robert
> > >
> > >
> > > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > Hey Robert,
> > > >
> > > > +1 to frequent regular frequent major releases (I guess you meant 3
> > weeks
> > > > and not 3 months, right?).
> > > >
> > > > Ufuk
> > > >
> > > >
> > > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <rm...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I agree with Fabian that the list of features is a lot of work. I
> > would
> > > > > prefer to have frequent regular major releases (say 3 months
> > schedule).
> > > > > This way, users can quickly access the latest features and we don't
> > > force
> > > > > them to use SNAPSHOT versions.
> > > > > Also, releases draw attention to our project.
> > > > >
> > > > > I would suggest to do a feature freeze on September 24 (3 weeks
> from
> > > now)
> > > > > and start the vote a few days afterwards. I assume that at least
> some
> > > of
> > > > > the suggested 7 features of the release are ready.
> > > > >
> > > > > Robert
> > > > >
> > > > >
> > > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fhueske@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > I agree that these should be features to add soon, but I'm a bit
> > > > doubtful
> > > > > > that we will have the next release in 5 weeks if we want to
> include
> > > all
> > > > > of
> > > > > > this.
> > > > > >
> > > > > > I think we should either have a feature- or time-oriented release
> > > plan.
> > > > > > If we want to have a fixed release date, we could make a feature
> > > stop 1
> > > > > > week (or so) in advance, include what's in until then and
> continue
> > to
> > > > > > release every n weeks.
> > > > > > Or we make a list of what should be in the next release and try
> to
> > > > give a
> > > > > > reasonable time estimate for that (which didn't work out so well
> in
> > > the
> > > > > > past.... ;-) )
> > > > > >
> > > > > > But having both, a close deadline and a long list of complex
> > features
> > > > > will
> > > > > > not work out well, IMO.
> > > > > >
> > > > > > Just my 2 cents, Fabian
> > > > > >
> > > > > >
> > > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > > >
> > > > > > > This is a nice list.
> > > > > > >
> > > > > > > I would like to add:
> > > > > > >
> > > > > > >  - Rework JobManager internals to support incremental program
> > > > rollout &
> > > > > > > execution
> > > > > > >  - First parts of dynamic memory assignments
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > > rmetzger@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > since we have our release infrastructure in place now, I
> would
> > > > > suggest
> > > > > > to
> > > > > > > > aim for a 0.7-incubating release in the near future (say 3-5
> > > > weeks).
> > > > > > > > While 0.6-incubating was mainly about getting the release
> > infra /
> > > > > legal
> > > > > > > > stuff sorted out, I think it would be nice to have a
> "feature"
> > > > > release
> > > > > > > > soon.
> > > > > > > >
> > > > > > > > The following new features would make a great 0.7-incubating
> > > > release:
> > > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to ship
> > the
> > > > > > > streaming,
> > > > > > > > but clearly mark it as a preview in the documentation.
> > > > > > > >  -* Java API Pojo improvements*: Code generation, key
> selection
> > > > > using a
> > > > > > > > string-expression:
> > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync with
> the
> > > > latest
> > > > > > > > developments in the Java API:
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > >   -* Akka-based RPC service*:
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > >   - *Kryo-based serialization*. This feature has been
> requested
> > > by
> > > > > many
> > > > > > > > users. Mostly because they wanted to use Collections inside
> > > POJOs:
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > >
> > > > > > > >
> > > > > > > > Opinions?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Márton Balassi <ba...@gmail.com>.
+1


On Wed, Sep 3, 2014 at 9:54 PM, Till Rohrmann <tr...@apache.org> wrote:

> +1 Sounds good to me.
>
>
> On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > sorry, my previous message was confusing. I suggest to release a new
> Flink
> > version all 3 months.
> > BUT, the 0.7-incubating release is going to be feature freeze in 3 weeks
> > because 0.6-incubating was more about getting the release infra set up
> and
> > the apache rename out.
> > The last release that contained a lot of features was 0.5 and it was
> > released on May 29. So aiming for end of September for the next feature
> > release should be somehow in the schedule.
> >
> >
> > Robert
> >
> >
> > On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Hey Robert,
> > >
> > > +1 to frequent regular frequent major releases (I guess you meant 3
> weeks
> > > and not 3 months, right?).
> > >
> > > Ufuk
> > >
> > >
> > > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > > I agree with Fabian that the list of features is a lot of work. I
> would
> > > > prefer to have frequent regular major releases (say 3 months
> schedule).
> > > > This way, users can quickly access the latest features and we don't
> > force
> > > > them to use SNAPSHOT versions.
> > > > Also, releases draw attention to our project.
> > > >
> > > > I would suggest to do a feature freeze on September 24 (3 weeks from
> > now)
> > > > and start the vote a few days afterwards. I assume that at least some
> > of
> > > > the suggested 7 features of the release are ready.
> > > >
> > > > Robert
> > > >
> > > >
> > > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fh...@apache.org>
> > > wrote:
> > > >
> > > > > I agree that these should be features to add soon, but I'm a bit
> > > doubtful
> > > > > that we will have the next release in 5 weeks if we want to include
> > all
> > > > of
> > > > > this.
> > > > >
> > > > > I think we should either have a feature- or time-oriented release
> > plan.
> > > > > If we want to have a fixed release date, we could make a feature
> > stop 1
> > > > > week (or so) in advance, include what's in until then and continue
> to
> > > > > release every n weeks.
> > > > > Or we make a list of what should be in the next release and try to
> > > give a
> > > > > reasonable time estimate for that (which didn't work out so well in
> > the
> > > > > past.... ;-) )
> > > > >
> > > > > But having both, a close deadline and a long list of complex
> features
> > > > will
> > > > > not work out well, IMO.
> > > > >
> > > > > Just my 2 cents, Fabian
> > > > >
> > > > >
> > > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > > >
> > > > > > This is a nice list.
> > > > > >
> > > > > > I would like to add:
> > > > > >
> > > > > >  - Rework JobManager internals to support incremental program
> > > rollout &
> > > > > > execution
> > > > > >  - First parts of dynamic memory assignments
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > since we have our release infrastructure in place now, I would
> > > > suggest
> > > > > to
> > > > > > > aim for a 0.7-incubating release in the near future (say 3-5
> > > weeks).
> > > > > > > While 0.6-incubating was mainly about getting the release
> infra /
> > > > legal
> > > > > > > stuff sorted out, I think it would be nice to have a "feature"
> > > > release
> > > > > > > soon.
> > > > > > >
> > > > > > > The following new features would make a great 0.7-incubating
> > > release:
> > > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to ship
> the
> > > > > > streaming,
> > > > > > > but clearly mark it as a preview in the documentation.
> > > > > > >  -* Java API Pojo improvements*: Code generation, key selection
> > > > using a
> > > > > > > string-expression:
> > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > >   - *Reworked Scala API*. Bring the Scala API in sync with the
> > > latest
> > > > > > > developments in the Java API:
> > > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > > >   -* Akka-based RPC service*:
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > >   - *Kryo-based serialization*. This feature has been requested
> > by
> > > > many
> > > > > > > users. Mostly because they wanted to use Collections inside
> > POJOs:
> > > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > > >
> > > > > > >
> > > > > > > Opinions?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Till Rohrmann <tr...@apache.org>.
+1 Sounds good to me.


On Wed, Sep 3, 2014 at 1:58 PM, Robert Metzger <rm...@apache.org> wrote:

> Hi,
>
> sorry, my previous message was confusing. I suggest to release a new Flink
> version all 3 months.
> BUT, the 0.7-incubating release is going to be feature freeze in 3 weeks
> because 0.6-incubating was more about getting the release infra set up and
> the apache rename out.
> The last release that contained a lot of features was 0.5 and it was
> released on May 29. So aiming for end of September for the next feature
> release should be somehow in the schedule.
>
>
> Robert
>
>
> On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > Hey Robert,
> >
> > +1 to frequent regular frequent major releases (I guess you meant 3 weeks
> > and not 3 months, right?).
> >
> > Ufuk
> >
> >
> > On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi,
> > > I agree with Fabian that the list of features is a lot of work. I would
> > > prefer to have frequent regular major releases (say 3 months schedule).
> > > This way, users can quickly access the latest features and we don't
> force
> > > them to use SNAPSHOT versions.
> > > Also, releases draw attention to our project.
> > >
> > > I would suggest to do a feature freeze on September 24 (3 weeks from
> now)
> > > and start the vote a few days afterwards. I assume that at least some
> of
> > > the suggested 7 features of the release are ready.
> > >
> > > Robert
> > >
> > >
> > > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fh...@apache.org>
> > wrote:
> > >
> > > > I agree that these should be features to add soon, but I'm a bit
> > doubtful
> > > > that we will have the next release in 5 weeks if we want to include
> all
> > > of
> > > > this.
> > > >
> > > > I think we should either have a feature- or time-oriented release
> plan.
> > > > If we want to have a fixed release date, we could make a feature
> stop 1
> > > > week (or so) in advance, include what's in until then and continue to
> > > > release every n weeks.
> > > > Or we make a list of what should be in the next release and try to
> > give a
> > > > reasonable time estimate for that (which didn't work out so well in
> the
> > > > past.... ;-) )
> > > >
> > > > But having both, a close deadline and a long list of complex features
> > > will
> > > > not work out well, IMO.
> > > >
> > > > Just my 2 cents, Fabian
> > > >
> > > >
> > > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > > >
> > > > > This is a nice list.
> > > > >
> > > > > I would like to add:
> > > > >
> > > > >  - Rework JobManager internals to support incremental program
> > rollout &
> > > > > execution
> > > > >  - First parts of dynamic memory assignments
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <
> rmetzger@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > since we have our release infrastructure in place now, I would
> > > suggest
> > > > to
> > > > > > aim for a 0.7-incubating release in the near future (say 3-5
> > weeks).
> > > > > > While 0.6-incubating was mainly about getting the release infra /
> > > legal
> > > > > > stuff sorted out, I think it would be nice to have a "feature"
> > > release
> > > > > > soon.
> > > > > >
> > > > > > The following new features would make a great 0.7-incubating
> > release:
> > > > > >  - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > > > > streaming,
> > > > > > but clearly mark it as a preview in the documentation.
> > > > > >  -* Java API Pojo improvements*: Code generation, key selection
> > > using a
> > > > > > string-expression:
> > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > >   - *Reworked Scala API*. Bring the Scala API in sync with the
> > latest
> > > > > > developments in the Java API:
> > > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > > >   -* Akka-based RPC service*:
> > > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > >   - *Kryo-based serialization*. This feature has been requested
> by
> > > many
> > > > > > users. Mostly because they wanted to use Collections inside
> POJOs:
> > > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > > >
> > > > > >
> > > > > > Opinions?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
Hi,

sorry, my previous message was confusing. I suggest to release a new Flink
version all 3 months.
BUT, the 0.7-incubating release is going to be feature freeze in 3 weeks
because 0.6-incubating was more about getting the release infra set up and
the apache rename out.
The last release that contained a lot of features was 0.5 and it was
released on May 29. So aiming for end of September for the next feature
release should be somehow in the schedule.


Robert


On Wed, Sep 3, 2014 at 11:47 AM, Ufuk Celebi <uc...@apache.org> wrote:

> Hey Robert,
>
> +1 to frequent regular frequent major releases (I guess you meant 3 weeks
> and not 3 months, right?).
>
> Ufuk
>
>
> On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> > I agree with Fabian that the list of features is a lot of work. I would
> > prefer to have frequent regular major releases (say 3 months schedule).
> > This way, users can quickly access the latest features and we don't force
> > them to use SNAPSHOT versions.
> > Also, releases draw attention to our project.
> >
> > I would suggest to do a feature freeze on September 24 (3 weeks from now)
> > and start the vote a few days afterwards. I assume that at least some of
> > the suggested 7 features of the release are ready.
> >
> > Robert
> >
> >
> > On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fh...@apache.org>
> wrote:
> >
> > > I agree that these should be features to add soon, but I'm a bit
> doubtful
> > > that we will have the next release in 5 weeks if we want to include all
> > of
> > > this.
> > >
> > > I think we should either have a feature- or time-oriented release plan.
> > > If we want to have a fixed release date, we could make a feature stop 1
> > > week (or so) in advance, include what's in until then and continue to
> > > release every n weeks.
> > > Or we make a list of what should be in the next release and try to
> give a
> > > reasonable time estimate for that (which didn't work out so well in the
> > > past.... ;-) )
> > >
> > > But having both, a close deadline and a long list of complex features
> > will
> > > not work out well, IMO.
> > >
> > > Just my 2 cents, Fabian
> > >
> > >
> > > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> > >
> > > > This is a nice list.
> > > >
> > > > I would like to add:
> > > >
> > > >  - Rework JobManager internals to support incremental program
> rollout &
> > > > execution
> > > >  - First parts of dynamic memory assignments
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <rmetzger@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > since we have our release infrastructure in place now, I would
> > suggest
> > > to
> > > > > aim for a 0.7-incubating release in the near future (say 3-5
> weeks).
> > > > > While 0.6-incubating was mainly about getting the release infra /
> > legal
> > > > > stuff sorted out, I think it would be nice to have a "feature"
> > release
> > > > > soon.
> > > > >
> > > > > The following new features would make a great 0.7-incubating
> release:
> > > > >  - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > > > streaming,
> > > > > but clearly mark it as a preview in the documentation.
> > > > >  -* Java API Pojo improvements*: Code generation, key selection
> > using a
> > > > > string-expression:
> https://issues.apache.org/jira/browse/FLINK-1032
> > > > >   - *Reworked Scala API*. Bring the Scala API in sync with the
> latest
> > > > > developments in the Java API:
> > > > > https://issues.apache.org/jira/browse/FLINK-641
> > > > >   -* Akka-based RPC service*:
> > > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > > >   - *Kryo-based serialization*. This feature has been requested by
> > many
> > > > > users. Mostly because they wanted to use Collections inside POJOs:
> > > > > https://issues.apache.org/jira/browse/FLINK-610
> > > > >
> > > > >
> > > > > Opinions?
> > > > >
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Ufuk Celebi <uc...@apache.org>.
Hey Robert,

+1 to frequent regular frequent major releases (I guess you meant 3 weeks
and not 3 months, right?).

Ufuk


On Tue, Sep 2, 2014 at 9:29 PM, Robert Metzger <rm...@apache.org> wrote:

> Hi,
> I agree with Fabian that the list of features is a lot of work. I would
> prefer to have frequent regular major releases (say 3 months schedule).
> This way, users can quickly access the latest features and we don't force
> them to use SNAPSHOT versions.
> Also, releases draw attention to our project.
>
> I would suggest to do a feature freeze on September 24 (3 weeks from now)
> and start the vote a few days afterwards. I assume that at least some of
> the suggested 7 features of the release are ready.
>
> Robert
>
>
> On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fh...@apache.org> wrote:
>
> > I agree that these should be features to add soon, but I'm a bit doubtful
> > that we will have the next release in 5 weeks if we want to include all
> of
> > this.
> >
> > I think we should either have a feature- or time-oriented release plan.
> > If we want to have a fixed release date, we could make a feature stop 1
> > week (or so) in advance, include what's in until then and continue to
> > release every n weeks.
> > Or we make a list of what should be in the next release and try to give a
> > reasonable time estimate for that (which didn't work out so well in the
> > past.... ;-) )
> >
> > But having both, a close deadline and a long list of complex features
> will
> > not work out well, IMO.
> >
> > Just my 2 cents, Fabian
> >
> >
> > 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
> >
> > > This is a nice list.
> > >
> > > I would like to add:
> > >
> > >  - Rework JobManager internals to support incremental program rollout &
> > > execution
> > >  - First parts of dynamic memory assignments
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <rm...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > since we have our release infrastructure in place now, I would
> suggest
> > to
> > > > aim for a 0.7-incubating release in the near future (say 3-5 weeks).
> > > > While 0.6-incubating was mainly about getting the release infra /
> legal
> > > > stuff sorted out, I think it would be nice to have a "feature"
> release
> > > > soon.
> > > >
> > > > The following new features would make a great 0.7-incubating release:
> > > >  - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > > streaming,
> > > > but clearly mark it as a preview in the documentation.
> > > >  -* Java API Pojo improvements*: Code generation, key selection
> using a
> > > > string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> > > >   - *Reworked Scala API*. Bring the Scala API in sync with the latest
> > > > developments in the Java API:
> > > > https://issues.apache.org/jira/browse/FLINK-641
> > > >   -* Akka-based RPC service*:
> > > > https://issues.apache.org/jira/browse/FLINK-1019
> > > >   - *Kryo-based serialization*. This feature has been requested by
> many
> > > > users. Mostly because they wanted to use Collections inside POJOs:
> > > > https://issues.apache.org/jira/browse/FLINK-610
> > > >
> > > >
> > > > Opinions?
> > > >
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Robert Metzger <rm...@apache.org>.
Hi,
I agree with Fabian that the list of features is a lot of work. I would
prefer to have frequent regular major releases (say 3 months schedule).
This way, users can quickly access the latest features and we don't force
them to use SNAPSHOT versions.
Also, releases draw attention to our project.

I would suggest to do a feature freeze on September 24 (3 weeks from now)
and start the vote a few days afterwards. I assume that at least some of
the suggested 7 features of the release are ready.

Robert


On Wed, Aug 27, 2014 at 9:20 PM, Fabian Hueske <fh...@apache.org> wrote:

> I agree that these should be features to add soon, but I'm a bit doubtful
> that we will have the next release in 5 weeks if we want to include all of
> this.
>
> I think we should either have a feature- or time-oriented release plan.
> If we want to have a fixed release date, we could make a feature stop 1
> week (or so) in advance, include what's in until then and continue to
> release every n weeks.
> Or we make a list of what should be in the next release and try to give a
> reasonable time estimate for that (which didn't work out so well in the
> past.... ;-) )
>
> But having both, a close deadline and a long list of complex features will
> not work out well, IMO.
>
> Just my 2 cents, Fabian
>
>
> 2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:
>
> > This is a nice list.
> >
> > I would like to add:
> >
> >  - Rework JobManager internals to support incremental program rollout &
> > execution
> >  - First parts of dynamic memory assignments
> >
> >
> >
> >
> >
> > On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > since we have our release infrastructure in place now, I would suggest
> to
> > > aim for a 0.7-incubating release in the near future (say 3-5 weeks).
> > > While 0.6-incubating was mainly about getting the release infra / legal
> > > stuff sorted out, I think it would be nice to have a "feature" release
> > > soon.
> > >
> > > The following new features would make a great 0.7-incubating release:
> > >  - *Flink Streaming* "Beta Preview". I would suggest to ship the
> > streaming,
> > > but clearly mark it as a preview in the documentation.
> > >  -* Java API Pojo improvements*: Code generation, key selection using a
> > > string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> > >   - *Reworked Scala API*. Bring the Scala API in sync with the latest
> > > developments in the Java API:
> > > https://issues.apache.org/jira/browse/FLINK-641
> > >   -* Akka-based RPC service*:
> > > https://issues.apache.org/jira/browse/FLINK-1019
> > >   - *Kryo-based serialization*. This feature has been requested by many
> > > users. Mostly because they wanted to use Collections inside POJOs:
> > > https://issues.apache.org/jira/browse/FLINK-610
> > >
> > >
> > > Opinions?
> > >
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Fabian Hueske <fh...@apache.org>.
I agree that these should be features to add soon, but I'm a bit doubtful
that we will have the next release in 5 weeks if we want to include all of
this.

I think we should either have a feature- or time-oriented release plan.
If we want to have a fixed release date, we could make a feature stop 1
week (or so) in advance, include what's in until then and continue to
release every n weeks.
Or we make a list of what should be in the next release and try to give a
reasonable time estimate for that (which didn't work out so well in the
past.... ;-) )

But having both, a close deadline and a long list of complex features will
not work out well, IMO.

Just my 2 cents, Fabian


2014-08-27 19:49 GMT+02:00 Stephan Ewen <se...@apache.org>:

> This is a nice list.
>
> I would like to add:
>
>  - Rework JobManager internals to support incremental program rollout &
> execution
>  - First parts of dynamic memory assignments
>
>
>
>
>
> On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > since we have our release infrastructure in place now, I would suggest to
> > aim for a 0.7-incubating release in the near future (say 3-5 weeks).
> > While 0.6-incubating was mainly about getting the release infra / legal
> > stuff sorted out, I think it would be nice to have a "feature" release
> > soon.
> >
> > The following new features would make a great 0.7-incubating release:
> >  - *Flink Streaming* "Beta Preview". I would suggest to ship the
> streaming,
> > but clearly mark it as a preview in the documentation.
> >  -* Java API Pojo improvements*: Code generation, key selection using a
> > string-expression: https://issues.apache.org/jira/browse/FLINK-1032
> >   - *Reworked Scala API*. Bring the Scala API in sync with the latest
> > developments in the Java API:
> > https://issues.apache.org/jira/browse/FLINK-641
> >   -* Akka-based RPC service*:
> > https://issues.apache.org/jira/browse/FLINK-1019
> >   - *Kryo-based serialization*. This feature has been requested by many
> > users. Mostly because they wanted to use Collections inside POJOs:
> > https://issues.apache.org/jira/browse/FLINK-610
> >
> >
> > Opinions?
> >
>

Re: Planning Flink release 0.7-incubating

Posted by Stephan Ewen <se...@apache.org>.
This is a nice list.

I would like to add:

 - Rework JobManager internals to support incremental program rollout &
execution
 - First parts of dynamic memory assignments





On Wed, Aug 27, 2014 at 7:21 PM, Robert Metzger <rm...@apache.org> wrote:

> Hi,
>
> since we have our release infrastructure in place now, I would suggest to
> aim for a 0.7-incubating release in the near future (say 3-5 weeks).
> While 0.6-incubating was mainly about getting the release infra / legal
> stuff sorted out, I think it would be nice to have a "feature" release
> soon.
>
> The following new features would make a great 0.7-incubating release:
>  - *Flink Streaming* "Beta Preview". I would suggest to ship the streaming,
> but clearly mark it as a preview in the documentation.
>  -* Java API Pojo improvements*: Code generation, key selection using a
> string-expression: https://issues.apache.org/jira/browse/FLINK-1032
>   - *Reworked Scala API*. Bring the Scala API in sync with the latest
> developments in the Java API:
> https://issues.apache.org/jira/browse/FLINK-641
>   -* Akka-based RPC service*:
> https://issues.apache.org/jira/browse/FLINK-1019
>   - *Kryo-based serialization*. This feature has been requested by many
> users. Mostly because they wanted to use Collections inside POJOs:
> https://issues.apache.org/jira/browse/FLINK-610
>
>
> Opinions?
>