You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Maximilian Michels <mx...@apache.org> on 2015/06/08 18:52:41 UTC
Testing Apache Flink 0.9.0-rc1
Hi everyone!
As previously discussed, the Flink developer community is very eager to get
out a new major release. Apache Flink 0.9.0 will contain lots of new
features and many bugfixes. This time, I'll try to coordinate the release
process. Feel free to correct me if I'm doing something wrong because I
don't no any better :)
To release a great version of Flink to the public, I'd like to ask everyone
to test the release candidate. Recently, Flink has received a lot of
attention. The expectations are quite high. Only through thorough testing
we will be able to satisfy all the Flink users out there.
Below is a list from the Wiki that we use to ensure the legal and
functional aspects of a release [1]. What I would like you to do is pick at
least one of the tasks, put your name as assignee in the link below, and
report back once you verified it. That way, I hope we can quickly and
thoroughly test the release candidate.
https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
Best,
Max
Git branch: release-0.9-rc1
Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
Maven artifacts:
https://repository.apache.org/content/repositories/orgapacheflink-1037/
PGP public key for verifying the signatures:
http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
Legal
====
L.1 Check if checksums and GPG files match the corresponding release files
L.2 Verify that the source archives do NOT contains any binaries
L.3 Check if the source release is building properly with Maven (including
license header check (default) and checkstyle). Also the tests should be
executed (mvn clean verify)
L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
source release.
L.5 All dependencies must be checked for their license and the license must
be ASL 2.0 compatible (http://www.apache.org/legal/resolved.html#category-x)
* The LICENSE and NOTICE files in the root directory refer to dependencies
in the source release, i.e., files in the git repository (such as fonts,
css, JavaScript, images)
* The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
the binary distribution and mention all of Flink's Maven dependencies as
well
L.6 Check that all POM files point to the same version (mostly relevant to
examine quickstart artifact files)
L.7 Read the README.md file
Functional
========
F.1 Run the start-local.sh/start-local-streaming.sh,
start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts and
verify that the processes come up
F.2 Examine the *.out files (should be empty) and the log files (should
contain no exceptions)
* Test for Linux, OS X, Windows (for Windows as far as possible, not all
scripts exist)
* Shutdown and verify there are no exceptions in the log output (after
shutdown)
* Check all start+submission scripts for paths with and without spaces
(./bin/* scripts are quite fragile for paths with spaces)
F.3 local mode (start-local.sh, see criteria below)
F.4 cluster mode (start-cluster.sh, see criteria below)
F.5 multi-node cluster (can simulate locally by starting two taskmanagers,
see criteria below)
Criteria for F.3 F.4 F.5
----------------------------
* Verify that the examples are running from both ./bin/flink and from the
web-based job submission tool
* flink-conf.yml should define more than one task slot
* Results of job are produced and correct
** Check also that the examples are running with the build-in data and
external sources.
* Examine the log output - no error messages should be encountered
** Web interface shows progress and finished job in history
F.6 Test on a cluster with HDFS.
* Check that a good amount of input splits is read locally (JobManager log
reveals local assignments)
F.7 Test against a Kafka installation
F.8 Test the ./bin/flink command line client
* Test "info" option, paste the JSON into the plan visualizer HTML file,
check that plan is rendered
* Test the parallelism flag (-p) to override the configured default
parallelism
F.9 Verify the plan visualizer with different browsers/operating systems
F.10 Verify that the quickstarts for scala and java are working with the
staging repository for both IntelliJ and Eclipse.
* In particular the dependencies of the quickstart project need to be set
correctly and the QS project needs to build from the staging repository
(replace the snapshot repo URL with the staging repo URL)
* The dependency tree of the QuickStart project must not contain any
dependencies we shade away upstream (guava, netty, ...)
F.11 Run examples on a YARN cluster
F.12 Run all examples from the IDE (Eclipse & IntelliJ)
F.13 Run an example with the RemoteEnvironment against a cluster started
from the shell script
F.14 Run manual Tests in "flink-tests" module.
* Marked with the @Ignore interface.
[1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
Sorry, it was already out. I was merely struggling with the Maven deploy
command because "tools/generate_specific_pom.sh" is not entirely compatible
with old versions of Perl or sed. The script was generating incorrect
pom.xml files..
On Sat, Jun 13, 2015 at 9:14 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> The new release candidate is not yet done? We have a very simple fix that
> allows the RowSerializer of the Table API to work with null-fields. I think
> we should include that. What do you think?
>
> On Fri, 12 Jun 2015 at 23:50 Ufuk Celebi <uc...@apache.org> wrote:
>
> > I'm with Till on this. Robert's position is valid as well. Again, there
> is
> > no core disagreement here. No one wants to add it to dist.
> >
> > On 12 Jun 2015, at 00:40, Ufuk Celebi <uce@apache.org
> > <javascript:_e(%7B%7D,'cvml','uce@apache.org');>> wrote:
> >
> >
> > On 11 Jun 2015, at 20:04, Fabian Hueske <fhueske@gmail.com
> > <javascript:_e(%7B%7D,'cvml','fhueske@gmail.com');>> wrote:
> >
> > How about the following issues?
> >
> > 1. The Hbase Hadoop Compat issue, Ufuk is working on
> >
> >
> > I was not able to reproduce this :( I ran HadoopInputFormats against
> > various sources and confirmed the results and everything was fine so far.
> >
> >
> > The issue has been resolved "Not a problem".
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
The new release candidate is not yet done? We have a very simple fix that
allows the RowSerializer of the Table API to work with null-fields. I think
we should include that. What do you think?
On Fri, 12 Jun 2015 at 23:50 Ufuk Celebi <uc...@apache.org> wrote:
> I'm with Till on this. Robert's position is valid as well. Again, there is
> no core disagreement here. No one wants to add it to dist.
>
> On 12 Jun 2015, at 00:40, Ufuk Celebi <uce@apache.org
> <javascript:_e(%7B%7D,'cvml','uce@apache.org');>> wrote:
>
>
> On 11 Jun 2015, at 20:04, Fabian Hueske <fhueske@gmail.com
> <javascript:_e(%7B%7D,'cvml','fhueske@gmail.com');>> wrote:
>
> How about the following issues?
>
> 1. The Hbase Hadoop Compat issue, Ufuk is working on
>
>
> I was not able to reproduce this :( I ran HadoopInputFormats against
> various sources and confirmed the results and everything was fine so far.
>
>
> The issue has been resolved "Not a problem".
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
I'm with Till on this. Robert's position is valid as well. Again, there is
no core disagreement here. No one wants to add it to dist.
On 12 Jun 2015, at 00:40, Ufuk Celebi <uce@apache.org
<javascript:_e(%7B%7D,'cvml','uce@apache.org');>> wrote:
On 11 Jun 2015, at 20:04, Fabian Hueske <fhueske@gmail.com
<javascript:_e(%7B%7D,'cvml','fhueske@gmail.com');>> wrote:
How about the following issues?
1. The Hbase Hadoop Compat issue, Ufuk is working on
I was not able to reproduce this :( I ran HadoopInputFormats against
various sources and confirmed the results and everything was fine so far.
The issue has been resolved "Not a problem".
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
I agree mostly with Robert. However, one could also argue that by not
including the libraries in the dist package, the user code jar will also be
blown up by the dependencies added by the library. This will slow down job
submission, because it has to be distributed on the cluster. Furthermore, I
wouldn't expect all our users to use the quickstarts archetypes or to set
up maven such that it builds a fat jar.
I think the best is to explicitly document how to use the libraries and
what to do in order to run it on the cluster.
On Jun 12, 2015 9:15 PM, "Robert Metzger" <rm...@apache.org> wrote:
> Regarding the discussion with including ML, Gelly, streaming connectors
> into "flink-dist".
> I'm strongly against adding those into our jar because they blow up the
> dependencies we are shipping by default.
>
> Also, the maven archetype sets up everything so that the dependencies are
> packaged into the usercode jar.
> I'd say most of the time users are using custom dependencies anyways
> (Guava), so they need to set this up properly.
>
> I would not start recommending our users putting their dependencies into
> the lib/ folder. Its much more convenient to let maven do the fat-jar
> packaging.
>
>
> On Fri, Jun 12, 2015 at 9:44 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I've finished the legal check of the source and binary distribution. The
> PR
> > with the LICENSE and NOTICE file updates can be found here [1].
> >
> > What I haven't done yet is addressing the issue with the shaded
> > dependencies. I think that we have to add to all jars which contain
> > dependencies as binary data a LICENSE/NOTICE file referencing the
> included
> > dependencies if they are not licensed under Apache-2.0 or contain a
> special
> > NOTICE portion.
> >
> > Cheers,
> > Till
> >
> > [1] https://github.com/apache/flink/pull/830
> >
> > On Fri, Jun 12, 2015 at 5:44 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >
> > > I almost finished creating the new release candidate. Then the maven
> > deploy
> > > command failed on me for the hadoop1 profile:
> > >
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] BUILD FAILURE
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Total time: 19:15.388s
> > > [INFO] Finished at: Fri Jun 12 15:25:50 UTC 2015
> > > [INFO] Final Memory: 126M/752M
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:check
> (validate)
> > on
> > > project flink-language-binding-generic: Failed during checkstyle
> executi
> > > on: Unable to find suppressions file at location:
> > > /tools/maven/suppressions.xml: Could not find resource
> > > '/tools/maven/suppressions.xml'. -> [Help 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> > -e
> > > switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ERROR] For more information about the errors and possible solutions,
> > > please read the following articles:
> > > [ERROR] [Help 1]
> > >
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> > > [ERROR]
> > > [ERROR] After correcting the problems, you can resume the build with
> the
> > > command
> > > [ERROR] mvn <goals> -rf :flink-language-binding-generic
> > >
> > > I need to look into this later. Unfortunately, I'm traveling this
> > weekend.
> > >
> > > On Fri, Jun 12, 2015 at 3:34 PM, Fabian Hueske <fh...@gmail.com>
> > wrote:
> > >
> > > > OK, guys. I merged and pushed the last outstanding commits to the
> > > > release-0.9 branch.
> > > > Good to go for a new candidate.
> > > >
> > > > 2015-06-12 14:30 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> > > >
> > > > > +1 Let's constitute the changes in a new release candidate.
> > > > >
> > > > > On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > +1 for b)
> > > > > >
> > > > > > I'm organizing + merging the commits that need to go the new
> > > candidate
> > > > > > right now. Will let you know, when I am done.
> > > > > >
> > > > > > 2015-06-12 14:03 GMT+02:00 Till Rohrmann <
> till.rohrmann@gmail.com
> > >:
> > > > > >
> > > > > > > I'm in favour of option b) as well.
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > Yes, the LICENSE files are definitely a release blocker.
> > > > > > > >
> > > > > > > > a) Either we wait with the RC until we have fixed the
> LICENSES,
> > > or
> > > > > > > >
> > > > > > > > b) Put out next RC to continue with testing and then update
> it
> > > with
> > > > > the
> > > > > > > > LICENSE [either we find something before the LICENSE update
> or
> > we
> > > > > only
> > > > > > > have
> > > > > > > > to review the LICENSE change]
> > > > > > > >
> > > > > > > > Since this is not a vote yet, it doesn't really matter, but
> I'm
> > > > > leaning
> > > > > > > > towards b).
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> > > > > > till.rohrmann@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What about the shaded jars?
> > > > > > > > >
> > > > > > > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <
> uce@apache.org
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > @Max: for the new RC. Can you make sure to set the
> > variables
> > > > > > > correctly
> > > > > > > > > > with regard to stable/snapshot versions in the docs?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Robert Metzger <rm...@apache.org>.
Regarding the discussion with including ML, Gelly, streaming connectors
into "flink-dist".
I'm strongly against adding those into our jar because they blow up the
dependencies we are shipping by default.
Also, the maven archetype sets up everything so that the dependencies are
packaged into the usercode jar.
I'd say most of the time users are using custom dependencies anyways
(Guava), so they need to set this up properly.
I would not start recommending our users putting their dependencies into
the lib/ folder. Its much more convenient to let maven do the fat-jar
packaging.
On Fri, Jun 12, 2015 at 9:44 AM, Till Rohrmann <tr...@apache.org> wrote:
> I've finished the legal check of the source and binary distribution. The PR
> with the LICENSE and NOTICE file updates can be found here [1].
>
> What I haven't done yet is addressing the issue with the shaded
> dependencies. I think that we have to add to all jars which contain
> dependencies as binary data a LICENSE/NOTICE file referencing the included
> dependencies if they are not licensed under Apache-2.0 or contain a special
> NOTICE portion.
>
> Cheers,
> Till
>
> [1] https://github.com/apache/flink/pull/830
>
> On Fri, Jun 12, 2015 at 5:44 PM Maximilian Michels <mx...@apache.org> wrote:
>
> > I almost finished creating the new release candidate. Then the maven
> deploy
> > command failed on me for the hadoop1 profile:
> >
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time: 19:15.388s
> > [INFO] Finished at: Fri Jun 12 15:25:50 UTC 2015
> > [INFO] Final Memory: 126M/752M
> > [INFO]
> > ------------------------------------------------------------------------
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:check (validate)
> on
> > project flink-language-binding-generic: Failed during checkstyle executi
> > on: Unable to find suppressions file at location:
> > /tools/maven/suppressions.xml: Could not find resource
> > '/tools/maven/suppressions.xml'. -> [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e
> > switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions,
> > please read the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> > [ERROR]
> > [ERROR] After correcting the problems, you can resume the build with the
> > command
> > [ERROR] mvn <goals> -rf :flink-language-binding-generic
> >
> > I need to look into this later. Unfortunately, I'm traveling this
> weekend.
> >
> > On Fri, Jun 12, 2015 at 3:34 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> > > OK, guys. I merged and pushed the last outstanding commits to the
> > > release-0.9 branch.
> > > Good to go for a new candidate.
> > >
> > > 2015-06-12 14:30 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> > >
> > > > +1 Let's constitute the changes in a new release candidate.
> > > >
> > > > On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com>
> > > wrote:
> > > >
> > > > > +1 for b)
> > > > >
> > > > > I'm organizing + merging the commits that need to go the new
> > candidate
> > > > > right now. Will let you know, when I am done.
> > > > >
> > > > > 2015-06-12 14:03 GMT+02:00 Till Rohrmann <till.rohrmann@gmail.com
> >:
> > > > >
> > > > > > I'm in favour of option b) as well.
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Yes, the LICENSE files are definitely a release blocker.
> > > > > > >
> > > > > > > a) Either we wait with the RC until we have fixed the LICENSES,
> > or
> > > > > > >
> > > > > > > b) Put out next RC to continue with testing and then update it
> > with
> > > > the
> > > > > > > LICENSE [either we find something before the LICENSE update or
> we
> > > > only
> > > > > > have
> > > > > > > to review the LICENSE change]
> > > > > > >
> > > > > > > Since this is not a vote yet, it doesn't really matter, but I'm
> > > > leaning
> > > > > > > towards b).
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> > > > > till.rohrmann@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What about the shaded jars?
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uce@apache.org
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > @Max: for the new RC. Can you make sure to set the
> variables
> > > > > > correctly
> > > > > > > > > with regard to stable/snapshot versions in the docs?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
I've finished the legal check of the source and binary distribution. The PR
with the LICENSE and NOTICE file updates can be found here [1].
What I haven't done yet is addressing the issue with the shaded
dependencies. I think that we have to add to all jars which contain
dependencies as binary data a LICENSE/NOTICE file referencing the included
dependencies if they are not licensed under Apache-2.0 or contain a special
NOTICE portion.
Cheers,
Till
[1] https://github.com/apache/flink/pull/830
On Fri, Jun 12, 2015 at 5:44 PM Maximilian Michels <mx...@apache.org> wrote:
> I almost finished creating the new release candidate. Then the maven deploy
> command failed on me for the hadoop1 profile:
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 19:15.388s
> [INFO] Finished at: Fri Jun 12 15:25:50 UTC 2015
> [INFO] Final Memory: 126M/752M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:check (validate) on
> project flink-language-binding-generic: Failed during checkstyle executi
> on: Unable to find suppressions file at location:
> /tools/maven/suppressions.xml: Could not find resource
> '/tools/maven/suppressions.xml'. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn <goals> -rf :flink-language-binding-generic
>
> I need to look into this later. Unfortunately, I'm traveling this weekend.
>
> On Fri, Jun 12, 2015 at 3:34 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > OK, guys. I merged and pushed the last outstanding commits to the
> > release-0.9 branch.
> > Good to go for a new candidate.
> >
> > 2015-06-12 14:30 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> >
> > > +1 Let's constitute the changes in a new release candidate.
> > >
> > > On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com>
> > wrote:
> > >
> > > > +1 for b)
> > > >
> > > > I'm organizing + merging the commits that need to go the new
> candidate
> > > > right now. Will let you know, when I am done.
> > > >
> > > > 2015-06-12 14:03 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> > > >
> > > > > I'm in favour of option b) as well.
> > > > >
> > > > > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org>
> wrote:
> > > > >
> > > > > > Yes, the LICENSE files are definitely a release blocker.
> > > > > >
> > > > > > a) Either we wait with the RC until we have fixed the LICENSES,
> or
> > > > > >
> > > > > > b) Put out next RC to continue with testing and then update it
> with
> > > the
> > > > > > LICENSE [either we find something before the LICENSE update or we
> > > only
> > > > > have
> > > > > > to review the LICENSE change]
> > > > > >
> > > > > > Since this is not a vote yet, it doesn't really matter, but I'm
> > > leaning
> > > > > > towards b).
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> > > > till.rohrmann@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > What about the shaded jars?
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > @Max: for the new RC. Can you make sure to set the variables
> > > > > correctly
> > > > > > > > with regard to stable/snapshot versions in the docs?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
I almost finished creating the new release candidate. Then the maven deploy
command failed on me for the hadoop1 profile:
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 19:15.388s
[INFO] Finished at: Fri Jun 12 15:25:50 UTC 2015
[INFO] Final Memory: 126M/752M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:check (validate) on
project flink-language-binding-generic: Failed during checkstyle executi
on: Unable to find suppressions file at location:
/tools/maven/suppressions.xml: Could not find resource
'/tools/maven/suppressions.xml'. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn <goals> -rf :flink-language-binding-generic
I need to look into this later. Unfortunately, I'm traveling this weekend.
On Fri, Jun 12, 2015 at 3:34 PM, Fabian Hueske <fh...@gmail.com> wrote:
> OK, guys. I merged and pushed the last outstanding commits to the
> release-0.9 branch.
> Good to go for a new candidate.
>
> 2015-06-12 14:30 GMT+02:00 Maximilian Michels <mx...@apache.org>:
>
> > +1 Let's constitute the changes in a new release candidate.
> >
> > On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> > > +1 for b)
> > >
> > > I'm organizing + merging the commits that need to go the new candidate
> > > right now. Will let you know, when I am done.
> > >
> > > 2015-06-12 14:03 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> > >
> > > > I'm in favour of option b) as well.
> > > >
> > > > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > > Yes, the LICENSE files are definitely a release blocker.
> > > > >
> > > > > a) Either we wait with the RC until we have fixed the LICENSES, or
> > > > >
> > > > > b) Put out next RC to continue with testing and then update it with
> > the
> > > > > LICENSE [either we find something before the LICENSE update or we
> > only
> > > > have
> > > > > to review the LICENSE change]
> > > > >
> > > > > Since this is not a vote yet, it doesn't really matter, but I'm
> > leaning
> > > > > towards b).
> > > > >
> > > > >
> > > > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> > > till.rohrmann@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > What about the shaded jars?
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org>
> > wrote:
> > > > > >
> > > > > > > @Max: for the new RC. Can you make sure to set the variables
> > > > correctly
> > > > > > > with regard to stable/snapshot versions in the docs?
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
OK, guys. I merged and pushed the last outstanding commits to the
release-0.9 branch.
Good to go for a new candidate.
2015-06-12 14:30 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> +1 Let's constitute the changes in a new release candidate.
>
> On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > +1 for b)
> >
> > I'm organizing + merging the commits that need to go the new candidate
> > right now. Will let you know, when I am done.
> >
> > 2015-06-12 14:03 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> >
> > > I'm in favour of option b) as well.
> > >
> > > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > Yes, the LICENSE files are definitely a release blocker.
> > > >
> > > > a) Either we wait with the RC until we have fixed the LICENSES, or
> > > >
> > > > b) Put out next RC to continue with testing and then update it with
> the
> > > > LICENSE [either we find something before the LICENSE update or we
> only
> > > have
> > > > to review the LICENSE change]
> > > >
> > > > Since this is not a vote yet, it doesn't really matter, but I'm
> leaning
> > > > towards b).
> > > >
> > > >
> > > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> > till.rohrmann@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > What about the shaded jars?
> > > > >
> > > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org>
> wrote:
> > > > >
> > > > > > @Max: for the new RC. Can you make sure to set the variables
> > > correctly
> > > > > > with regard to stable/snapshot versions in the docs?
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
+1 Let's constitute the changes in a new release candidate.
On Fri, Jun 12, 2015 at 2:06 PM, Fabian Hueske <fh...@gmail.com> wrote:
> +1 for b)
>
> I'm organizing + merging the commits that need to go the new candidate
> right now. Will let you know, when I am done.
>
> 2015-06-12 14:03 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
>
> > I'm in favour of option b) as well.
> >
> > On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Yes, the LICENSE files are definitely a release blocker.
> > >
> > > a) Either we wait with the RC until we have fixed the LICENSES, or
> > >
> > > b) Put out next RC to continue with testing and then update it with the
> > > LICENSE [either we find something before the LICENSE update or we only
> > have
> > > to review the LICENSE change]
> > >
> > > Since this is not a vote yet, it doesn't really matter, but I'm leaning
> > > towards b).
> > >
> > >
> > > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <
> till.rohrmann@gmail.com
> > >
> > > wrote:
> > >
> > > > What about the shaded jars?
> > > >
> > > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > > @Max: for the new RC. Can you make sure to set the variables
> > correctly
> > > > > with regard to stable/snapshot versions in the docs?
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
+1 for b)
I'm organizing + merging the commits that need to go the new candidate
right now. Will let you know, when I am done.
2015-06-12 14:03 GMT+02:00 Till Rohrmann <ti...@gmail.com>:
> I'm in favour of option b) as well.
>
> On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org> wrote:
>
> > Yes, the LICENSE files are definitely a release blocker.
> >
> > a) Either we wait with the RC until we have fixed the LICENSES, or
> >
> > b) Put out next RC to continue with testing and then update it with the
> > LICENSE [either we find something before the LICENSE update or we only
> have
> > to review the LICENSE change]
> >
> > Since this is not a vote yet, it doesn't really matter, but I'm leaning
> > towards b).
> >
> >
> > On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <till.rohrmann@gmail.com
> >
> > wrote:
> >
> > > What about the shaded jars?
> > >
> > > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > @Max: for the new RC. Can you make sure to set the variables
> correctly
> > > > with regard to stable/snapshot versions in the docs?
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <ti...@gmail.com>.
I'm in favour of option b) as well.
On Fri, Jun 12, 2015 at 12:05 PM Ufuk Celebi <uc...@apache.org> wrote:
> Yes, the LICENSE files are definitely a release blocker.
>
> a) Either we wait with the RC until we have fixed the LICENSES, or
>
> b) Put out next RC to continue with testing and then update it with the
> LICENSE [either we find something before the LICENSE update or we only have
> to review the LICENSE change]
>
> Since this is not a vote yet, it doesn't really matter, but I'm leaning
> towards b).
>
>
> On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <ti...@gmail.com>
> wrote:
>
> > What about the shaded jars?
> >
> > On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > @Max: for the new RC. Can you make sure to set the variables correctly
> > > with regard to stable/snapshot versions in the docs?
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
Yes, the LICENSE files are definitely a release blocker.
a) Either we wait with the RC until we have fixed the LICENSES, or
b) Put out next RC to continue with testing and then update it with the
LICENSE [either we find something before the LICENSE update or we only have
to review the LICENSE change]
Since this is not a vote yet, it doesn't really matter, but I'm leaning
towards b).
On Fri, Jun 12, 2015 at 11:43 AM, Till Rohrmann <ti...@gmail.com>
wrote:
> What about the shaded jars?
>
> On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org> wrote:
>
> > @Max: for the new RC. Can you make sure to set the variables correctly
> > with regard to stable/snapshot versions in the docs?
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <ti...@gmail.com>.
What about the shaded jars?
On Fri, Jun 12, 2015 at 11:32 AM Ufuk Celebi <uc...@apache.org> wrote:
> @Max: for the new RC. Can you make sure to set the variables correctly
> with regard to stable/snapshot versions in the docs?
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
@Max: for the new RC. Can you make sure to set the variables correctly with regard to stable/snapshot versions in the docs?
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
After thinking about it a bit more, I think that's fine.
+1 to document and keep it as it is.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
Adding one more thing to the list:
The code contains a misplaced class (mea culpa) in flink-java,
org.apache.flink.api.java.SortPartitionOperator which is API facing and
should be moved to the operators package. If we do that after the release,
it will break binary compatibility. I created FLINK-2196 and will open a PR
soon.
If nobody objects, I'll merge it into the 0.9 release branch as well.
2015-06-10 11:02 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> I'm not against including the feature but I'd like to discuss it first. I
> believe that only very carefully selected commits should be added to
> release-0.9. If that feature happens to be tested extensively and is very
> important for user satisfactory then we might include it.
>
> On Wed, Jun 10, 2015 at 10:59 AM, F. Beligianni <fa...@gmail.com>
> wrote:
>
> > I agree with Gyula regarding the iteration partitioning.
> > I have also been using this feature for developing machine learning
> > algorithms. And I think SAMOA also needs this feature.
> >
> > Faye
> >
> > 2015-06-10 10:54 GMT+02:00 Gyula Fóra <gy...@gmail.com>:
> >
> > > This feature needs to be included in the release, it has been tested
> and
> > > used extensively. And many applciations depend on it.
> > >
> > > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10.,
> > > Sze,
> > > 10:47):
> > >
> > > > With all the issues discovered, it looks like we'll have another
> > release
> > > > candidate. Right now, we have discovered the following problems:
> > > >
> > > > 1 YARN ITCase fails [fixed via 2eb5cfe]
> > > > 2 No Jar for SessionWindowing example [fixed in #809]
> > > > 3 Wrong description of the input format for the graph examples (eg.
> > > > ConnectedComponents) [fixed in #809]
> > > > 4 TaskManagerFailsWithSlotSharingITCase fails
> > > > 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
> > > > 6 Submitting KMeans example to Web Submission Client does not work on
> > > > Firefox.
> > > > 7 Zooming is buggy in Web Submission Client (Firefox)
> > > >
> > > > Do we have someone familiar with the web interface who could take a
> > look
> > > at
> > > > the Firefox issues?
> > > >
> > > > One more important thing: The release-0.9 branch should only be used
> > for
> > > > bug fixes or prior discussed feature changes. Adding new features
> > defies
> > > > the purpose of carefully testing in advance and can have
> unforeseeable
> > > > consequences. In particular, I'm referring to #810 pull request:
> > > > https://github.com/apache/flink/pull/810
> > > >
> > > > IMHO, this one shouldn't have been cherry-picked onto the release-0.9
> > > > branch. I would like to remove it from there if no objections are
> > raised.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c947f8ce
> > > >
> > > > On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek <
> aljoscha@apache.org
> > >
> > > > wrote:
> > > >
> > > > > This doesn't look good, yes.
> > > > >
> > > > > On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org>
> wrote:
> > > > >
> > > > > > While looking into FLINK-2188 (HBase input) I've discovered that
> > > Hadoop
> > > > > > input formats implementing Configurable (like
> > > > mapreduce.TableInputFormat)
> > > > > > don't have the Hadoop configuration set via
> setConf(Configuration).
> > > > > >
> > > > > > I have a small fix for this, which I have to clean up. First, I
> > > wanted
> > > > to
> > > > > > check what you think about this issue wrt the release.
> Personally,
> > I
> > > > > think
> > > > > > this is a release blocker, because it essentially means that no
> > > Hadoop
> > > > > > input format, which relies on the Configuration instance to be
> set
> > > this
> > > > > way
> > > > > > will work (this is to some extent a bug of the respective input
> > > > formats)
> > > > > –
> > > > > > most notably the HBase TableInputFormat.
> > > > > >
> > > > > > – Ufuk
> > > > > >
> > > > > > On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com>
> > wrote:
> > > > > >
> > > > > > > I attached jps and jstack log about hanging
> > > > > > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Chiwan Park
> > > > > > >
> > > > > > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <
> > > aljoscha@apache.org
> > > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> I discovered something that might be a feature, rather than a
> > bug.
> > > > > When
> > > > > > you
> > > > > > >> submit an example using the web client without giving
> parameters
> > > the
> > > > > > >> program fails with this:
> > > > > > >>
> > > > > > >> org.apache.flink.client.program.ProgramInvocationException:
> The
> > > main
> > > > > > method
> > > > > > >> caused an error.
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > > > > > >>
> > > > > > >> at org.apache.flink.client.program.Client.run(Client.java:315)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> > > > > > >>
> > > > > > >> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> > > > > > >>
> > > > > > >> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> > > > > > >>
> > > > > > >> at
> > > > > >
> > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > > > > > >>
> > > > > > >> at
> > > > > >
> > > >
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > > > > > >>
> > > > > > >> at
> > > > > >
> > > >
> > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > > > > > >>
> > > > > > >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > > > > > >>
> > > > > > >> at
> > > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> > > > > > >>
> > > > > > >> at
> > > > >
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > > > > > >>
> > > > > > >> at
> > > > > >
> > > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > > > > > >>
> > > > > > >> at java.lang.Thread.run(Thread.java:745)
> > > > > > >>
> > > > > > >> Caused by: java.lang.NullPointerException
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> > > > > > >>
> > > > > > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> > > > > > >>
> > > > > > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> > > > > > >>
> > > > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > >>
> > > > > > >> at java.lang.reflect.Method.invoke(Method.java:497)
> > > > > > >>
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > > > > > >>
> > > > > > >> ... 24 more
> > > > > > >>
> > > > > > >>
> > > > > > >> This also only occurs when you uncheck the "suspend execution
> > > while
> > > > > > showing
> > > > > > >> plan".
> > > > > > >>
> > > > > > >> I think this arises because the new print() uses collect()
> which
> > > > tries
> > > > > > to
> > > > > > >> get the job execution result. I guess the result is Null since
> > the
> > > > job
> > > > > > is
> > > > > > >> submitted asynchronously when the checkbox is unchecked.
> > > > > > >>
> > > > > > >>
> > > > > > >> Other than that, the new print() is pretty sweet when you run
> > the
> > > > > > builtin
> > > > > > >> examples from the CLI. You get all the state changes and also
> > the
> > > > > > result,
> > > > > > >> even when running in cluster mode on several task managers. :D
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <
> > > > aljoscha@apache.org
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> I discovered another problem:
> > > > > > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure
> > > > cleaner
> > > > > > >>> cannot be disabled in part of the Streaming Java API and all
> of
> > > the
> > > > > > >>> Streaming Scala API. I think this is a release blocker (in
> > > addition
> > > > > > >>> the the other bugs found so far.)
> > > > > > >>>
> > > > > > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <
> > > > > aljoscha@apache.org>
> > > > > > >>> wrote:
> > > > > > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was
> > > > > comparing
> > > > > > >>>> the hostname to a hostname in some yarn config. In one case
> it
> > > was
> > > > > > >>>> capitalised, in the other case it wasn't.
> > > > > > >>>>
> > > > > > >>>> Pushing fix to master and release-0.9 branch.
> > > > > > >>>>
> > > > > > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <
> > > > > sachingoel0101@gmail.com
> > > > > > >
> > > > > > >>> wrote:
> > > > > > >>>>> A re-ran lead to reproducibility of 11 failures again.
> > > > > > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a
> > > > > time-out
> > > > > > but
> > > > > > >>>>> managed to succeed in a re-run. Here is the log output
> again:
> > > > > > >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> > > > > > >>>>>
> > > > > > >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> > > > > > >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> > > > > > >>>>>
> > > > > > >>>>> Regards
> > > > > > >>>>> Sachin Goel
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <
> uce@apache.org>
> > > > > wrote:
> > > > > > >>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <
> > > sachingoel0101@gmail.com
> > > > >
> > > > > > >>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> On my local machine, several flink runtime tests are
> > failing
> > > on
> > > > > > "mvn
> > > > > > >>>>>> clean
> > > > > > >>>>>>> verify". Here is the log output:
> > > > > > >>> http://pastebin.com/raw.php?i=VWbx2ppf
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for reporting this. Have you tried it multiple
> times?
> > > Is
> > > > it
> > > > > > >>> failing
> > > > > > >>>>>> reproducibly with the same tests? What's your setup?
> > > > > > >>>>>>
> > > > > > >>>>>> – Ufuk
> > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
I'm not against including the feature but I'd like to discuss it first. I
believe that only very carefully selected commits should be added to
release-0.9. If that feature happens to be tested extensively and is very
important for user satisfactory then we might include it.
On Wed, Jun 10, 2015 at 10:59 AM, F. Beligianni <fa...@gmail.com>
wrote:
> I agree with Gyula regarding the iteration partitioning.
> I have also been using this feature for developing machine learning
> algorithms. And I think SAMOA also needs this feature.
>
> Faye
>
> 2015-06-10 10:54 GMT+02:00 Gyula Fóra <gy...@gmail.com>:
>
> > This feature needs to be included in the release, it has been tested and
> > used extensively. And many applciations depend on it.
> >
> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10.,
> > Sze,
> > 10:47):
> >
> > > With all the issues discovered, it looks like we'll have another
> release
> > > candidate. Right now, we have discovered the following problems:
> > >
> > > 1 YARN ITCase fails [fixed via 2eb5cfe]
> > > 2 No Jar for SessionWindowing example [fixed in #809]
> > > 3 Wrong description of the input format for the graph examples (eg.
> > > ConnectedComponents) [fixed in #809]
> > > 4 TaskManagerFailsWithSlotSharingITCase fails
> > > 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
> > > 6 Submitting KMeans example to Web Submission Client does not work on
> > > Firefox.
> > > 7 Zooming is buggy in Web Submission Client (Firefox)
> > >
> > > Do we have someone familiar with the web interface who could take a
> look
> > at
> > > the Firefox issues?
> > >
> > > One more important thing: The release-0.9 branch should only be used
> for
> > > bug fixes or prior discussed feature changes. Adding new features
> defies
> > > the purpose of carefully testing in advance and can have unforeseeable
> > > consequences. In particular, I'm referring to #810 pull request:
> > > https://github.com/apache/flink/pull/810
> > >
> > > IMHO, this one shouldn't have been cherry-picked onto the release-0.9
> > > branch. I would like to remove it from there if no objections are
> raised.
> > >
> > >
> > >
> >
> https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c947f8ce
> > >
> > > On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > >
> > > > This doesn't look good, yes.
> > > >
> > > > On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > > While looking into FLINK-2188 (HBase input) I've discovered that
> > Hadoop
> > > > > input formats implementing Configurable (like
> > > mapreduce.TableInputFormat)
> > > > > don't have the Hadoop configuration set via setConf(Configuration).
> > > > >
> > > > > I have a small fix for this, which I have to clean up. First, I
> > wanted
> > > to
> > > > > check what you think about this issue wrt the release. Personally,
> I
> > > > think
> > > > > this is a release blocker, because it essentially means that no
> > Hadoop
> > > > > input format, which relies on the Configuration instance to be set
> > this
> > > > way
> > > > > will work (this is to some extent a bug of the respective input
> > > formats)
> > > > –
> > > > > most notably the HBase TableInputFormat.
> > > > >
> > > > > – Ufuk
> > > > >
> > > > > On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com>
> wrote:
> > > > >
> > > > > > I attached jps and jstack log about hanging
> > > > > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> > > > > >
> > > > > > Regards,
> > > > > > Chiwan Park
> > > > > >
> > > > > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <
> > aljoscha@apache.org
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> I discovered something that might be a feature, rather than a
> bug.
> > > > When
> > > > > you
> > > > > >> submit an example using the web client without giving parameters
> > the
> > > > > >> program fails with this:
> > > > > >>
> > > > > >> org.apache.flink.client.program.ProgramInvocationException: The
> > main
> > > > > method
> > > > > >> caused an error.
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > > > > >>
> > > > > >> at org.apache.flink.client.program.Client.run(Client.java:315)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> > > > > >>
> > > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> > > > > >>
> > > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> > > > > >>
> > > > > >> at
> > > > >
> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > > > > >>
> > > > > >> at
> > > > >
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > > > > >>
> > > > > >> at
> > > > >
> > >
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > > > > >>
> > > > > >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > > > > >>
> > > > > >> at
> > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> > > > > >>
> > > > > >> at
> > > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > > > > >>
> > > > > >> at
> > > > >
> > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > > > > >>
> > > > > >> at java.lang.Thread.run(Thread.java:745)
> > > > > >>
> > > > > >> Caused by: java.lang.NullPointerException
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> > > > > >>
> > > > > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> > > > > >>
> > > > > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> > > > > >>
> > > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > >>
> > > > > >> at java.lang.reflect.Method.invoke(Method.java:497)
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > > > > >>
> > > > > >> ... 24 more
> > > > > >>
> > > > > >>
> > > > > >> This also only occurs when you uncheck the "suspend execution
> > while
> > > > > showing
> > > > > >> plan".
> > > > > >>
> > > > > >> I think this arises because the new print() uses collect() which
> > > tries
> > > > > to
> > > > > >> get the job execution result. I guess the result is Null since
> the
> > > job
> > > > > is
> > > > > >> submitted asynchronously when the checkbox is unchecked.
> > > > > >>
> > > > > >>
> > > > > >> Other than that, the new print() is pretty sweet when you run
> the
> > > > > builtin
> > > > > >> examples from the CLI. You get all the state changes and also
> the
> > > > > result,
> > > > > >> even when running in cluster mode on several task managers. :D
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <
> > > aljoscha@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I discovered another problem:
> > > > > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure
> > > cleaner
> > > > > >>> cannot be disabled in part of the Streaming Java API and all of
> > the
> > > > > >>> Streaming Scala API. I think this is a release blocker (in
> > addition
> > > > > >>> the the other bugs found so far.)
> > > > > >>>
> > > > > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <
> > > > aljoscha@apache.org>
> > > > > >>> wrote:
> > > > > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was
> > > > comparing
> > > > > >>>> the hostname to a hostname in some yarn config. In one case it
> > was
> > > > > >>>> capitalised, in the other case it wasn't.
> > > > > >>>>
> > > > > >>>> Pushing fix to master and release-0.9 branch.
> > > > > >>>>
> > > > > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <
> > > > sachingoel0101@gmail.com
> > > > > >
> > > > > >>> wrote:
> > > > > >>>>> A re-ran lead to reproducibility of 11 failures again.
> > > > > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a
> > > > time-out
> > > > > but
> > > > > >>>>> managed to succeed in a re-run. Here is the log output again:
> > > > > >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> > > > > >>>>>
> > > > > >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> > > > > >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> > > > > >>>>>
> > > > > >>>>> Regards
> > > > > >>>>> Sachin Goel
> > > > > >>>>>
> > > > > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org>
> > > > wrote:
> > > > > >>>>>
> > > > > >>>>>>
> > > > > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <
> > sachingoel0101@gmail.com
> > > >
> > > > > >>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> On my local machine, several flink runtime tests are
> failing
> > on
> > > > > "mvn
> > > > > >>>>>> clean
> > > > > >>>>>>> verify". Here is the log output:
> > > > > >>> http://pastebin.com/raw.php?i=VWbx2ppf
> > > > > >>>>>>
> > > > > >>>>>> Thanks for reporting this. Have you tried it multiple times?
> > Is
> > > it
> > > > > >>> failing
> > > > > >>>>>> reproducibly with the same tests? What's your setup?
> > > > > >>>>>>
> > > > > >>>>>> – Ufuk
> > > > > >>>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by "F. Beligianni" <fa...@gmail.com>.
I agree with Gyula regarding the iteration partitioning.
I have also been using this feature for developing machine learning
algorithms. And I think SAMOA also needs this feature.
Faye
2015-06-10 10:54 GMT+02:00 Gyula Fóra <gy...@gmail.com>:
> This feature needs to be included in the release, it has been tested and
> used extensively. And many applciations depend on it.
>
> Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10.,
> Sze,
> 10:47):
>
> > With all the issues discovered, it looks like we'll have another release
> > candidate. Right now, we have discovered the following problems:
> >
> > 1 YARN ITCase fails [fixed via 2eb5cfe]
> > 2 No Jar for SessionWindowing example [fixed in #809]
> > 3 Wrong description of the input format for the graph examples (eg.
> > ConnectedComponents) [fixed in #809]
> > 4 TaskManagerFailsWithSlotSharingITCase fails
> > 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
> > 6 Submitting KMeans example to Web Submission Client does not work on
> > Firefox.
> > 7 Zooming is buggy in Web Submission Client (Firefox)
> >
> > Do we have someone familiar with the web interface who could take a look
> at
> > the Firefox issues?
> >
> > One more important thing: The release-0.9 branch should only be used for
> > bug fixes or prior discussed feature changes. Adding new features defies
> > the purpose of carefully testing in advance and can have unforeseeable
> > consequences. In particular, I'm referring to #810 pull request:
> > https://github.com/apache/flink/pull/810
> >
> > IMHO, this one shouldn't have been cherry-picked onto the release-0.9
> > branch. I would like to remove it from there if no objections are raised.
> >
> >
> >
> https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c947f8ce
> >
> > On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > This doesn't look good, yes.
> > >
> > > On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > While looking into FLINK-2188 (HBase input) I've discovered that
> Hadoop
> > > > input formats implementing Configurable (like
> > mapreduce.TableInputFormat)
> > > > don't have the Hadoop configuration set via setConf(Configuration).
> > > >
> > > > I have a small fix for this, which I have to clean up. First, I
> wanted
> > to
> > > > check what you think about this issue wrt the release. Personally, I
> > > think
> > > > this is a release blocker, because it essentially means that no
> Hadoop
> > > > input format, which relies on the Configuration instance to be set
> this
> > > way
> > > > will work (this is to some extent a bug of the respective input
> > formats)
> > > –
> > > > most notably the HBase TableInputFormat.
> > > >
> > > > – Ufuk
> > > >
> > > > On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com> wrote:
> > > >
> > > > > I attached jps and jstack log about hanging
> > > > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> > > > >
> > > > > Regards,
> > > > > Chiwan Park
> > > > >
> > > > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <
> aljoscha@apache.org
> > >
> > > > wrote:
> > > > >>
> > > > >> I discovered something that might be a feature, rather than a bug.
> > > When
> > > > you
> > > > >> submit an example using the web client without giving parameters
> the
> > > > >> program fails with this:
> > > > >>
> > > > >> org.apache.flink.client.program.ProgramInvocationException: The
> main
> > > > method
> > > > >> caused an error.
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > > > >>
> > > > >> at org.apache.flink.client.program.Client.run(Client.java:315)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> > > > >>
> > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> > > > >>
> > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> > > > >>
> > > > >> at
> > > >
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > > > >>
> > > > >> at
> > > >
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > > > >>
> > > > >> at
> > > >
> > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > > > >>
> > > > >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > > > >>
> > > > >> at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> > > > >>
> > > > >> at
> > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > > > >>
> > > > >> at
> > > >
> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > > > >>
> > > > >> at java.lang.Thread.run(Thread.java:745)
> > > > >>
> > > > >> Caused by: java.lang.NullPointerException
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> > > > >>
> > > > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> > > > >>
> > > > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> > > > >>
> > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > >>
> > > > >> at java.lang.reflect.Method.invoke(Method.java:497)
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > > > >>
> > > > >> ... 24 more
> > > > >>
> > > > >>
> > > > >> This also only occurs when you uncheck the "suspend execution
> while
> > > > showing
> > > > >> plan".
> > > > >>
> > > > >> I think this arises because the new print() uses collect() which
> > tries
> > > > to
> > > > >> get the job execution result. I guess the result is Null since the
> > job
> > > > is
> > > > >> submitted asynchronously when the checkbox is unchecked.
> > > > >>
> > > > >>
> > > > >> Other than that, the new print() is pretty sweet when you run the
> > > > builtin
> > > > >> examples from the CLI. You get all the state changes and also the
> > > > result,
> > > > >> even when running in cluster mode on several task managers. :D
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <
> > aljoscha@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> I discovered another problem:
> > > > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure
> > cleaner
> > > > >>> cannot be disabled in part of the Streaming Java API and all of
> the
> > > > >>> Streaming Scala API. I think this is a release blocker (in
> addition
> > > > >>> the the other bugs found so far.)
> > > > >>>
> > > > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <
> > > aljoscha@apache.org>
> > > > >>> wrote:
> > > > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was
> > > comparing
> > > > >>>> the hostname to a hostname in some yarn config. In one case it
> was
> > > > >>>> capitalised, in the other case it wasn't.
> > > > >>>>
> > > > >>>> Pushing fix to master and release-0.9 branch.
> > > > >>>>
> > > > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <
> > > sachingoel0101@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>>> A re-ran lead to reproducibility of 11 failures again.
> > > > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a
> > > time-out
> > > > but
> > > > >>>>> managed to succeed in a re-run. Here is the log output again:
> > > > >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> > > > >>>>>
> > > > >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> > > > >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> > > > >>>>>
> > > > >>>>> Regards
> > > > >>>>> Sachin Goel
> > > > >>>>>
> > > > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org>
> > > wrote:
> > > > >>>>>
> > > > >>>>>>
> > > > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <
> sachingoel0101@gmail.com
> > >
> > > > >>> wrote:
> > > > >>>>>>
> > > > >>>>>>> On my local machine, several flink runtime tests are failing
> on
> > > > "mvn
> > > > >>>>>> clean
> > > > >>>>>>> verify". Here is the log output:
> > > > >>> http://pastebin.com/raw.php?i=VWbx2ppf
> > > > >>>>>>
> > > > >>>>>> Thanks for reporting this. Have you tried it multiple times?
> Is
> > it
> > > > >>> failing
> > > > >>>>>> reproducibly with the same tests? What's your setup?
> > > > >>>>>>
> > > > >>>>>> – Ufuk
> > > > >>>
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
Yes, we would include those in the new release candidate.
On Jun 11, 2015 5:22 PM, "Aljoscha Krettek" <al...@apache.org> wrote:
> Aren't there still some commits at the top of the release document that
> need to be cherry-picked to the release branch?
>
> On Thu, 11 Jun 2015 at 17:13 Maximilian Michels <mx...@apache.org> wrote:
>
> > The deadlock in the scheduler is now fixed. Based on the changes that
> have
> > been push to the release-0.9 branch, I'd like to create a new release
> > candidate later on. I think we have gotten the most critical issues out
> of
> > the way. Would that be ok for you?
> >
> > On Wed, Jun 10, 2015 at 5:56 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >
> > > Yes, that needs to be fixed IMO
> > >
> > > 2015-06-10 17:51 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> > >
> > > > Yes since it is clearly a deadlock in the scheduler, the current
> > version
> > > > shouldn't be released.
> > > >
> > > > On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > >
> > > > > On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org>
> wrote:
> > > > >
> > > > > > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've
> > located
> > > > its
> > > > > > cause but still need to find out how to fix it.
> > > > >
> > > > > Very good find, Max!
> > > > >
> > > > > Max, Till, and I have looked into this and it is a reproducible
> > > deadlock
> > > > > in the scheduler during concurrent slot release (in failure cases).
> > Max
> > > > > will attach the relevant stack trace to the issue.
> > > > >
> > > > > I think this is a release blocker. Any opinions?
> > > > >
> > > > > – Ufuk
> > > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
Aren't there still some commits at the top of the release document that
need to be cherry-picked to the release branch?
On Thu, 11 Jun 2015 at 17:13 Maximilian Michels <mx...@apache.org> wrote:
> The deadlock in the scheduler is now fixed. Based on the changes that have
> been push to the release-0.9 branch, I'd like to create a new release
> candidate later on. I think we have gotten the most critical issues out of
> the way. Would that be ok for you?
>
> On Wed, Jun 10, 2015 at 5:56 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Yes, that needs to be fixed IMO
> >
> > 2015-06-10 17:51 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> >
> > > Yes since it is clearly a deadlock in the scheduler, the current
> version
> > > shouldn't be released.
> > >
> > > On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > >
> > > > On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
> > > >
> > > > > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've
> located
> > > its
> > > > > cause but still need to find out how to fix it.
> > > >
> > > > Very good find, Max!
> > > >
> > > > Max, Till, and I have looked into this and it is a reproducible
> > deadlock
> > > > in the scheduler during concurrent slot release (in failure cases).
> Max
> > > > will attach the relevant stack trace to the issue.
> > > >
> > > > I think this is a release blocker. Any opinions?
> > > >
> > > > – Ufuk
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Márton Balassi <ba...@gmail.com>.
@Till: This also apples to the streaming connectors.
On Fri, Jun 12, 2015 at 9:45 AM, Till Rohrmann <tr...@apache.org> wrote:
> Hi guys,
>
> I just noticed while testing the TableAPI on the cluster that it is not
> part of the dist module. Therefore, programs using the TableAPI will only
> run when you put the TableAPI jar directly on the cluster or if you build a
> fat jar including the TableAPI jar. This is nowhere documented.
> Furthermore, this also applies to Gelly and FlinkML.
>
> Cheers,
> Till
>
> On Fri, Jun 12, 2015 at 9:16 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I'm currently going through the license file and I discovered some
> > skeletons in our closet. This has to be merged as well. But I'm still
> > working on it (we have a lot of dependencies).
> >
> > Cheers,
> > Till
> >
> >
> > On Fri, Jun 12, 2015 at 12:51 AM Ufuk Celebi <uc...@apache.org> wrote:
> >
> >>
> >> On 12 Jun 2015, at 00:49, Fabian Hueske <fh...@gmail.com> wrote:
> >>
> >> > 2. is basically done. I have a patch which updates the counters on
> page
> >> > reload but that shouldn't be hard to extend to dynamic updates.
> >>
> >> Very nice! :-) Thanks!
> >>
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
I have another fix, but this is just a documentation update (FLINK-2207)
and will be done soon.
2015-06-12 10:02 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> We should have a nightly cluster test for every library. Let's keep that in
> mind for the future. Very nice find, Till!
>
> Since there were not objections, I cherry-picked the proposed commits from
> the document to the release-0.9 branch. If I understand correctly, we can
> create the new release candidate once Till has checked the licenses, Ufuk's
> TableInput fix has been merged, and Fabian's web interface improvement are
> in. Plus, we need to include all Flink libraries in flink-dist. Are you
> going to fix that as well, Till?
>
> On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
> >
> > > Hi guys,
> > >
> > > I just noticed while testing the TableAPI on the cluster that it is not
> > > part of the dist module. Therefore, programs using the TableAPI will
> only
> > > run when you put the TableAPI jar directly on the cluster or if you
> > build a
> > > fat jar including the TableAPI jar. This is nowhere documented.
> > > Furthermore, this also applies to Gelly and FlinkML.
> >
> > I think all of these should be included in the fat jar. They are all
> > highly advertized components.
> >
> > Very good catch, Till! I didn't get around to testing Table API on a
> > cluster, yet.
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 Jun 2015, at 10:29, Till Rohrmann <tr...@apache.org> wrote:
> Well I think the initial idea was to keep the dist jar as small a possible
> and therefore we did not include the libraries. I'm not sure whether we can
> decide this here ad-hoc. If the community says that we shall include these
> libraries then I can add them. But bear in mind that all of them have some
> transitive dependencies which will be added as well.
I'm against the connectors as well, but not having Table API, Flink ML, and Gelly not in seems odd to me.
Or maybe I'm missing something. Someone who wants to try this out has to place the dependencies manually into the lib folder of the Flink installation, right?
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
Just to clarify. If you write a Flink program and include the Table API as
a dependency, then you have to package your program in the JAR with the
Table API and submit it to the cluster. IMHO that's ok but it should be
documented to inform users which libraries are included in Flink binaries
out of the box. Let's document and postpone a discussion whether Flink
should ship all libraries (and all their dependencies, as Till pointed out)
in the future. I think this is too big of a change for a release candidate.
On Fri, Jun 12, 2015 at 10:44 AM, Till Rohrmann <tr...@apache.org>
wrote:
> Yes you're right Ufuk. At the moment the user has to place the jars in the
> lib folder of Flink. If this folder is not shared then he has to do it for
> every node on which Flink runs.
>
> On Fri, Jun 12, 2015 at 10:42 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I think I found a real release blocker. Currently we don't add license
> > files to our shaded jars. For example
> > the flink-shaded-include-yarn-0.9.0-milestone-1.jar shades hadoop code.
> > This code also includes the `org.apache.util.bloom.*` classes. These
> > classes are licensed under The European Commission project OneLab. We
> have
> > a notice in the LICENSE file of our binary distribution but I think we
> also
> > have to add them in the shaded jar. There might even be more code bundled
> > as part of some shaded jars which I have not spotted yet.
> >
> > Furthermore, I noticed that we list all Apache License dependencies in
> our
> > LICENSE file of our binary distribution (which we don't have to do).
> > However, we don't do it in our jars which contain for example guava and
> asm
> > as shaded dependencies. Maybe we should be consistent here.
> >
> > But maybe I overlook something here and we don't have to do it.
> >
> > On Fri, Jun 12, 2015 at 10:29 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> >> Well I think the initial idea was to keep the dist jar as small a
> >> possible and therefore we did not include the libraries. I'm not sure
> >> whether we can decide this here ad-hoc. If the community says that we
> shall
> >> include these libraries then I can add them. But bear in mind that all
> of
> >> them have some transitive dependencies which will be added as well.
> >>
> >>
> >> On Fri, Jun 12, 2015 at 10:15 AM Márton Balassi <
> balassi.marton@gmail.com>
> >> wrote:
> >>
> >>> As for outstanding issues I think streaming is good to go as far as I
> >>> know.
> >>> I am personally against including all libraries - at least speaking for
> >>> the
> >>> streaming connectors. Robert, Stephan and myself had a detailed
> >>> discussion
> >>> on that some time ago and the disadvantage of having all the libraries
> in
> >>> the distribution is the dependency mess that they pull. In this case I
> >>> would rather add documentation on putting them in the user jar then. As
> >>> for
> >>> the other libraries they do not depend on so much external code, so +1
> >>> for
> >>> putting them in.
> >>>
> >>> On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mx...@apache.org>
> >>> wrote:
> >>>
> >>> > We should have a nightly cluster test for every library. Let's keep
> >>> that in
> >>> > mind for the future. Very nice find, Till!
> >>> >
> >>> > Since there were not objections, I cherry-picked the proposed commits
> >>> from
> >>> > the document to the release-0.9 branch. If I understand correctly, we
> >>> can
> >>> > create the new release candidate once Till has checked the licenses,
> >>> Ufuk's
> >>> > TableInput fix has been merged, and Fabian's web interface
> improvement
> >>> are
> >>> > in. Plus, we need to include all Flink libraries in flink-dist. Are
> you
> >>> > going to fix that as well, Till?
> >>> >
> >>> > On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >>> >
> >>> > >
> >>> > > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org>
> >>> wrote:
> >>> > >
> >>> > > > Hi guys,
> >>> > > >
> >>> > > > I just noticed while testing the TableAPI on the cluster that it
> >>> is not
> >>> > > > part of the dist module. Therefore, programs using the TableAPI
> >>> will
> >>> > only
> >>> > > > run when you put the TableAPI jar directly on the cluster or if
> you
> >>> > > build a
> >>> > > > fat jar including the TableAPI jar. This is nowhere documented.
> >>> > > > Furthermore, this also applies to Gelly and FlinkML.
> >>> > >
> >>> > > I think all of these should be included in the fat jar. They are
> all
> >>> > > highly advertized components.
> >>> > >
> >>> > > Very good catch, Till! I didn't get around to testing Table API on
> a
> >>> > > cluster, yet.
> >>> >
> >>>
> >>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 Jun 2015, at 10:44, Till Rohrmann <tr...@apache.org> wrote:
> Yes you're right Ufuk. At the moment the user has to place the jars in the
> lib folder of Flink. If this folder is not shared then he has to do it for
> every node on which Flink runs.
OK. I guess there is a nice way to do this with YARN as well. I think it decreases the out-of-the-box experience quite a bit if you want to use these nice features.
What's your stand on this issue?
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
Yes you're right Ufuk. At the moment the user has to place the jars in the
lib folder of Flink. If this folder is not shared then he has to do it for
every node on which Flink runs.
On Fri, Jun 12, 2015 at 10:42 AM Till Rohrmann <tr...@apache.org> wrote:
> I think I found a real release blocker. Currently we don't add license
> files to our shaded jars. For example
> the flink-shaded-include-yarn-0.9.0-milestone-1.jar shades hadoop code.
> This code also includes the `org.apache.util.bloom.*` classes. These
> classes are licensed under The European Commission project OneLab. We have
> a notice in the LICENSE file of our binary distribution but I think we also
> have to add them in the shaded jar. There might even be more code bundled
> as part of some shaded jars which I have not spotted yet.
>
> Furthermore, I noticed that we list all Apache License dependencies in our
> LICENSE file of our binary distribution (which we don't have to do).
> However, we don't do it in our jars which contain for example guava and asm
> as shaded dependencies. Maybe we should be consistent here.
>
> But maybe I overlook something here and we don't have to do it.
>
> On Fri, Jun 12, 2015 at 10:29 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Well I think the initial idea was to keep the dist jar as small a
>> possible and therefore we did not include the libraries. I'm not sure
>> whether we can decide this here ad-hoc. If the community says that we shall
>> include these libraries then I can add them. But bear in mind that all of
>> them have some transitive dependencies which will be added as well.
>>
>>
>> On Fri, Jun 12, 2015 at 10:15 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>>> As for outstanding issues I think streaming is good to go as far as I
>>> know.
>>> I am personally against including all libraries - at least speaking for
>>> the
>>> streaming connectors. Robert, Stephan and myself had a detailed
>>> discussion
>>> on that some time ago and the disadvantage of having all the libraries in
>>> the distribution is the dependency mess that they pull. In this case I
>>> would rather add documentation on putting them in the user jar then. As
>>> for
>>> the other libraries they do not depend on so much external code, so +1
>>> for
>>> putting them in.
>>>
>>> On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mx...@apache.org>
>>> wrote:
>>>
>>> > We should have a nightly cluster test for every library. Let's keep
>>> that in
>>> > mind for the future. Very nice find, Till!
>>> >
>>> > Since there were not objections, I cherry-picked the proposed commits
>>> from
>>> > the document to the release-0.9 branch. If I understand correctly, we
>>> can
>>> > create the new release candidate once Till has checked the licenses,
>>> Ufuk's
>>> > TableInput fix has been merged, and Fabian's web interface improvement
>>> are
>>> > in. Plus, we need to include all Flink libraries in flink-dist. Are you
>>> > going to fix that as well, Till?
>>> >
>>> > On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>> >
>>> > >
>>> > > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org>
>>> wrote:
>>> > >
>>> > > > Hi guys,
>>> > > >
>>> > > > I just noticed while testing the TableAPI on the cluster that it
>>> is not
>>> > > > part of the dist module. Therefore, programs using the TableAPI
>>> will
>>> > only
>>> > > > run when you put the TableAPI jar directly on the cluster or if you
>>> > > build a
>>> > > > fat jar including the TableAPI jar. This is nowhere documented.
>>> > > > Furthermore, this also applies to Gelly and FlinkML.
>>> > >
>>> > > I think all of these should be included in the fat jar. They are all
>>> > > highly advertized components.
>>> > >
>>> > > Very good catch, Till! I didn't get around to testing Table API on a
>>> > > cluster, yet.
>>> >
>>>
>>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
I think I found a real release blocker. Currently we don't add license
files to our shaded jars. For example
the flink-shaded-include-yarn-0.9.0-milestone-1.jar shades hadoop code.
This code also includes the `org.apache.util.bloom.*` classes. These
classes are licensed under The European Commission project OneLab. We have
a notice in the LICENSE file of our binary distribution but I think we also
have to add them in the shaded jar. There might even be more code bundled
as part of some shaded jars which I have not spotted yet.
Furthermore, I noticed that we list all Apache License dependencies in our
LICENSE file of our binary distribution (which we don't have to do).
However, we don't do it in our jars which contain for example guava and asm
as shaded dependencies. Maybe we should be consistent here.
But maybe I overlook something here and we don't have to do it.
On Fri, Jun 12, 2015 at 10:29 AM Till Rohrmann <tr...@apache.org> wrote:
> Well I think the initial idea was to keep the dist jar as small a possible
> and therefore we did not include the libraries. I'm not sure whether we can
> decide this here ad-hoc. If the community says that we shall include these
> libraries then I can add them. But bear in mind that all of them have some
> transitive dependencies which will be added as well.
>
>
> On Fri, Jun 12, 2015 at 10:15 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
>> As for outstanding issues I think streaming is good to go as far as I
>> know.
>> I am personally against including all libraries - at least speaking for
>> the
>> streaming connectors. Robert, Stephan and myself had a detailed discussion
>> on that some time ago and the disadvantage of having all the libraries in
>> the distribution is the dependency mess that they pull. In this case I
>> would rather add documentation on putting them in the user jar then. As
>> for
>> the other libraries they do not depend on so much external code, so +1 for
>> putting them in.
>>
>> On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>> > We should have a nightly cluster test for every library. Let's keep
>> that in
>> > mind for the future. Very nice find, Till!
>> >
>> > Since there were not objections, I cherry-picked the proposed commits
>> from
>> > the document to the release-0.9 branch. If I understand correctly, we
>> can
>> > create the new release candidate once Till has checked the licenses,
>> Ufuk's
>> > TableInput fix has been merged, and Fabian's web interface improvement
>> are
>> > in. Plus, we need to include all Flink libraries in flink-dist. Are you
>> > going to fix that as well, Till?
>> >
>> > On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
>> >
>> > >
>> > > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
>> > >
>> > > > Hi guys,
>> > > >
>> > > > I just noticed while testing the TableAPI on the cluster that it is
>> not
>> > > > part of the dist module. Therefore, programs using the TableAPI will
>> > only
>> > > > run when you put the TableAPI jar directly on the cluster or if you
>> > > build a
>> > > > fat jar including the TableAPI jar. This is nowhere documented.
>> > > > Furthermore, this also applies to Gelly and FlinkML.
>> > >
>> > > I think all of these should be included in the fat jar. They are all
>> > > highly advertized components.
>> > >
>> > > Very good catch, Till! I didn't get around to testing Table API on a
>> > > cluster, yet.
>> >
>>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
Well I think the initial idea was to keep the dist jar as small a possible
and therefore we did not include the libraries. I'm not sure whether we can
decide this here ad-hoc. If the community says that we shall include these
libraries then I can add them. But bear in mind that all of them have some
transitive dependencies which will be added as well.
On Fri, Jun 12, 2015 at 10:15 AM Márton Balassi <ba...@gmail.com>
wrote:
> As for outstanding issues I think streaming is good to go as far as I know.
> I am personally against including all libraries - at least speaking for the
> streaming connectors. Robert, Stephan and myself had a detailed discussion
> on that some time ago and the disadvantage of having all the libraries in
> the distribution is the dependency mess that they pull. In this case I
> would rather add documentation on putting them in the user jar then. As for
> the other libraries they do not depend on so much external code, so +1 for
> putting them in.
>
> On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mx...@apache.org>
> wrote:
>
> > We should have a nightly cluster test for every library. Let's keep that
> in
> > mind for the future. Very nice find, Till!
> >
> > Since there were not objections, I cherry-picked the proposed commits
> from
> > the document to the release-0.9 branch. If I understand correctly, we can
> > create the new release candidate once Till has checked the licenses,
> Ufuk's
> > TableInput fix has been merged, and Fabian's web interface improvement
> are
> > in. Plus, we need to include all Flink libraries in flink-dist. Are you
> > going to fix that as well, Till?
> >
> > On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > >
> > > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I just noticed while testing the TableAPI on the cluster that it is
> not
> > > > part of the dist module. Therefore, programs using the TableAPI will
> > only
> > > > run when you put the TableAPI jar directly on the cluster or if you
> > > build a
> > > > fat jar including the TableAPI jar. This is nowhere documented.
> > > > Furthermore, this also applies to Gelly and FlinkML.
> > >
> > > I think all of these should be included in the fat jar. They are all
> > > highly advertized components.
> > >
> > > Very good catch, Till! I didn't get around to testing Table API on a
> > > cluster, yet.
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Márton Balassi <ba...@gmail.com>.
As for outstanding issues I think streaming is good to go as far as I know.
I am personally against including all libraries - at least speaking for the
streaming connectors. Robert, Stephan and myself had a detailed discussion
on that some time ago and the disadvantage of having all the libraries in
the distribution is the dependency mess that they pull. In this case I
would rather add documentation on putting them in the user jar then. As for
the other libraries they do not depend on so much external code, so +1 for
putting them in.
On Fri, Jun 12, 2015 at 10:02 AM, Maximilian Michels <mx...@apache.org> wrote:
> We should have a nightly cluster test for every library. Let's keep that in
> mind for the future. Very nice find, Till!
>
> Since there were not objections, I cherry-picked the proposed commits from
> the document to the release-0.9 branch. If I understand correctly, we can
> create the new release candidate once Till has checked the licenses, Ufuk's
> TableInput fix has been merged, and Fabian's web interface improvement are
> in. Plus, we need to include all Flink libraries in flink-dist. Are you
> going to fix that as well, Till?
>
> On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
> >
> > > Hi guys,
> > >
> > > I just noticed while testing the TableAPI on the cluster that it is not
> > > part of the dist module. Therefore, programs using the TableAPI will
> only
> > > run when you put the TableAPI jar directly on the cluster or if you
> > build a
> > > fat jar including the TableAPI jar. This is nowhere documented.
> > > Furthermore, this also applies to Gelly and FlinkML.
> >
> > I think all of these should be included in the fat jar. They are all
> > highly advertized components.
> >
> > Very good catch, Till! I didn't get around to testing Table API on a
> > cluster, yet.
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
We should have a nightly cluster test for every library. Let's keep that in
mind for the future. Very nice find, Till!
Since there were not objections, I cherry-picked the proposed commits from
the document to the release-0.9 branch. If I understand correctly, we can
create the new release candidate once Till has checked the licenses, Ufuk's
TableInput fix has been merged, and Fabian's web interface improvement are
in. Plus, we need to include all Flink libraries in flink-dist. Are you
going to fix that as well, Till?
On Fri, Jun 12, 2015 at 9:53 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
>
> > Hi guys,
> >
> > I just noticed while testing the TableAPI on the cluster that it is not
> > part of the dist module. Therefore, programs using the TableAPI will only
> > run when you put the TableAPI jar directly on the cluster or if you
> build a
> > fat jar including the TableAPI jar. This is nowhere documented.
> > Furthermore, this also applies to Gelly and FlinkML.
>
> I think all of these should be included in the fat jar. They are all
> highly advertized components.
>
> Very good catch, Till! I didn't get around to testing Table API on a
> cluster, yet.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 Jun 2015, at 09:45, Till Rohrmann <tr...@apache.org> wrote:
> Hi guys,
>
> I just noticed while testing the TableAPI on the cluster that it is not
> part of the dist module. Therefore, programs using the TableAPI will only
> run when you put the TableAPI jar directly on the cluster or if you build a
> fat jar including the TableAPI jar. This is nowhere documented.
> Furthermore, this also applies to Gelly and FlinkML.
I think all of these should be included in the fat jar. They are all highly advertized components.
Very good catch, Till! I didn't get around to testing Table API on a cluster, yet.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
Hi guys,
I just noticed while testing the TableAPI on the cluster that it is not
part of the dist module. Therefore, programs using the TableAPI will only
run when you put the TableAPI jar directly on the cluster or if you build a
fat jar including the TableAPI jar. This is nowhere documented.
Furthermore, this also applies to Gelly and FlinkML.
Cheers,
Till
On Fri, Jun 12, 2015 at 9:16 AM Till Rohrmann <tr...@apache.org> wrote:
> I'm currently going through the license file and I discovered some
> skeletons in our closet. This has to be merged as well. But I'm still
> working on it (we have a lot of dependencies).
>
> Cheers,
> Till
>
>
> On Fri, Jun 12, 2015 at 12:51 AM Ufuk Celebi <uc...@apache.org> wrote:
>
>>
>> On 12 Jun 2015, at 00:49, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > 2. is basically done. I have a patch which updates the counters on page
>> > reload but that shouldn't be hard to extend to dynamic updates.
>>
>> Very nice! :-) Thanks!
>>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
I'm currently going through the license file and I discovered some
skeletons in our closet. This has to be merged as well. But I'm still
working on it (we have a lot of dependencies).
Cheers,
Till
On Fri, Jun 12, 2015 at 12:51 AM Ufuk Celebi <uc...@apache.org> wrote:
>
> On 12 Jun 2015, at 00:49, Fabian Hueske <fh...@gmail.com> wrote:
>
> > 2. is basically done. I have a patch which updates the counters on page
> > reload but that shouldn't be hard to extend to dynamic updates.
>
> Very nice! :-) Thanks!
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 Jun 2015, at 00:49, Fabian Hueske <fh...@gmail.com> wrote:
> 2. is basically done. I have a patch which updates the counters on page
> reload but that shouldn't be hard to extend to dynamic updates.
Very nice! :-) Thanks!
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
2. is basically done. I have a patch which updates the counters on page
reload but that shouldn't be hard to extend to dynamic updates.
2015-06-12 0:40 GMT+02:00 Ufuk Celebi <uc...@apache.org>:
>
> On 11 Jun 2015, at 20:04, Fabian Hueske <fh...@gmail.com> wrote:
>
> > How about the following issues?
> >
> > 1. The Hbase Hadoop Compat issue, Ufuk is working on
>
> I was not able to reproduce this :( I ran HadoopInputFormats against
> various sources and confirmed the results and everything was fine so far.
>
> I think I will open a PR for the small HadoopInputFormat fix and then we
> can start a new RC.
>
> If we manage to reproduce the problem over the next days (I'm in contact
> with the user who reported the original issue [thanks Himi!]), we can still
> include a fix.
>
> > 2. The incorrect webinterface counts
>
> Personally, I don't think 2) is a blocker if it takes much more time. I
> still think it would be good to have it in.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 12 Jun 2015, at 00:40, Ufuk Celebi <uc...@apache.org> wrote:
>
> On 11 Jun 2015, at 20:04, Fabian Hueske <fh...@gmail.com> wrote:
>
>> How about the following issues?
>>
>> 1. The Hbase Hadoop Compat issue, Ufuk is working on
>
> I was not able to reproduce this :( I ran HadoopInputFormats against various sources and confirmed the results and everything was fine so far.
The issue has been resolved as "Not a problem". There was some misconfiguration in the user code.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 11 Jun 2015, at 20:04, Fabian Hueske <fh...@gmail.com> wrote:
> How about the following issues?
>
> 1. The Hbase Hadoop Compat issue, Ufuk is working on
I was not able to reproduce this :( I ran HadoopInputFormats against various sources and confirmed the results and everything was fine so far.
I think I will open a PR for the small HadoopInputFormat fix and then we can start a new RC.
If we manage to reproduce the problem over the next days (I'm in contact with the user who reported the original issue [thanks Himi!]), we can still include a fix.
> 2. The incorrect webinterface counts
Personally, I don't think 2) is a blocker if it takes much more time. I still think it would be good to have it in.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
How about the following issues?
1. The Hbase Hadoop Compat issue, Ufuk is working on
2. The incorrect webinterface counts
@Ufuk were you able to reproduce the bug?
The deadlock in the scheduler is now fixed. Based on the changes that have
been push to the release-0.9 branch, I'd like to create a new release
candidate later on. I think we have gotten the most critical issues out of
the way. Would that be ok for you?
On Wed, Jun 10, 2015 at 5:56 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Yes, that needs to be fixed IMO
>
> 2015-06-10 17:51 GMT+02:00 Till Rohrmann <tr...@apache.org>:
>
> > Yes since it is clearly a deadlock in the scheduler, the current version
> > shouldn't be released.
> >
> > On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
> >
> > >
> > > On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
> > >
> > > > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've
located
> > its
> > > > cause but still need to find out how to fix it.
> > >
> > > Very good find, Max!
> > >
> > > Max, Till, and I have looked into this and it is a reproducible
> deadlock
> > > in the scheduler during concurrent slot release (in failure cases).
Max
> > > will attach the relevant stack trace to the issue.
> > >
> > > I think this is a release blocker. Any opinions?
> > >
> > > – Ufuk
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
The deadlock in the scheduler is now fixed. Based on the changes that have
been push to the release-0.9 branch, I'd like to create a new release
candidate later on. I think we have gotten the most critical issues out of
the way. Would that be ok for you?
On Wed, Jun 10, 2015 at 5:56 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Yes, that needs to be fixed IMO
>
> 2015-06-10 17:51 GMT+02:00 Till Rohrmann <tr...@apache.org>:
>
> > Yes since it is clearly a deadlock in the scheduler, the current version
> > shouldn't be released.
> >
> > On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
> >
> > >
> > > On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
> > >
> > > > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've located
> > its
> > > > cause but still need to find out how to fix it.
> > >
> > > Very good find, Max!
> > >
> > > Max, Till, and I have looked into this and it is a reproducible
> deadlock
> > > in the scheduler during concurrent slot release (in failure cases). Max
> > > will attach the relevant stack trace to the issue.
> > >
> > > I think this is a release blocker. Any opinions?
> > >
> > > – Ufuk
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
Yes, that needs to be fixed IMO
2015-06-10 17:51 GMT+02:00 Till Rohrmann <tr...@apache.org>:
> Yes since it is clearly a deadlock in the scheduler, the current version
> shouldn't be released.
>
> On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
> >
> > > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've located
> its
> > > cause but still need to find out how to fix it.
> >
> > Very good find, Max!
> >
> > Max, Till, and I have looked into this and it is a reproducible deadlock
> > in the scheduler during concurrent slot release (in failure cases). Max
> > will attach the relevant stack trace to the issue.
> >
> > I think this is a release blocker. Any opinions?
> >
> > – Ufuk
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
Yes since it is clearly a deadlock in the scheduler, the current version
shouldn't be released.
On Wed, Jun 10, 2015 at 5:48 PM Ufuk Celebi <uc...@apache.org> wrote:
>
> On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
>
> > I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've located its
> > cause but still need to find out how to fix it.
>
> Very good find, Max!
>
> Max, Till, and I have looked into this and it is a reproducible deadlock
> in the scheduler during concurrent slot release (in failure cases). Max
> will attach the relevant stack trace to the issue.
>
> I think this is a release blocker. Any opinions?
>
> – Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 10 Jun 2015, at 16:18, Maximilian Michels <mx...@apache.org> wrote:
> I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've located its
> cause but still need to find out how to fix it.
Very good find, Max!
Max, Till, and I have looked into this and it is a reproducible deadlock in the scheduler during concurrent slot release (in failure cases). Max will attach the relevant stack trace to the issue.
I think this is a release blocker. Any opinions?
– Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
I'm debugging the TaskManagerFailsWithSlotSharingITCase. I've located its
cause but still need to find out how to fix it.
On Wed, Jun 10, 2015 at 2:25 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I added a section at the top of the release testing document to keep
> track of commits that we might want to cherry-pick to the release. I
> included the YARNSessionFIFOITCase fix and the optional stream
> iteration partitioning (both already on release branch).
>
> On Wed, Jun 10, 2015 at 12:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
> > @Sachin: I reproduced the build error on my Windows machine.
> >
> > 2015-06-10 12:22 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> >
> >> @Sachin: This looks like a file permission issue. We should have someone
> >> else verify that on a Windows system.
> >>
> >> On Wed, Jun 10, 2015 at 11:28 AM, Sachin Goel <sachingoel0101@gmail.com
> >
> >> wrote:
> >>
> >> > I have run "mvn clean verify" five times now and every time I'm
> getting
> >> > these failed tests:
> >> >
> >> > BlobUtilsTest.before:45 null
> >> > BlobUtilsTest.before:45 null
> >> > BlobServerDeleteTest.testDeleteFails:291 null
> >> > BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
> >> > remove write permissions from cache directory
> >> > BlobServerPutTest.testPutBufferFails:224 null
> >> > BlobServerPutTest.testPutNamedBufferFails:286 null
> >> > JobManagerStartupTest.before:55 null
> >> > JobManagerStartupTest.before:55 null
> >> > DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
> >> > not been removed
> >> > DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
> >> > has not been removed
> >> > TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
> >> > timeout (19998080696 nanoseconds) during expectMsgClass waiting for
> >> > class
> >> >
> >>
> org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
> >> > TaskManagerProcessReapingTest.testReapProcessOnFailure:133
> >> > TaskManager process did not launch the TaskManager properly. Failed to
> >> > look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
> >> >
> >> > ** fails randomly.
> >> >
> >> > Is someone able to reproduce these while building on a windows
> machine? I
> >> > would try to debug these myself but I'm not yet familiar with the core
> >> > architecture and API.
> >> >
> >> > -- Sachin
> >> >
> >> > On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <
> aljoscha@apache.org>
> >> > wrote:
> >> >
> >> > > The KMeans quickstart example does not work with the current state
> of
> >> > > the KMeansDataGenerator. I created PR that brings the two in sync.
> >> > > This should probably go into the release since it affects initial
> user
> >> > > "satisfaction".
> >> > >
> >> > > On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
> >> > > <ba...@gmail.com> wrote:
> >> > > > As for the streaming commit cherry-picked to the release branch:
> >> > > > This is an unfortunate communication issue, let us make sure that
> we
> >> > > > clearly communicate similar issues in the future.
> >> > > >
> >> > > > As for FLINK-2192: This is essentially a duplicate issue of the
> >> > > testability
> >> > > > of the streaming iteration. Not a blocker, I will comment on the
> JIRA
> >> > > > ticket, Gabor Hermann is already working on the root cause.
> >> > > >
> >> > > > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org>
> >> wrote:
> >> > > >
> >> > > >> Hey Gyula, Max,
> >> > > >>
> >> > > >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com>
> wrote:
> >> > > >>
> >> > > >> > This feature needs to be included in the release, it has been
> >> tested
> >> > > and
> >> > > >> > used extensively. And many applciations depend on it.
> >> > > >>
> >> > > >> It would be nice to announce/discuss this before just
> cherry-picking
> >> > it
> >> > > >> into the release branch. The issue is that no one (except you)
> knows
> >> > > that
> >> > > >> this is important. Let's just make sure to do this for future
> fixes.
> >> > > >>
> >> > > >> Having said that... it seems to be an important fix. Does someone
> >> have
> >> > > >> time (looking at Aljoscha ;)) to review the changes?
> >> > > >>
> >> > > >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015.
> jún.
> >> > > 10.,
> >> > > >> Sze,
> >> > > >> > 10:47):
> >> > > >> >
> >> > > >> >> With all the issues discovered, it looks like we'll have
> another
> >> > > release
> >> > > >> >> candidate. Right now, we have discovered the following
> problems:
> >> > > >> >>
> >> > > >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> >> > > >> >> 2 No Jar for SessionWindowing example [fixed in #809]
> >> > > >> >> 3 Wrong description of the input format for the graph examples
> >> (eg.
> >> > > >> >> ConnectedComponents) [fixed in #809]
> >> > > >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> >> > > >> >> 5 ComplexIntegrationTest.complexIntegrationTest1()
> (FLINK-2192)
> >> > fails
> >> > > >>
> >> > > >> Can we verify that the tests are defect and not the tested
> >> component?
> >> > ;)
> >> > > >> Otherwise, I would not block the release on flakey tests.
> >> > > >>
> >> > > >> >> 6 Submitting KMeans example to Web Submission Client does not
> >> work
> >> > on
> >> > > >> >> Firefox.
> >> > > >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> >> > > >> >> Do we have someone familiar with the web interface who could
> >> take a
> >> > > >> look at
> >> > > >> >> the Firefox issues?
> >> > > >>
> >> > > >> If not, I would not block the release on this.
> >> > >
> >> >
> >>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I added a section at the top of the release testing document to keep
track of commits that we might want to cherry-pick to the release. I
included the YARNSessionFIFOITCase fix and the optional stream
iteration partitioning (both already on release branch).
On Wed, Jun 10, 2015 at 12:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
> @Sachin: I reproduced the build error on my Windows machine.
>
> 2015-06-10 12:22 GMT+02:00 Maximilian Michels <mx...@apache.org>:
>
>> @Sachin: This looks like a file permission issue. We should have someone
>> else verify that on a Windows system.
>>
>> On Wed, Jun 10, 2015 at 11:28 AM, Sachin Goel <sa...@gmail.com>
>> wrote:
>>
>> > I have run "mvn clean verify" five times now and every time I'm getting
>> > these failed tests:
>> >
>> > BlobUtilsTest.before:45 null
>> > BlobUtilsTest.before:45 null
>> > BlobServerDeleteTest.testDeleteFails:291 null
>> > BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
>> > remove write permissions from cache directory
>> > BlobServerPutTest.testPutBufferFails:224 null
>> > BlobServerPutTest.testPutNamedBufferFails:286 null
>> > JobManagerStartupTest.before:55 null
>> > JobManagerStartupTest.before:55 null
>> > DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
>> > not been removed
>> > DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
>> > has not been removed
>> > TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
>> > timeout (19998080696 nanoseconds) during expectMsgClass waiting for
>> > class
>> >
>> org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
>> > TaskManagerProcessReapingTest.testReapProcessOnFailure:133
>> > TaskManager process did not launch the TaskManager properly. Failed to
>> > look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
>> >
>> > ** fails randomly.
>> >
>> > Is someone able to reproduce these while building on a windows machine? I
>> > would try to debug these myself but I'm not yet familiar with the core
>> > architecture and API.
>> >
>> > -- Sachin
>> >
>> > On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <al...@apache.org>
>> > wrote:
>> >
>> > > The KMeans quickstart example does not work with the current state of
>> > > the KMeansDataGenerator. I created PR that brings the two in sync.
>> > > This should probably go into the release since it affects initial user
>> > > "satisfaction".
>> > >
>> > > On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
>> > > <ba...@gmail.com> wrote:
>> > > > As for the streaming commit cherry-picked to the release branch:
>> > > > This is an unfortunate communication issue, let us make sure that we
>> > > > clearly communicate similar issues in the future.
>> > > >
>> > > > As for FLINK-2192: This is essentially a duplicate issue of the
>> > > testability
>> > > > of the streaming iteration. Not a blocker, I will comment on the JIRA
>> > > > ticket, Gabor Hermann is already working on the root cause.
>> > > >
>> > > > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org>
>> wrote:
>> > > >
>> > > >> Hey Gyula, Max,
>> > > >>
>> > > >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
>> > > >>
>> > > >> > This feature needs to be included in the release, it has been
>> tested
>> > > and
>> > > >> > used extensively. And many applciations depend on it.
>> > > >>
>> > > >> It would be nice to announce/discuss this before just cherry-picking
>> > it
>> > > >> into the release branch. The issue is that no one (except you) knows
>> > > that
>> > > >> this is important. Let's just make sure to do this for future fixes.
>> > > >>
>> > > >> Having said that... it seems to be an important fix. Does someone
>> have
>> > > >> time (looking at Aljoscha ;)) to review the changes?
>> > > >>
>> > > >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún.
>> > > 10.,
>> > > >> Sze,
>> > > >> > 10:47):
>> > > >> >
>> > > >> >> With all the issues discovered, it looks like we'll have another
>> > > release
>> > > >> >> candidate. Right now, we have discovered the following problems:
>> > > >> >>
>> > > >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
>> > > >> >> 2 No Jar for SessionWindowing example [fixed in #809]
>> > > >> >> 3 Wrong description of the input format for the graph examples
>> (eg.
>> > > >> >> ConnectedComponents) [fixed in #809]
>> > > >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
>> > > >> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192)
>> > fails
>> > > >>
>> > > >> Can we verify that the tests are defect and not the tested
>> component?
>> > ;)
>> > > >> Otherwise, I would not block the release on flakey tests.
>> > > >>
>> > > >> >> 6 Submitting KMeans example to Web Submission Client does not
>> work
>> > on
>> > > >> >> Firefox.
>> > > >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
>> > > >> >> Do we have someone familiar with the web interface who could
>> take a
>> > > >> look at
>> > > >> >> the Firefox issues?
>> > > >>
>> > > >> If not, I would not block the release on this.
>> > >
>> >
>>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Fabian Hueske <fh...@gmail.com>.
@Sachin: I reproduced the build error on my Windows machine.
2015-06-10 12:22 GMT+02:00 Maximilian Michels <mx...@apache.org>:
> @Sachin: This looks like a file permission issue. We should have someone
> else verify that on a Windows system.
>
> On Wed, Jun 10, 2015 at 11:28 AM, Sachin Goel <sa...@gmail.com>
> wrote:
>
> > I have run "mvn clean verify" five times now and every time I'm getting
> > these failed tests:
> >
> > BlobUtilsTest.before:45 null
> > BlobUtilsTest.before:45 null
> > BlobServerDeleteTest.testDeleteFails:291 null
> > BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
> > remove write permissions from cache directory
> > BlobServerPutTest.testPutBufferFails:224 null
> > BlobServerPutTest.testPutNamedBufferFails:286 null
> > JobManagerStartupTest.before:55 null
> > JobManagerStartupTest.before:55 null
> > DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
> > not been removed
> > DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
> > has not been removed
> > TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
> > timeout (19998080696 nanoseconds) during expectMsgClass waiting for
> > class
> >
> org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
> > TaskManagerProcessReapingTest.testReapProcessOnFailure:133
> > TaskManager process did not launch the TaskManager properly. Failed to
> > look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
> >
> > ** fails randomly.
> >
> > Is someone able to reproduce these while building on a windows machine? I
> > would try to debug these myself but I'm not yet familiar with the core
> > architecture and API.
> >
> > -- Sachin
> >
> > On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > The KMeans quickstart example does not work with the current state of
> > > the KMeansDataGenerator. I created PR that brings the two in sync.
> > > This should probably go into the release since it affects initial user
> > > "satisfaction".
> > >
> > > On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
> > > <ba...@gmail.com> wrote:
> > > > As for the streaming commit cherry-picked to the release branch:
> > > > This is an unfortunate communication issue, let us make sure that we
> > > > clearly communicate similar issues in the future.
> > > >
> > > > As for FLINK-2192: This is essentially a duplicate issue of the
> > > testability
> > > > of the streaming iteration. Not a blocker, I will comment on the JIRA
> > > > ticket, Gabor Hermann is already working on the root cause.
> > > >
> > > > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org>
> wrote:
> > > >
> > > >> Hey Gyula, Max,
> > > >>
> > > >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
> > > >>
> > > >> > This feature needs to be included in the release, it has been
> tested
> > > and
> > > >> > used extensively. And many applciations depend on it.
> > > >>
> > > >> It would be nice to announce/discuss this before just cherry-picking
> > it
> > > >> into the release branch. The issue is that no one (except you) knows
> > > that
> > > >> this is important. Let's just make sure to do this for future fixes.
> > > >>
> > > >> Having said that... it seems to be an important fix. Does someone
> have
> > > >> time (looking at Aljoscha ;)) to review the changes?
> > > >>
> > > >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún.
> > > 10.,
> > > >> Sze,
> > > >> > 10:47):
> > > >> >
> > > >> >> With all the issues discovered, it looks like we'll have another
> > > release
> > > >> >> candidate. Right now, we have discovered the following problems:
> > > >> >>
> > > >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> > > >> >> 2 No Jar for SessionWindowing example [fixed in #809]
> > > >> >> 3 Wrong description of the input format for the graph examples
> (eg.
> > > >> >> ConnectedComponents) [fixed in #809]
> > > >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> > > >> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192)
> > fails
> > > >>
> > > >> Can we verify that the tests are defect and not the tested
> component?
> > ;)
> > > >> Otherwise, I would not block the release on flakey tests.
> > > >>
> > > >> >> 6 Submitting KMeans example to Web Submission Client does not
> work
> > on
> > > >> >> Firefox.
> > > >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> > > >> >> Do we have someone familiar with the web interface who could
> take a
> > > >> look at
> > > >> >> the Firefox issues?
> > > >>
> > > >> If not, I would not block the release on this.
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
@Sachin: This looks like a file permission issue. We should have someone
else verify that on a Windows system.
On Wed, Jun 10, 2015 at 11:28 AM, Sachin Goel <sa...@gmail.com>
wrote:
> I have run "mvn clean verify" five times now and every time I'm getting
> these failed tests:
>
> BlobUtilsTest.before:45 null
> BlobUtilsTest.before:45 null
> BlobServerDeleteTest.testDeleteFails:291 null
> BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
> remove write permissions from cache directory
> BlobServerPutTest.testPutBufferFails:224 null
> BlobServerPutTest.testPutNamedBufferFails:286 null
> JobManagerStartupTest.before:55 null
> JobManagerStartupTest.before:55 null
> DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
> not been removed
> DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
> has not been removed
> TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
> timeout (19998080696 nanoseconds) during expectMsgClass waiting for
> class
> org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
> TaskManagerProcessReapingTest.testReapProcessOnFailure:133
> TaskManager process did not launch the TaskManager properly. Failed to
> look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
>
> ** fails randomly.
>
> Is someone able to reproduce these while building on a windows machine? I
> would try to debug these myself but I'm not yet familiar with the core
> architecture and API.
>
> -- Sachin
>
> On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > The KMeans quickstart example does not work with the current state of
> > the KMeansDataGenerator. I created PR that brings the two in sync.
> > This should probably go into the release since it affects initial user
> > "satisfaction".
> >
> > On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
> > <ba...@gmail.com> wrote:
> > > As for the streaming commit cherry-picked to the release branch:
> > > This is an unfortunate communication issue, let us make sure that we
> > > clearly communicate similar issues in the future.
> > >
> > > As for FLINK-2192: This is essentially a duplicate issue of the
> > testability
> > > of the streaming iteration. Not a blocker, I will comment on the JIRA
> > > ticket, Gabor Hermann is already working on the root cause.
> > >
> > > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > >> Hey Gyula, Max,
> > >>
> > >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
> > >>
> > >> > This feature needs to be included in the release, it has been tested
> > and
> > >> > used extensively. And many applciations depend on it.
> > >>
> > >> It would be nice to announce/discuss this before just cherry-picking
> it
> > >> into the release branch. The issue is that no one (except you) knows
> > that
> > >> this is important. Let's just make sure to do this for future fixes.
> > >>
> > >> Having said that... it seems to be an important fix. Does someone have
> > >> time (looking at Aljoscha ;)) to review the changes?
> > >>
> > >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún.
> > 10.,
> > >> Sze,
> > >> > 10:47):
> > >> >
> > >> >> With all the issues discovered, it looks like we'll have another
> > release
> > >> >> candidate. Right now, we have discovered the following problems:
> > >> >>
> > >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> > >> >> 2 No Jar for SessionWindowing example [fixed in #809]
> > >> >> 3 Wrong description of the input format for the graph examples (eg.
> > >> >> ConnectedComponents) [fixed in #809]
> > >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> > >> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192)
> fails
> > >>
> > >> Can we verify that the tests are defect and not the tested component?
> ;)
> > >> Otherwise, I would not block the release on flakey tests.
> > >>
> > >> >> 6 Submitting KMeans example to Web Submission Client does not work
> on
> > >> >> Firefox.
> > >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> > >> >> Do we have someone familiar with the web interface who could take a
> > >> look at
> > >> >> the Firefox issues?
> > >>
> > >> If not, I would not block the release on this.
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by "F. Beligianni" <fa...@gmail.com>.
Regarding the iteration partitioning feature, since I use it of course I
find it very useful, but it is true that it needs to be tested more
extensively and also be discussed by the community before it is added in a
release.
Moreover, given the fact that I can still use it for research purposes (I
had already cherry-picked before it is being merged to the master branch),
there is no actual reason to put it in the next release, so that the
community has more time to discuss and decide about the feature.
Lastly, I cross checked the SAMOA application, and till now, there is still
no algorithm implemented in the SAMOA API which needs the new feature.
Faye.
2015-06-10 11:28 GMT+02:00 Sachin Goel <sa...@gmail.com>:
> I have run "mvn clean verify" five times now and every time I'm getting
> these failed tests:
>
> BlobUtilsTest.before:45 null
> BlobUtilsTest.before:45 null
> BlobServerDeleteTest.testDeleteFails:291 null
> BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
> remove write permissions from cache directory
> BlobServerPutTest.testPutBufferFails:224 null
> BlobServerPutTest.testPutNamedBufferFails:286 null
> JobManagerStartupTest.before:55 null
> JobManagerStartupTest.before:55 null
> DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
> not been removed
> DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
> has not been removed
> TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
> timeout (19998080696 nanoseconds) during expectMsgClass waiting for
> class
> org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
> TaskManagerProcessReapingTest.testReapProcessOnFailure:133
> TaskManager process did not launch the TaskManager properly. Failed to
> look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
>
> ** fails randomly.
>
> Is someone able to reproduce these while building on a windows machine? I
> would try to debug these myself but I'm not yet familiar with the core
> architecture and API.
>
> -- Sachin
>
> On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > The KMeans quickstart example does not work with the current state of
> > the KMeansDataGenerator. I created PR that brings the two in sync.
> > This should probably go into the release since it affects initial user
> > "satisfaction".
> >
> > On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
> > <ba...@gmail.com> wrote:
> > > As for the streaming commit cherry-picked to the release branch:
> > > This is an unfortunate communication issue, let us make sure that we
> > > clearly communicate similar issues in the future.
> > >
> > > As for FLINK-2192: This is essentially a duplicate issue of the
> > testability
> > > of the streaming iteration. Not a blocker, I will comment on the JIRA
> > > ticket, Gabor Hermann is already working on the root cause.
> > >
> > > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > >> Hey Gyula, Max,
> > >>
> > >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
> > >>
> > >> > This feature needs to be included in the release, it has been tested
> > and
> > >> > used extensively. And many applciations depend on it.
> > >>
> > >> It would be nice to announce/discuss this before just cherry-picking
> it
> > >> into the release branch. The issue is that no one (except you) knows
> > that
> > >> this is important. Let's just make sure to do this for future fixes.
> > >>
> > >> Having said that... it seems to be an important fix. Does someone have
> > >> time (looking at Aljoscha ;)) to review the changes?
> > >>
> > >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún.
> > 10.,
> > >> Sze,
> > >> > 10:47):
> > >> >
> > >> >> With all the issues discovered, it looks like we'll have another
> > release
> > >> >> candidate. Right now, we have discovered the following problems:
> > >> >>
> > >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> > >> >> 2 No Jar for SessionWindowing example [fixed in #809]
> > >> >> 3 Wrong description of the input format for the graph examples (eg.
> > >> >> ConnectedComponents) [fixed in #809]
> > >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> > >> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192)
> fails
> > >>
> > >> Can we verify that the tests are defect and not the tested component?
> ;)
> > >> Otherwise, I would not block the release on flakey tests.
> > >>
> > >> >> 6 Submitting KMeans example to Web Submission Client does not work
> on
> > >> >> Firefox.
> > >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> > >> >> Do we have someone familiar with the web interface who could take a
> > >> look at
> > >> >> the Firefox issues?
> > >>
> > >> If not, I would not block the release on this.
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Sachin Goel <sa...@gmail.com>.
I have run "mvn clean verify" five times now and every time I'm getting
these failed tests:
BlobUtilsTest.before:45 null
BlobUtilsTest.before:45 null
BlobServerDeleteTest.testDeleteFails:291 null
BlobLibraryCacheManagerTest.testRegisterAndDownload:196 Could not
remove write permissions from cache directory
BlobServerPutTest.testPutBufferFails:224 null
BlobServerPutTest.testPutNamedBufferFails:286 null
JobManagerStartupTest.before:55 null
JobManagerStartupTest.before:55 null
DataSinkTaskTest.testFailingDataSinkTask:317 Temp output file has
not been removed
DataSinkTaskTest.testFailingSortingDataSinkTask:358 Temp output file
has not been removed
TaskManagerTest.testSubmitAndExecuteTask**:123 assertion failed:
timeout (19998080696 nanoseconds) during expectMsgClass waiting for
class org.apache.flink.runtime.messages.RegistrationMessages$RegisterTaskManager
TaskManagerProcessReapingTest.testReapProcessOnFailure:133
TaskManager process did not launch the TaskManager properly. Failed to
look up akka.tcp://flink@127.0.0.1:50673/user/taskmanager
** fails randomly.
Is someone able to reproduce these while building on a windows machine? I
would try to debug these myself but I'm not yet familiar with the core
architecture and API.
-- Sachin
On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> The KMeans quickstart example does not work with the current state of
> the KMeansDataGenerator. I created PR that brings the two in sync.
> This should probably go into the release since it affects initial user
> "satisfaction".
>
> On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
> <ba...@gmail.com> wrote:
> > As for the streaming commit cherry-picked to the release branch:
> > This is an unfortunate communication issue, let us make sure that we
> > clearly communicate similar issues in the future.
> >
> > As for FLINK-2192: This is essentially a duplicate issue of the
> testability
> > of the streaming iteration. Not a blocker, I will comment on the JIRA
> > ticket, Gabor Hermann is already working on the root cause.
> >
> > On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> >> Hey Gyula, Max,
> >>
> >> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
> >>
> >> > This feature needs to be included in the release, it has been tested
> and
> >> > used extensively. And many applciations depend on it.
> >>
> >> It would be nice to announce/discuss this before just cherry-picking it
> >> into the release branch. The issue is that no one (except you) knows
> that
> >> this is important. Let's just make sure to do this for future fixes.
> >>
> >> Having said that... it seems to be an important fix. Does someone have
> >> time (looking at Aljoscha ;)) to review the changes?
> >>
> >> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún.
> 10.,
> >> Sze,
> >> > 10:47):
> >> >
> >> >> With all the issues discovered, it looks like we'll have another
> release
> >> >> candidate. Right now, we have discovered the following problems:
> >> >>
> >> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> >> >> 2 No Jar for SessionWindowing example [fixed in #809]
> >> >> 3 Wrong description of the input format for the graph examples (eg.
> >> >> ConnectedComponents) [fixed in #809]
> >> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> >> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
> >>
> >> Can we verify that the tests are defect and not the tested component? ;)
> >> Otherwise, I would not block the release on flakey tests.
> >>
> >> >> 6 Submitting KMeans example to Web Submission Client does not work on
> >> >> Firefox.
> >> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> >> >> Do we have someone familiar with the web interface who could take a
> >> look at
> >> >> the Firefox issues?
> >>
> >> If not, I would not block the release on this.
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
The KMeans quickstart example does not work with the current state of
the KMeansDataGenerator. I created PR that brings the two in sync.
This should probably go into the release since it affects initial user
"satisfaction".
On Wed, Jun 10, 2015 at 11:14 AM, Márton Balassi
<ba...@gmail.com> wrote:
> As for the streaming commit cherry-picked to the release branch:
> This is an unfortunate communication issue, let us make sure that we
> clearly communicate similar issues in the future.
>
> As for FLINK-2192: This is essentially a duplicate issue of the testability
> of the streaming iteration. Not a blocker, I will comment on the JIRA
> ticket, Gabor Hermann is already working on the root cause.
>
> On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
>> Hey Gyula, Max,
>>
>> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
>>
>> > This feature needs to be included in the release, it has been tested and
>> > used extensively. And many applciations depend on it.
>>
>> It would be nice to announce/discuss this before just cherry-picking it
>> into the release branch. The issue is that no one (except you) knows that
>> this is important. Let's just make sure to do this for future fixes.
>>
>> Having said that... it seems to be an important fix. Does someone have
>> time (looking at Aljoscha ;)) to review the changes?
>>
>> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10.,
>> Sze,
>> > 10:47):
>> >
>> >> With all the issues discovered, it looks like we'll have another release
>> >> candidate. Right now, we have discovered the following problems:
>> >>
>> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
>> >> 2 No Jar for SessionWindowing example [fixed in #809]
>> >> 3 Wrong description of the input format for the graph examples (eg.
>> >> ConnectedComponents) [fixed in #809]
>> >> 4 TaskManagerFailsWithSlotSharingITCase fails
>> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
>>
>> Can we verify that the tests are defect and not the tested component? ;)
>> Otherwise, I would not block the release on flakey tests.
>>
>> >> 6 Submitting KMeans example to Web Submission Client does not work on
>> >> Firefox.
>> >> 7 Zooming is buggy in Web Submission Client (Firefox)
>> >> Do we have someone familiar with the web interface who could take a
>> look at
>> >> the Firefox issues?
>>
>> If not, I would not block the release on this.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Márton Balassi <ba...@gmail.com>.
As for the streaming commit cherry-picked to the release branch:
This is an unfortunate communication issue, let us make sure that we
clearly communicate similar issues in the future.
As for FLINK-2192: This is essentially a duplicate issue of the testability
of the streaming iteration. Not a blocker, I will comment on the JIRA
ticket, Gabor Hermann is already working on the root cause.
On Wed, Jun 10, 2015 at 11:07 AM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Gyula, Max,
>
> On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
>
> > This feature needs to be included in the release, it has been tested and
> > used extensively. And many applciations depend on it.
>
> It would be nice to announce/discuss this before just cherry-picking it
> into the release branch. The issue is that no one (except you) knows that
> this is important. Let's just make sure to do this for future fixes.
>
> Having said that... it seems to be an important fix. Does someone have
> time (looking at Aljoscha ;)) to review the changes?
>
> > Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10.,
> Sze,
> > 10:47):
> >
> >> With all the issues discovered, it looks like we'll have another release
> >> candidate. Right now, we have discovered the following problems:
> >>
> >> 1 YARN ITCase fails [fixed via 2eb5cfe]
> >> 2 No Jar for SessionWindowing example [fixed in #809]
> >> 3 Wrong description of the input format for the graph examples (eg.
> >> ConnectedComponents) [fixed in #809]
> >> 4 TaskManagerFailsWithSlotSharingITCase fails
> >> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
>
> Can we verify that the tests are defect and not the tested component? ;)
> Otherwise, I would not block the release on flakey tests.
>
> >> 6 Submitting KMeans example to Web Submission Client does not work on
> >> Firefox.
> >> 7 Zooming is buggy in Web Submission Client (Firefox)
> >> Do we have someone familiar with the web interface who could take a
> look at
> >> the Firefox issues?
>
> If not, I would not block the release on this.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Gyula, Max,
On 10 Jun 2015, at 10:54, Gyula Fóra <gy...@gmail.com> wrote:
> This feature needs to be included in the release, it has been tested and
> used extensively. And many applciations depend on it.
It would be nice to announce/discuss this before just cherry-picking it into the release branch. The issue is that no one (except you) knows that this is important. Let's just make sure to do this for future fixes.
Having said that... it seems to be an important fix. Does someone have time (looking at Aljoscha ;)) to review the changes?
> Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10., Sze,
> 10:47):
>
>> With all the issues discovered, it looks like we'll have another release
>> candidate. Right now, we have discovered the following problems:
>>
>> 1 YARN ITCase fails [fixed via 2eb5cfe]
>> 2 No Jar for SessionWindowing example [fixed in #809]
>> 3 Wrong description of the input format for the graph examples (eg.
>> ConnectedComponents) [fixed in #809]
>> 4 TaskManagerFailsWithSlotSharingITCase fails
>> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
Can we verify that the tests are defect and not the tested component? ;) Otherwise, I would not block the release on flakey tests.
>> 6 Submitting KMeans example to Web Submission Client does not work on
>> Firefox.
>> 7 Zooming is buggy in Web Submission Client (Firefox)
>> Do we have someone familiar with the web interface who could take a look at
>> the Firefox issues?
If not, I would not block the release on this.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Gyula Fóra <gy...@gmail.com>.
This feature needs to be included in the release, it has been tested and
used extensively. And many applciations depend on it.
Maximilian Michels <mx...@apache.org> ezt írta (időpont: 2015. jún. 10., Sze,
10:47):
> With all the issues discovered, it looks like we'll have another release
> candidate. Right now, we have discovered the following problems:
>
> 1 YARN ITCase fails [fixed via 2eb5cfe]
> 2 No Jar for SessionWindowing example [fixed in #809]
> 3 Wrong description of the input format for the graph examples (eg.
> ConnectedComponents) [fixed in #809]
> 4 TaskManagerFailsWithSlotSharingITCase fails
> 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
> 6 Submitting KMeans example to Web Submission Client does not work on
> Firefox.
> 7 Zooming is buggy in Web Submission Client (Firefox)
>
> Do we have someone familiar with the web interface who could take a look at
> the Firefox issues?
>
> One more important thing: The release-0.9 branch should only be used for
> bug fixes or prior discussed feature changes. Adding new features defies
> the purpose of carefully testing in advance and can have unforeseeable
> consequences. In particular, I'm referring to #810 pull request:
> https://github.com/apache/flink/pull/810
>
> IMHO, this one shouldn't have been cherry-picked onto the release-0.9
> branch. I would like to remove it from there if no objections are raised.
>
>
> https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c947f8ce
>
> On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > This doesn't look good, yes.
> >
> > On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > While looking into FLINK-2188 (HBase input) I've discovered that Hadoop
> > > input formats implementing Configurable (like
> mapreduce.TableInputFormat)
> > > don't have the Hadoop configuration set via setConf(Configuration).
> > >
> > > I have a small fix for this, which I have to clean up. First, I wanted
> to
> > > check what you think about this issue wrt the release. Personally, I
> > think
> > > this is a release blocker, because it essentially means that no Hadoop
> > > input format, which relies on the Configuration instance to be set this
> > way
> > > will work (this is to some extent a bug of the respective input
> formats)
> > –
> > > most notably the HBase TableInputFormat.
> > >
> > > – Ufuk
> > >
> > > On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com> wrote:
> > >
> > > > I attached jps and jstack log about hanging
> > > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> > > >
> > > > Regards,
> > > > Chiwan Park
> > > >
> > > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > > >>
> > > >> I discovered something that might be a feature, rather than a bug.
> > When
> > > you
> > > >> submit an example using the web client without giving parameters the
> > > >> program fails with this:
> > > >>
> > > >> org.apache.flink.client.program.ProgramInvocationException: The main
> > > method
> > > >> caused an error.
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > > >>
> > > >> at org.apache.flink.client.program.Client.run(Client.java:315)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> > > >>
> > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> > > >>
> > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> > > >>
> > > >> at
> > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > > >>
> > > >> at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > > >>
> > > >> at
> > >
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > > >>
> > > >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > > >>
> > > >> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> > > >>
> > > >> at
> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > > >>
> > > >> at
> > > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > > >>
> > > >> at java.lang.Thread.run(Thread.java:745)
> > > >>
> > > >> Caused by: java.lang.NullPointerException
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> > > >>
> > > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> > > >>
> > > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> > > >>
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >>
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > >>
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >>
> > > >> at java.lang.reflect.Method.invoke(Method.java:497)
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > > >>
> > > >> ... 24 more
> > > >>
> > > >>
> > > >> This also only occurs when you uncheck the "suspend execution while
> > > showing
> > > >> plan".
> > > >>
> > > >> I think this arises because the new print() uses collect() which
> tries
> > > to
> > > >> get the job execution result. I guess the result is Null since the
> job
> > > is
> > > >> submitted asynchronously when the checkbox is unchecked.
> > > >>
> > > >>
> > > >> Other than that, the new print() is pretty sweet when you run the
> > > builtin
> > > >> examples from the CLI. You get all the state changes and also the
> > > result,
> > > >> even when running in cluster mode on several task managers. :D
> > > >>
> > > >>
> > > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <
> aljoscha@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >>> I discovered another problem:
> > > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure
> cleaner
> > > >>> cannot be disabled in part of the Streaming Java API and all of the
> > > >>> Streaming Scala API. I think this is a release blocker (in addition
> > > >>> the the other bugs found so far.)
> > > >>>
> > > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <
> > aljoscha@apache.org>
> > > >>> wrote:
> > > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was
> > comparing
> > > >>>> the hostname to a hostname in some yarn config. In one case it was
> > > >>>> capitalised, in the other case it wasn't.
> > > >>>>
> > > >>>> Pushing fix to master and release-0.9 branch.
> > > >>>>
> > > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <
> > sachingoel0101@gmail.com
> > > >
> > > >>> wrote:
> > > >>>>> A re-ran lead to reproducibility of 11 failures again.
> > > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a
> > time-out
> > > but
> > > >>>>> managed to succeed in a re-run. Here is the log output again:
> > > >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> > > >>>>>
> > > >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> > > >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> > > >>>>>
> > > >>>>> Regards
> > > >>>>> Sachin Goel
> > > >>>>>
> > > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org>
> > wrote:
> > > >>>>>
> > > >>>>>>
> > > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sachingoel0101@gmail.com
> >
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>>> On my local machine, several flink runtime tests are failing on
> > > "mvn
> > > >>>>>> clean
> > > >>>>>>> verify". Here is the log output:
> > > >>> http://pastebin.com/raw.php?i=VWbx2ppf
> > > >>>>>>
> > > >>>>>> Thanks for reporting this. Have you tried it multiple times? Is
> it
> > > >>> failing
> > > >>>>>> reproducibly with the same tests? What's your setup?
> > > >>>>>>
> > > >>>>>> – Ufuk
> > > >>>
> > > >
> > > >
> > > >
> > >
> > >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
With all the issues discovered, it looks like we'll have another release
candidate. Right now, we have discovered the following problems:
1 YARN ITCase fails [fixed via 2eb5cfe]
2 No Jar for SessionWindowing example [fixed in #809]
3 Wrong description of the input format for the graph examples (eg.
ConnectedComponents) [fixed in #809]
4 TaskManagerFailsWithSlotSharingITCase fails
5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails
6 Submitting KMeans example to Web Submission Client does not work on
Firefox.
7 Zooming is buggy in Web Submission Client (Firefox)
Do we have someone familiar with the web interface who could take a look at
the Firefox issues?
One more important thing: The release-0.9 branch should only be used for
bug fixes or prior discussed feature changes. Adding new features defies
the purpose of carefully testing in advance and can have unforeseeable
consequences. In particular, I'm referring to #810 pull request:
https://github.com/apache/flink/pull/810
IMHO, this one shouldn't have been cherry-picked onto the release-0.9
branch. I would like to remove it from there if no objections are raised.
https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c947f8ce
On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> This doesn't look good, yes.
>
> On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > While looking into FLINK-2188 (HBase input) I've discovered that Hadoop
> > input formats implementing Configurable (like mapreduce.TableInputFormat)
> > don't have the Hadoop configuration set via setConf(Configuration).
> >
> > I have a small fix for this, which I have to clean up. First, I wanted to
> > check what you think about this issue wrt the release. Personally, I
> think
> > this is a release blocker, because it essentially means that no Hadoop
> > input format, which relies on the Configuration instance to be set this
> way
> > will work (this is to some extent a bug of the respective input formats)
> –
> > most notably the HBase TableInputFormat.
> >
> > – Ufuk
> >
> > On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com> wrote:
> >
> > > I attached jps and jstack log about hanging
> > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> > >
> > > Regards,
> > > Chiwan Park
> > >
> > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> > >>
> > >> I discovered something that might be a feature, rather than a bug.
> When
> > you
> > >> submit an example using the web client without giving parameters the
> > >> program fails with this:
> > >>
> > >> org.apache.flink.client.program.ProgramInvocationException: The main
> > method
> > >> caused an error.
> > >>
> > >> at
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> > >>
> > >> at
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > >>
> > >> at org.apache.flink.client.program.Client.run(Client.java:315)
> > >>
> > >> at
> > >>
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> > >>
> > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> > >>
> > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> > >>
> > >> at
> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > >>
> > >> at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > >>
> > >> at
> > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > >>
> > >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > >>
> > >> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> > >>
> > >> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > >>
> > >> at
> > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > >>
> > >> at
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > >>
> > >> at java.lang.Thread.run(Thread.java:745)
> > >>
> > >> Caused by: java.lang.NullPointerException
> > >>
> > >> at
> > >>
> >
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> > >>
> > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> > >>
> > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> > >>
> > >> at
> > >>
> >
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> > >>
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >>
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >>
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >>
> > >> at java.lang.reflect.Method.invoke(Method.java:497)
> > >>
> > >> at
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > >>
> > >> ... 24 more
> > >>
> > >>
> > >> This also only occurs when you uncheck the "suspend execution while
> > showing
> > >> plan".
> > >>
> > >> I think this arises because the new print() uses collect() which tries
> > to
> > >> get the job execution result. I guess the result is Null since the job
> > is
> > >> submitted asynchronously when the checkbox is unchecked.
> > >>
> > >>
> > >> Other than that, the new print() is pretty sweet when you run the
> > builtin
> > >> examples from the CLI. You get all the state changes and also the
> > result,
> > >> even when running in cluster mode on several task managers. :D
> > >>
> > >>
> > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <aljoscha@apache.org
> >
> > >> wrote:
> > >>
> > >>> I discovered another problem:
> > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
> > >>> cannot be disabled in part of the Streaming Java API and all of the
> > >>> Streaming Scala API. I think this is a release blocker (in addition
> > >>> the the other bugs found so far.)
> > >>>
> > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <
> aljoscha@apache.org>
> > >>> wrote:
> > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was
> comparing
> > >>>> the hostname to a hostname in some yarn config. In one case it was
> > >>>> capitalised, in the other case it wasn't.
> > >>>>
> > >>>> Pushing fix to master and release-0.9 branch.
> > >>>>
> > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <
> sachingoel0101@gmail.com
> > >
> > >>> wrote:
> > >>>>> A re-ran lead to reproducibility of 11 failures again.
> > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a
> time-out
> > but
> > >>>>> managed to succeed in a re-run. Here is the log output again:
> > >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> > >>>>>
> > >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> > >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> > >>>>>
> > >>>>> Regards
> > >>>>> Sachin Goel
> > >>>>>
> > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org>
> wrote:
> > >>>>>
> > >>>>>>
> > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> On my local machine, several flink runtime tests are failing on
> > "mvn
> > >>>>>> clean
> > >>>>>>> verify". Here is the log output:
> > >>> http://pastebin.com/raw.php?i=VWbx2ppf
> > >>>>>>
> > >>>>>> Thanks for reporting this. Have you tried it multiple times? Is it
> > >>> failing
> > >>>>>> reproducibly with the same tests? What's your setup?
> > >>>>>>
> > >>>>>> – Ufuk
> > >>>
> > >
> > >
> > >
> >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
This doesn't look good, yes.
On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi <uc...@apache.org> wrote:
> While looking into FLINK-2188 (HBase input) I've discovered that Hadoop
> input formats implementing Configurable (like mapreduce.TableInputFormat)
> don't have the Hadoop configuration set via setConf(Configuration).
>
> I have a small fix for this, which I have to clean up. First, I wanted to
> check what you think about this issue wrt the release. Personally, I think
> this is a release blocker, because it essentially means that no Hadoop
> input format, which relies on the Configuration instance to be set this way
> will work (this is to some extent a bug of the respective input formats) –
> most notably the HBase TableInputFormat.
>
> – Ufuk
>
> On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com> wrote:
>
> > I attached jps and jstack log about hanging
> TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
> >
> > Regards,
> > Chiwan Park
> >
> >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
> >>
> >> I discovered something that might be a feature, rather than a bug. When
> you
> >> submit an example using the web client without giving parameters the
> >> program fails with this:
> >>
> >> org.apache.flink.client.program.ProgramInvocationException: The main
> method
> >> caused an error.
> >>
> >> at
> >>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> >>
> >> at
> >>
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> >>
> >> at org.apache.flink.client.program.Client.run(Client.java:315)
> >>
> >> at
> >>
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
> >>
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> >>
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> >>
> >> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> >>
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> >>
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> >>
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> >>
> >> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> >>
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> >>
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> >>
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> >>
> >> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> >>
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> >>
> >> at org.eclipse.jetty.server.Server.handle(Server.java:352)
> >>
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> >>
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> >>
> >> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
> >>
> >> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> >>
> >> at
> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> >>
> >> at
> >>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> >>
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> >>
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> Caused by: java.lang.NullPointerException
> >>
> >> at
> >>
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
> >>
> >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
> >>
> >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
> >>
> >> at
> >>
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
> >>
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:497)
> >>
> >> at
> >>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> >>
> >> ... 24 more
> >>
> >>
> >> This also only occurs when you uncheck the "suspend execution while
> showing
> >> plan".
> >>
> >> I think this arises because the new print() uses collect() which tries
> to
> >> get the job execution result. I guess the result is Null since the job
> is
> >> submitted asynchronously when the checkbox is unchecked.
> >>
> >>
> >> Other than that, the new print() is pretty sweet when you run the
> builtin
> >> examples from the CLI. You get all the state changes and also the
> result,
> >> even when running in cluster mode on several task managers. :D
> >>
> >>
> >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <al...@apache.org>
> >> wrote:
> >>
> >>> I discovered another problem:
> >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
> >>> cannot be disabled in part of the Streaming Java API and all of the
> >>> Streaming Scala API. I think this is a release blocker (in addition
> >>> the the other bugs found so far.)
> >>>
> >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <al...@apache.org>
> >>> wrote:
> >>>> I found the bug in the failing YARNSessionFIFOITCase: It was comparing
> >>>> the hostname to a hostname in some yarn config. In one case it was
> >>>> capitalised, in the other case it wasn't.
> >>>>
> >>>> Pushing fix to master and release-0.9 branch.
> >>>>
> >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sachingoel0101@gmail.com
> >
> >>> wrote:
> >>>>> A re-ran lead to reproducibility of 11 failures again.
> >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out
> but
> >>>>> managed to succeed in a re-run. Here is the log output again:
> >>>>> http://pastebin.com/raw.php?i=N4cm1J18
> >>>>>
> >>>>> Setup: JDK 1.8.0_40 on windows 8.1
> >>>>> System memory: 8GB, quad-core with maximum 8 threads.
> >>>>>
> >>>>> Regards
> >>>>> Sachin Goel
> >>>>>
> >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >>>>>
> >>>>>>
> >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>>> On my local machine, several flink runtime tests are failing on
> "mvn
> >>>>>> clean
> >>>>>>> verify". Here is the log output:
> >>> http://pastebin.com/raw.php?i=VWbx2ppf
> >>>>>>
> >>>>>> Thanks for reporting this. Have you tried it multiple times? Is it
> >>> failing
> >>>>>> reproducibly with the same tests? What's your setup?
> >>>>>>
> >>>>>> – Ufuk
> >>>
> >
> >
> >
>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
While looking into FLINK-2188 (HBase input) I've discovered that Hadoop input formats implementing Configurable (like mapreduce.TableInputFormat) don't have the Hadoop configuration set via setConf(Configuration).
I have a small fix for this, which I have to clean up. First, I wanted to check what you think about this issue wrt the release. Personally, I think this is a release blocker, because it essentially means that no Hadoop input format, which relies on the Configuration instance to be set this way will work (this is to some extent a bug of the respective input formats) – most notably the HBase TableInputFormat.
– Ufuk
On 09 Jun 2015, at 18:07, Chiwan Park <ch...@icloud.com> wrote:
> I attached jps and jstack log about hanging TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
>
> Regards,
> Chiwan Park
>
>> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <al...@apache.org> wrote:
>>
>> I discovered something that might be a feature, rather than a bug. When you
>> submit an example using the web client without giving parameters the
>> program fails with this:
>>
>> org.apache.flink.client.program.ProgramInvocationException: The main method
>> caused an error.
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>>
>> at org.apache.flink.client.program.Client.run(Client.java:315)
>>
>> at
>> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
>>
>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
>>
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>>
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>>
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
>>
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
>>
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
>>
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
>>
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>>
>> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
>>
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
>>
>> at org.eclipse.jetty.server.Server.handle(Server.java:352)
>>
>> at
>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
>>
>> at
>> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
>>
>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
>>
>> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
>>
>> at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
>>
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
>>
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: java.lang.NullPointerException
>>
>> at
>> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
>>
>> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
>>
>> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
>>
>> at
>> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>>
>> ... 24 more
>>
>>
>> This also only occurs when you uncheck the "suspend execution while showing
>> plan".
>>
>> I think this arises because the new print() uses collect() which tries to
>> get the job execution result. I guess the result is Null since the job is
>> submitted asynchronously when the checkbox is unchecked.
>>
>>
>> Other than that, the new print() is pretty sweet when you run the builtin
>> examples from the CLI. You get all the state changes and also the result,
>> even when running in cluster mode on several task managers. :D
>>
>>
>> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> I discovered another problem:
>>> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
>>> cannot be disabled in part of the Streaming Java API and all of the
>>> Streaming Scala API. I think this is a release blocker (in addition
>>> the the other bugs found so far.)
>>>
>>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>> I found the bug in the failing YARNSessionFIFOITCase: It was comparing
>>>> the hostname to a hostname in some yarn config. In one case it was
>>>> capitalised, in the other case it wasn't.
>>>>
>>>> Pushing fix to master and release-0.9 branch.
>>>>
>>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sa...@gmail.com>
>>> wrote:
>>>>> A re-ran lead to reproducibility of 11 failures again.
>>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
>>>>> managed to succeed in a re-run. Here is the log output again:
>>>>> http://pastebin.com/raw.php?i=N4cm1J18
>>>>>
>>>>> Setup: JDK 1.8.0_40 on windows 8.1
>>>>> System memory: 8GB, quad-core with maximum 8 threads.
>>>>>
>>>>> Regards
>>>>> Sachin Goel
>>>>>
>>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>>
>>>>>>
>>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> On my local machine, several flink runtime tests are failing on "mvn
>>>>>> clean
>>>>>>> verify". Here is the log output:
>>> http://pastebin.com/raw.php?i=VWbx2ppf
>>>>>>
>>>>>> Thanks for reporting this. Have you tried it multiple times? Is it
>>> failing
>>>>>> reproducibly with the same tests? What's your setup?
>>>>>>
>>>>>> – Ufuk
>>>
>
>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Chiwan Park <ch...@icloud.com>.
I attached jps and jstack log about hanging TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183.
Regards,
Chiwan Park
> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <al...@apache.org> wrote:
>
> I discovered something that might be a feature, rather than a bug. When you
> submit an example using the web client without giving parameters the
> program fails with this:
>
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>
> at org.apache.flink.client.program.Client.run(Client.java:315)
>
> at
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
>
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
>
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>
> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
>
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
>
> at org.eclipse.jetty.server.Server.handle(Server.java:352)
>
> at
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
>
> at
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
>
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
>
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
>
> at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
>
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
>
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.NullPointerException
>
> at
> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
>
> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
>
> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
>
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>
> ... 24 more
>
>
> This also only occurs when you uncheck the "suspend execution while showing
> plan".
>
> I think this arises because the new print() uses collect() which tries to
> get the job execution result. I guess the result is Null since the job is
> submitted asynchronously when the checkbox is unchecked.
>
>
> Other than that, the new print() is pretty sweet when you run the builtin
> examples from the CLI. You get all the state changes and also the result,
> even when running in cluster mode on several task managers. :D
>
>
> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> I discovered another problem:
>> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
>> cannot be disabled in part of the Streaming Java API and all of the
>> Streaming Scala API. I think this is a release blocker (in addition
>> the the other bugs found so far.)
>>
>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>> I found the bug in the failing YARNSessionFIFOITCase: It was comparing
>>> the hostname to a hostname in some yarn config. In one case it was
>>> capitalised, in the other case it wasn't.
>>>
>>> Pushing fix to master and release-0.9 branch.
>>>
>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sa...@gmail.com>
>> wrote:
>>>> A re-ran lead to reproducibility of 11 failures again.
>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
>>>> managed to succeed in a re-run. Here is the log output again:
>>>> http://pastebin.com/raw.php?i=N4cm1J18
>>>>
>>>> Setup: JDK 1.8.0_40 on windows 8.1
>>>> System memory: 8GB, quad-core with maximum 8 threads.
>>>>
>>>> Regards
>>>> Sachin Goel
>>>>
>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>>>
>>>>>
>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com>
>> wrote:
>>>>>
>>>>>> On my local machine, several flink runtime tests are failing on "mvn
>>>>> clean
>>>>>> verify". Here is the log output:
>> http://pastebin.com/raw.php?i=VWbx2ppf
>>>>>
>>>>> Thanks for reporting this. Have you tried it multiple times? Is it
>> failing
>>>>> reproducibly with the same tests? What's your setup?
>>>>>
>>>>> – Ufuk
>>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I discovered something that might be a feature, rather than a bug. When you
submit an example using the web client without giving parameters the
program fails with this:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:315)
at
org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:352)
at
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at
org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
... 24 more
This also only occurs when you uncheck the "suspend execution while showing
plan".
I think this arises because the new print() uses collect() which tries to
get the job execution result. I guess the result is Null since the job is
submitted asynchronously when the checkbox is unchecked.
Other than that, the new print() is pretty sweet when you run the builtin
examples from the CLI. You get all the state changes and also the result,
even when running in cluster mode on several task managers. :D
On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I discovered another problem:
> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
> cannot be disabled in part of the Streaming Java API and all of the
> Streaming Scala API. I think this is a release blocker (in addition
> the the other bugs found so far.)
>
> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
> > I found the bug in the failing YARNSessionFIFOITCase: It was comparing
> > the hostname to a hostname in some yarn config. In one case it was
> > capitalised, in the other case it wasn't.
> >
> > Pushing fix to master and release-0.9 branch.
> >
> > On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sa...@gmail.com>
> wrote:
> >> A re-ran lead to reproducibility of 11 failures again.
> >> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
> >> managed to succeed in a re-run. Here is the log output again:
> >> http://pastebin.com/raw.php?i=N4cm1J18
> >>
> >> Setup: JDK 1.8.0_40 on windows 8.1
> >> System memory: 8GB, quad-core with maximum 8 threads.
> >>
> >> Regards
> >> Sachin Goel
> >>
> >> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >>
> >>>
> >>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com>
> wrote:
> >>>
> >>> > On my local machine, several flink runtime tests are failing on "mvn
> >>> clean
> >>> > verify". Here is the log output:
> http://pastebin.com/raw.php?i=VWbx2ppf
> >>>
> >>> Thanks for reporting this. Have you tried it multiple times? Is it
> failing
> >>> reproducibly with the same tests? What's your setup?
> >>>
> >>> – Ufuk
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I discovered another problem:
https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
cannot be disabled in part of the Streaming Java API and all of the
Streaming Scala API. I think this is a release blocker (in addition
the the other bugs found so far.)
On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <al...@apache.org> wrote:
> I found the bug in the failing YARNSessionFIFOITCase: It was comparing
> the hostname to a hostname in some yarn config. In one case it was
> capitalised, in the other case it wasn't.
>
> Pushing fix to master and release-0.9 branch.
>
> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sa...@gmail.com> wrote:
>> A re-ran lead to reproducibility of 11 failures again.
>> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
>> managed to succeed in a re-run. Here is the log output again:
>> http://pastebin.com/raw.php?i=N4cm1J18
>>
>> Setup: JDK 1.8.0_40 on windows 8.1
>> System memory: 8GB, quad-core with maximum 8 threads.
>>
>> Regards
>> Sachin Goel
>>
>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>>
>>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com> wrote:
>>>
>>> > On my local machine, several flink runtime tests are failing on "mvn
>>> clean
>>> > verify". Here is the log output: http://pastebin.com/raw.php?i=VWbx2ppf
>>>
>>> Thanks for reporting this. Have you tried it multiple times? Is it failing
>>> reproducibly with the same tests? What's your setup?
>>>
>>> – Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I found the bug in the failing YARNSessionFIFOITCase: It was comparing
the hostname to a hostname in some yarn config. In one case it was
capitalised, in the other case it wasn't.
Pushing fix to master and release-0.9 branch.
On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sa...@gmail.com> wrote:
> A re-ran lead to reproducibility of 11 failures again.
> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
> managed to succeed in a re-run. Here is the log output again:
> http://pastebin.com/raw.php?i=N4cm1J18
>
> Setup: JDK 1.8.0_40 on windows 8.1
> System memory: 8GB, quad-core with maximum 8 threads.
>
> Regards
> Sachin Goel
>
> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
>>
>> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com> wrote:
>>
>> > On my local machine, several flink runtime tests are failing on "mvn
>> clean
>> > verify". Here is the log output: http://pastebin.com/raw.php?i=VWbx2ppf
>>
>> Thanks for reporting this. Have you tried it multiple times? Is it failing
>> reproducibly with the same tests? What's your setup?
>>
>> – Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Sachin Goel <sa...@gmail.com>.
A re-ran lead to reproducibility of 11 failures again.
TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out but
managed to succeed in a re-run. Here is the log output again:
http://pastebin.com/raw.php?i=N4cm1J18
Setup: JDK 1.8.0_40 on windows 8.1
System memory: 8GB, quad-core with maximum 8 threads.
Regards
Sachin Goel
On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com> wrote:
>
> > On my local machine, several flink runtime tests are failing on "mvn
> clean
> > verify". Here is the log output: http://pastebin.com/raw.php?i=VWbx2ppf
>
> Thanks for reporting this. Have you tried it multiple times? Is it failing
> reproducibly with the same tests? What's your setup?
>
> – Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
On 09 Jun 2015, at 13:58, Sachin Goel <sa...@gmail.com> wrote:
> On my local machine, several flink runtime tests are failing on "mvn clean
> verify". Here is the log output: http://pastebin.com/raw.php?i=VWbx2ppf
Thanks for reporting this. Have you tried it multiple times? Is it failing reproducibly with the same tests? What's your setup?
– Ufuk
Re: Testing Apache Flink 0.9.0-rc1
Posted by Sachin Goel <sa...@gmail.com>.
On my local machine, several flink runtime tests are failing on "mvn clean
verify". Here is the log output: http://pastebin.com/raw.php?i=VWbx2ppf
--
Sachin
On Tue, Jun 9, 2015 at 4:04 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I also saw the same error on my third "mvn clean verify" run. Before it
> always failed in the YARN tests.
>
> On Tue, Jun 9, 2015 at 12:23 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I also encountered a failing TaskManagerFailsWithSlotSharingITCase using
> > Java8. I could, however, not reproduce the error a second time. The stack
> > trace is:
> >
> > The JobManager should handle hard failing task manager with slot
> >
> >
> sharing(org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase)
> > Time elapsed: 1,400.148 sec <<< ERROR!
> > java.util.concurrent.TimeoutException: Futures timed out after [200000
> > milliseconds]
> > at
> > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> > at
> > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
> > at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
> > at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
> > at
> >
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> > at scala.concurrent.Await$.ready(package.scala:86)
> > at
> >
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.shutdown(FlinkMiniCluster.scala:162)
> > at
> >
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:149)
> > at
> >
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply$mcV$sp(TaskManagerFailsWithSlotSharingITCase.scala:140)
> > at
> >
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
> > at
> >
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
> > at
> >
> org.scalatest.Transformer$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> > at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> > at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> > at org.scalatest.Transformer.apply(Transformer.scala:22)
> > at org.scalatest.Transformer.apply(Transformer.scala:20)
> > at org.scalatest.WordSpecLike$anon$1.apply(WordSpecLike.scala:953)
> > at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> > at
> >
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase.withFixture(TaskManagerFailsWithSlotSharingITCase.scala:36)
> >
> > Results :
> >
> > Tests in error:
> >
> >
> TaskManagerFailsWithSlotSharingITCase.run:36->org$scalatest$BeforeAndAfterAll$super$run:36->org$scalatest$WordSpecLike$super$run:36->runTests:36->runTest:36->withFixture:36
> > » Timeout
> >
> > On Tue, Jun 9, 2015 at 11:26 AM Maximilian Michels mxm@apache.org
> > <ht...@apache.org> wrote:
> >
> > The name of the Git branch was not correct. Thank you, Aljoscha, for
> > > noticing. I've changed it from "release-0.9-rc1" to
> "release-0.9.0-rc1".
> > > This has no affect on the validity of the release candidate.
> > >
> >
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I did five "mvn clean verify" runs by now. All of them failed. One
with the TaskmanagerFailsWithSlotSharingITCase and the other ones with
YARNSessionFIFOITCase
On Tue, Jun 9, 2015 at 12:34 PM, Aljoscha Krettek <al...@apache.org> wrote:
> I also saw the same error on my third "mvn clean verify" run. Before it
> always failed in the YARN tests.
>
> On Tue, Jun 9, 2015 at 12:23 PM, Till Rohrmann <tr...@apache.org> wrote:
>>
>> I also encountered a failing TaskManagerFailsWithSlotSharingITCase using
>> Java8. I could, however, not reproduce the error a second time. The stack
>> trace is:
>>
>> The JobManager should handle hard failing task manager with slot
>>
>> sharing(org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase)
>> Time elapsed: 1,400.148 sec <<< ERROR!
>> java.util.concurrent.TimeoutException: Futures timed out after [200000
>> milliseconds]
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
>> at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
>> at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
>> at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.ready(package.scala:86)
>> at
>> org.apache.flink.runtime.minicluster.FlinkMiniCluster.shutdown(FlinkMiniCluster.scala:162)
>> at
>> org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:149)
>> at
>> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply$mcV$sp(TaskManagerFailsWithSlotSharingITCase.scala:140)
>> at
>> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
>> at
>> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
>> at
>> org.scalatest.Transformer$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>> at org.scalatest.Transformer.apply(Transformer.scala:20)
>> at org.scalatest.WordSpecLike$anon$1.apply(WordSpecLike.scala:953)
>> at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
>> at
>> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase.withFixture(TaskManagerFailsWithSlotSharingITCase.scala:36)
>>
>> Results :
>>
>> Tests in error:
>>
>> TaskManagerFailsWithSlotSharingITCase.run:36->org$scalatest$BeforeAndAfterAll$super$run:36->org$scalatest$WordSpecLike$super$run:36->runTests:36->runTest:36->withFixture:36
>> » Timeout
>>
>> On Tue, Jun 9, 2015 at 11:26 AM Maximilian Michels mxm@apache.org
>> <ht...@apache.org> wrote:
>>
>> The name of the Git branch was not correct. Thank you, Aljoscha, for
>> > noticing. I've changed it from "release-0.9-rc1" to "release-0.9.0-rc1".
>> > This has no affect on the validity of the release candidate.
>> >
>>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I also saw the same error on my third "mvn clean verify" run. Before it
always failed in the YARN tests.
On Tue, Jun 9, 2015 at 12:23 PM, Till Rohrmann <tr...@apache.org> wrote:
> I also encountered a failing TaskManagerFailsWithSlotSharingITCase using
> Java8. I could, however, not reproduce the error a second time. The stack
> trace is:
>
> The JobManager should handle hard failing task manager with slot
>
> sharing(org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase)
> Time elapsed: 1,400.148 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Futures timed out after [200000
> milliseconds]
> at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
> at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.shutdown(FlinkMiniCluster.scala:162)
> at
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:149)
> at
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply$mcV$sp(TaskManagerFailsWithSlotSharingITCase.scala:140)
> at
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
> at
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
> at
> org.scalatest.Transformer$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.WordSpecLike$anon$1.apply(WordSpecLike.scala:953)
> at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> at
> org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase.withFixture(TaskManagerFailsWithSlotSharingITCase.scala:36)
>
> Results :
>
> Tests in error:
>
> TaskManagerFailsWithSlotSharingITCase.run:36->org$scalatest$BeforeAndAfterAll$super$run:36->org$scalatest$WordSpecLike$super$run:36->runTests:36->runTest:36->withFixture:36
> » Timeout
>
> On Tue, Jun 9, 2015 at 11:26 AM Maximilian Michels mxm@apache.org
> <ht...@apache.org> wrote:
>
> The name of the Git branch was not correct. Thank you, Aljoscha, for
> > noticing. I've changed it from "release-0.9-rc1" to "release-0.9.0-rc1".
> > This has no affect on the validity of the release candidate.
> >
>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Till Rohrmann <tr...@apache.org>.
I also encountered a failing TaskManagerFailsWithSlotSharingITCase using
Java8. I could, however, not reproduce the error a second time. The stack
trace is:
The JobManager should handle hard failing task manager with slot
sharing(org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase)
Time elapsed: 1,400.148 sec <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [200000
milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
at scala.concurrent.Await$anonfun$ready$1.apply(package.scala:86)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:86)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.shutdown(FlinkMiniCluster.scala:162)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:149)
at org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply$mcV$sp(TaskManagerFailsWithSlotSharingITCase.scala:140)
at org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
at org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase$anonfun$1$anonfun$apply$mcV$sp$3.apply(TaskManagerFailsWithSlotSharingITCase.scala:95)
at org.scalatest.Transformer$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.WordSpecLike$anon$1.apply(WordSpecLike.scala:953)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.apache.flink.runtime.jobmanager.TaskManagerFailsWithSlotSharingITCase.withFixture(TaskManagerFailsWithSlotSharingITCase.scala:36)
Results :
Tests in error:
TaskManagerFailsWithSlotSharingITCase.run:36->org$scalatest$BeforeAndAfterAll$super$run:36->org$scalatest$WordSpecLike$super$run:36->runTests:36->runTest:36->withFixture:36
» Timeout
On Tue, Jun 9, 2015 at 11:26 AM Maximilian Michels mxm@apache.org
<ht...@apache.org> wrote:
The name of the Git branch was not correct. Thank you, Aljoscha, for
> noticing. I've changed it from "release-0.9-rc1" to "release-0.9.0-rc1".
> This has no affect on the validity of the release candidate.
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
The name of the Git branch was not correct. Thank you, Aljoscha, for
noticing. I've changed it from "release-0.9-rc1" to "release-0.9.0-rc1".
This has no affect on the validity of the release candidate.
Re: Testing Apache Flink 0.9.0-rc1
Posted by Maximilian Michels <mx...@apache.org>.
+1 makes sense.
On Tue, Jun 9, 2015 at 10:48 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> I would suggest we use this format to notify others that we did a task:
>
> Assignees:
> - Aljoscha: done
> - Ufuk: found bug in such an such...
> - Chiwan Park: done, ...
>
> The simple "status" doesn't work with multiple people on one task.
>
> On Tue, Jun 9, 2015 at 9:40 AM, Ufuk Celebi <uc...@apache.org> wrote:
> > Hey all,
> >
> > 1. it would be nice if we find more people to also do testing of the
> streaming API. I think it's especially good to have people on it, which did
> not use it before.
> >
> > 2. Just to make sure: the "assignee" field of each task is a list, i.e.
> we can and should have more people testing per task. ;-)
> >
> > – Ufuk
> >
> > On 08 Jun 2015, at 19:00, Chiwan Park <ch...@icloud.com> wrote:
> >
> >> Hi. I’m very excited about preparing a new major release. :)
> >> I just picked two tests. I will report status as soon as possible.
> >>
> >> Regards,
> >> Chiwan Park
> >>
> >>> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
> >>>
> >>> Hi everyone!
> >>>
> >>> As previously discussed, the Flink developer community is very eager
> to get
> >>> out a new major release. Apache Flink 0.9.0 will contain lots of new
> >>> features and many bugfixes. This time, I'll try to coordinate the
> release
> >>> process. Feel free to correct me if I'm doing something wrong because I
> >>> don't no any better :)
> >>>
> >>> To release a great version of Flink to the public, I'd like to ask
> everyone
> >>> to test the release candidate. Recently, Flink has received a lot of
> >>> attention. The expectations are quite high. Only through thorough
> testing
> >>> we will be able to satisfy all the Flink users out there.
> >>>
> >>> Below is a list from the Wiki that we use to ensure the legal and
> >>> functional aspects of a release [1]. What I would like you to do is
> pick at
> >>> least one of the tasks, put your name as assignee in the link below,
> and
> >>> report back once you verified it. That way, I hope we can quickly and
> >>> thoroughly test the release candidate.
> >>>
> >>>
> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
> >>>
> >>> Best,
> >>> Max
> >>>
> >>> Git branch: release-0.9-rc1
> >>> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
> >>> Maven artifacts:
> >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1037/
> >>> PGP public key for verifying the signatures:
> >>> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
> >>>
> >>>
> >>> Legal
> >>> ====
> >>>
> >>> L.1 Check if checksums and GPG files match the corresponding release
> files
> >>>
> >>> L.2 Verify that the source archives do NOT contains any binaries
> >>>
> >>> L.3 Check if the source release is building properly with Maven
> (including
> >>> license header check (default) and checkstyle). Also the tests should
> be
> >>> executed (mvn clean verify)
> >>>
> >>> L.4 Verify that the LICENSE and NOTICE file is correct for the binary
> and
> >>> source release.
> >>>
> >>> L.5 All dependencies must be checked for their license and the license
> must
> >>> be ASL 2.0 compatible (
> http://www.apache.org/legal/resolved.html#category-x)
> >>> * The LICENSE and NOTICE files in the root directory refer to
> dependencies
> >>> in the source release, i.e., files in the git repository (such as
> fonts,
> >>> css, JavaScript, images)
> >>> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer
> to
> >>> the binary distribution and mention all of Flink's Maven dependencies
> as
> >>> well
> >>>
> >>> L.6 Check that all POM files point to the same version (mostly
> relevant to
> >>> examine quickstart artifact files)
> >>>
> >>> L.7 Read the README.md file
> >>>
> >>>
> >>> Functional
> >>> ========
> >>>
> >>> F.1 Run the start-local.sh/start-local-streaming.sh,
> >>> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh
> scripts and
> >>> verify that the processes come up
> >>>
> >>> F.2 Examine the *.out files (should be empty) and the log files (should
> >>> contain no exceptions)
> >>> * Test for Linux, OS X, Windows (for Windows as far as possible, not
> all
> >>> scripts exist)
> >>> * Shutdown and verify there are no exceptions in the log output (after
> >>> shutdown)
> >>> * Check all start+submission scripts for paths with and without spaces
> >>> (./bin/* scripts are quite fragile for paths with spaces)
> >>>
> >>> F.3 local mode (start-local.sh, see criteria below)
> >>> F.4 cluster mode (start-cluster.sh, see criteria below)
> >>> F.5 multi-node cluster (can simulate locally by starting two
> taskmanagers,
> >>> see criteria below)
> >>>
> >>> Criteria for F.3 F.4 F.5
> >>> ----------------------------
> >>> * Verify that the examples are running from both ./bin/flink and from
> the
> >>> web-based job submission tool
> >>> * flink-conf.yml should define more than one task slot
> >>> * Results of job are produced and correct
> >>> ** Check also that the examples are running with the build-in data and
> >>> external sources.
> >>> * Examine the log output - no error messages should be encountered
> >>> ** Web interface shows progress and finished job in history
> >>>
> >>>
> >>> F.6 Test on a cluster with HDFS.
> >>> * Check that a good amount of input splits is read locally (JobManager
> log
> >>> reveals local assignments)
> >>>
> >>> F.7 Test against a Kafka installation
> >>>
> >>> F.8 Test the ./bin/flink command line client
> >>> * Test "info" option, paste the JSON into the plan visualizer HTML
> file,
> >>> check that plan is rendered
> >>> * Test the parallelism flag (-p) to override the configured default
> >>> parallelism
> >>>
> >>> F.9 Verify the plan visualizer with different browsers/operating
> systems
> >>>
> >>> F.10 Verify that the quickstarts for scala and java are working with
> the
> >>> staging repository for both IntelliJ and Eclipse.
> >>> * In particular the dependencies of the quickstart project need to be
> set
> >>> correctly and the QS project needs to build from the staging repository
> >>> (replace the snapshot repo URL with the staging repo URL)
> >>> * The dependency tree of the QuickStart project must not contain any
> >>> dependencies we shade away upstream (guava, netty, ...)
> >>>
> >>> F.11 Run examples on a YARN cluster
> >>>
> >>> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
> >>>
> >>> F.13 Run an example with the RemoteEnvironment against a cluster
> started
> >>> from the shell script
> >>>
> >>> F.14 Run manual Tests in "flink-tests" module.
> >>> * Marked with the @Ignore interface.
> >>>
> >>>
> >>> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
> >>
> >>
> >>
> >>
> >>
> >
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Aljoscha Krettek <al...@apache.org>.
I would suggest we use this format to notify others that we did a task:
Assignees:
- Aljoscha: done
- Ufuk: found bug in such an such...
- Chiwan Park: done, ...
The simple "status" doesn't work with multiple people on one task.
On Tue, Jun 9, 2015 at 9:40 AM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey all,
>
> 1. it would be nice if we find more people to also do testing of the streaming API. I think it's especially good to have people on it, which did not use it before.
>
> 2. Just to make sure: the "assignee" field of each task is a list, i.e. we can and should have more people testing per task. ;-)
>
> – Ufuk
>
> On 08 Jun 2015, at 19:00, Chiwan Park <ch...@icloud.com> wrote:
>
>> Hi. I’m very excited about preparing a new major release. :)
>> I just picked two tests. I will report status as soon as possible.
>>
>> Regards,
>> Chiwan Park
>>
>>> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Hi everyone!
>>>
>>> As previously discussed, the Flink developer community is very eager to get
>>> out a new major release. Apache Flink 0.9.0 will contain lots of new
>>> features and many bugfixes. This time, I'll try to coordinate the release
>>> process. Feel free to correct me if I'm doing something wrong because I
>>> don't no any better :)
>>>
>>> To release a great version of Flink to the public, I'd like to ask everyone
>>> to test the release candidate. Recently, Flink has received a lot of
>>> attention. The expectations are quite high. Only through thorough testing
>>> we will be able to satisfy all the Flink users out there.
>>>
>>> Below is a list from the Wiki that we use to ensure the legal and
>>> functional aspects of a release [1]. What I would like you to do is pick at
>>> least one of the tasks, put your name as assignee in the link below, and
>>> report back once you verified it. That way, I hope we can quickly and
>>> thoroughly test the release candidate.
>>>
>>> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
>>>
>>> Best,
>>> Max
>>>
>>> Git branch: release-0.9-rc1
>>> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
>>> Maven artifacts:
>>> https://repository.apache.org/content/repositories/orgapacheflink-1037/
>>> PGP public key for verifying the signatures:
>>> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
>>>
>>>
>>> Legal
>>> ====
>>>
>>> L.1 Check if checksums and GPG files match the corresponding release files
>>>
>>> L.2 Verify that the source archives do NOT contains any binaries
>>>
>>> L.3 Check if the source release is building properly with Maven (including
>>> license header check (default) and checkstyle). Also the tests should be
>>> executed (mvn clean verify)
>>>
>>> L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
>>> source release.
>>>
>>> L.5 All dependencies must be checked for their license and the license must
>>> be ASL 2.0 compatible (http://www.apache.org/legal/resolved.html#category-x)
>>> * The LICENSE and NOTICE files in the root directory refer to dependencies
>>> in the source release, i.e., files in the git repository (such as fonts,
>>> css, JavaScript, images)
>>> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
>>> the binary distribution and mention all of Flink's Maven dependencies as
>>> well
>>>
>>> L.6 Check that all POM files point to the same version (mostly relevant to
>>> examine quickstart artifact files)
>>>
>>> L.7 Read the README.md file
>>>
>>>
>>> Functional
>>> ========
>>>
>>> F.1 Run the start-local.sh/start-local-streaming.sh,
>>> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts and
>>> verify that the processes come up
>>>
>>> F.2 Examine the *.out files (should be empty) and the log files (should
>>> contain no exceptions)
>>> * Test for Linux, OS X, Windows (for Windows as far as possible, not all
>>> scripts exist)
>>> * Shutdown and verify there are no exceptions in the log output (after
>>> shutdown)
>>> * Check all start+submission scripts for paths with and without spaces
>>> (./bin/* scripts are quite fragile for paths with spaces)
>>>
>>> F.3 local mode (start-local.sh, see criteria below)
>>> F.4 cluster mode (start-cluster.sh, see criteria below)
>>> F.5 multi-node cluster (can simulate locally by starting two taskmanagers,
>>> see criteria below)
>>>
>>> Criteria for F.3 F.4 F.5
>>> ----------------------------
>>> * Verify that the examples are running from both ./bin/flink and from the
>>> web-based job submission tool
>>> * flink-conf.yml should define more than one task slot
>>> * Results of job are produced and correct
>>> ** Check also that the examples are running with the build-in data and
>>> external sources.
>>> * Examine the log output - no error messages should be encountered
>>> ** Web interface shows progress and finished job in history
>>>
>>>
>>> F.6 Test on a cluster with HDFS.
>>> * Check that a good amount of input splits is read locally (JobManager log
>>> reveals local assignments)
>>>
>>> F.7 Test against a Kafka installation
>>>
>>> F.8 Test the ./bin/flink command line client
>>> * Test "info" option, paste the JSON into the plan visualizer HTML file,
>>> check that plan is rendered
>>> * Test the parallelism flag (-p) to override the configured default
>>> parallelism
>>>
>>> F.9 Verify the plan visualizer with different browsers/operating systems
>>>
>>> F.10 Verify that the quickstarts for scala and java are working with the
>>> staging repository for both IntelliJ and Eclipse.
>>> * In particular the dependencies of the quickstart project need to be set
>>> correctly and the QS project needs to build from the staging repository
>>> (replace the snapshot repo URL with the staging repo URL)
>>> * The dependency tree of the QuickStart project must not contain any
>>> dependencies we shade away upstream (guava, netty, ...)
>>>
>>> F.11 Run examples on a YARN cluster
>>>
>>> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
>>>
>>> F.13 Run an example with the RemoteEnvironment against a cluster started
>>> from the shell script
>>>
>>> F.14 Run manual Tests in "flink-tests" module.
>>> * Marked with the @Ignore interface.
>>>
>>>
>>> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
>>
>>
>>
>>
>>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
Hey all,
1. it would be nice if we find more people to also do testing of the streaming API. I think it's especially good to have people on it, which did not use it before.
2. Just to make sure: the "assignee" field of each task is a list, i.e. we can and should have more people testing per task. ;-)
– Ufuk
On 08 Jun 2015, at 19:00, Chiwan Park <ch...@icloud.com> wrote:
> Hi. I’m very excited about preparing a new major release. :)
> I just picked two tests. I will report status as soon as possible.
>
> Regards,
> Chiwan Park
>
>> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi everyone!
>>
>> As previously discussed, the Flink developer community is very eager to get
>> out a new major release. Apache Flink 0.9.0 will contain lots of new
>> features and many bugfixes. This time, I'll try to coordinate the release
>> process. Feel free to correct me if I'm doing something wrong because I
>> don't no any better :)
>>
>> To release a great version of Flink to the public, I'd like to ask everyone
>> to test the release candidate. Recently, Flink has received a lot of
>> attention. The expectations are quite high. Only through thorough testing
>> we will be able to satisfy all the Flink users out there.
>>
>> Below is a list from the Wiki that we use to ensure the legal and
>> functional aspects of a release [1]. What I would like you to do is pick at
>> least one of the tasks, put your name as assignee in the link below, and
>> report back once you verified it. That way, I hope we can quickly and
>> thoroughly test the release candidate.
>>
>> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
>>
>> Best,
>> Max
>>
>> Git branch: release-0.9-rc1
>> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
>> Maven artifacts:
>> https://repository.apache.org/content/repositories/orgapacheflink-1037/
>> PGP public key for verifying the signatures:
>> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
>>
>>
>> Legal
>> ====
>>
>> L.1 Check if checksums and GPG files match the corresponding release files
>>
>> L.2 Verify that the source archives do NOT contains any binaries
>>
>> L.3 Check if the source release is building properly with Maven (including
>> license header check (default) and checkstyle). Also the tests should be
>> executed (mvn clean verify)
>>
>> L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
>> source release.
>>
>> L.5 All dependencies must be checked for their license and the license must
>> be ASL 2.0 compatible (http://www.apache.org/legal/resolved.html#category-x)
>> * The LICENSE and NOTICE files in the root directory refer to dependencies
>> in the source release, i.e., files in the git repository (such as fonts,
>> css, JavaScript, images)
>> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
>> the binary distribution and mention all of Flink's Maven dependencies as
>> well
>>
>> L.6 Check that all POM files point to the same version (mostly relevant to
>> examine quickstart artifact files)
>>
>> L.7 Read the README.md file
>>
>>
>> Functional
>> ========
>>
>> F.1 Run the start-local.sh/start-local-streaming.sh,
>> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts and
>> verify that the processes come up
>>
>> F.2 Examine the *.out files (should be empty) and the log files (should
>> contain no exceptions)
>> * Test for Linux, OS X, Windows (for Windows as far as possible, not all
>> scripts exist)
>> * Shutdown and verify there are no exceptions in the log output (after
>> shutdown)
>> * Check all start+submission scripts for paths with and without spaces
>> (./bin/* scripts are quite fragile for paths with spaces)
>>
>> F.3 local mode (start-local.sh, see criteria below)
>> F.4 cluster mode (start-cluster.sh, see criteria below)
>> F.5 multi-node cluster (can simulate locally by starting two taskmanagers,
>> see criteria below)
>>
>> Criteria for F.3 F.4 F.5
>> ----------------------------
>> * Verify that the examples are running from both ./bin/flink and from the
>> web-based job submission tool
>> * flink-conf.yml should define more than one task slot
>> * Results of job are produced and correct
>> ** Check also that the examples are running with the build-in data and
>> external sources.
>> * Examine the log output - no error messages should be encountered
>> ** Web interface shows progress and finished job in history
>>
>>
>> F.6 Test on a cluster with HDFS.
>> * Check that a good amount of input splits is read locally (JobManager log
>> reveals local assignments)
>>
>> F.7 Test against a Kafka installation
>>
>> F.8 Test the ./bin/flink command line client
>> * Test "info" option, paste the JSON into the plan visualizer HTML file,
>> check that plan is rendered
>> * Test the parallelism flag (-p) to override the configured default
>> parallelism
>>
>> F.9 Verify the plan visualizer with different browsers/operating systems
>>
>> F.10 Verify that the quickstarts for scala and java are working with the
>> staging repository for both IntelliJ and Eclipse.
>> * In particular the dependencies of the quickstart project need to be set
>> correctly and the QS project needs to build from the staging repository
>> (replace the snapshot repo URL with the staging repo URL)
>> * The dependency tree of the QuickStart project must not contain any
>> dependencies we shade away upstream (guava, netty, ...)
>>
>> F.11 Run examples on a YARN cluster
>>
>> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
>>
>> F.13 Run an example with the RemoteEnvironment against a cluster started
>> from the shell script
>>
>> F.14 Run manual Tests in "flink-tests" module.
>> * Marked with the @Ignore interface.
>>
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
>
>
>
>
>
Testing Apache Flink 0.9.0-rc1
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Chiwan!
Is the problem reproducible? Does it always deadlock? Can you please wait
for it to deadlock and then post a stacktrace (jps and jstack) of the
process? Please post it to this issue: FLINK-2183.
Thanks :)
– Ufuk
On Monday, June 8, 2015, Chiwan Park <chiwanpark@icloud.com
<javascript:_e(%7B%7D,'cvml','chiwanpark@icloud.com');>> wrote:
> Hi. I have a problem running `mvn clean verify` command.
> TaskManagerFailsWithSlotSharingITCase hangs in Oracle JDK 7 (1.7.0_80).
> But in Oracle JDK 8 the test case doesn’t hang.
>
> I’ve investigated about this problem but I cannot found the bug.
>
> Regards,
> Chiwan Park
>
> > On Jun 9, 2015, at 2:11 AM, Márton Balassi <ba...@gmail.com>
> wrote:
> >
> > Added F7 Running against Kafka cluster for me in the doc. Doing it
> > tomorrow.
> >
> > On Mon, Jun 8, 2015 at 7:00 PM, Chiwan Park <ch...@icloud.com>
> wrote:
> >
> >> Hi. I’m very excited about preparing a new major release. :)
> >> I just picked two tests. I will report status as soon as possible.
> >>
> >> Regards,
> >> Chiwan Park
> >>
> >>> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
> >>>
> >>> Hi everyone!
> >>>
> >>> As previously discussed, the Flink developer community is very eager to
> >> get
> >>> out a new major release. Apache Flink 0.9.0 will contain lots of new
> >>> features and many bugfixes. This time, I'll try to coordinate the
> release
> >>> process. Feel free to correct me if I'm doing something wrong because I
> >>> don't no any better :)
> >>>
> >>> To release a great version of Flink to the public, I'd like to ask
> >> everyone
> >>> to test the release candidate. Recently, Flink has received a lot of
> >>> attention. The expectations are quite high. Only through thorough
> testing
> >>> we will be able to satisfy all the Flink users out there.
> >>>
> >>> Below is a list from the Wiki that we use to ensure the legal and
> >>> functional aspects of a release [1]. What I would like you to do is
> pick
> >> at
> >>> least one of the tasks, put your name as assignee in the link below,
> and
> >>> report back once you verified it. That way, I hope we can quickly and
> >>> thoroughly test the release candidate.
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
> >>>
> >>> Best,
> >>> Max
> >>>
> >>> Git branch: release-0.9-rc1
> >>> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
> >>> Maven artifacts:
> >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1037/
> >>> PGP public key for verifying the signatures:
> >>> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
> >>>
> >>>
> >>> Legal
> >>> ====
> >>>
> >>> L.1 Check if checksums and GPG files match the corresponding release
> >> files
> >>>
> >>> L.2 Verify that the source archives do NOT contains any binaries
> >>>
> >>> L.3 Check if the source release is building properly with Maven
> >> (including
> >>> license header check (default) and checkstyle). Also the tests should
> be
> >>> executed (mvn clean verify)
> >>>
> >>> L.4 Verify that the LICENSE and NOTICE file is correct for the binary
> and
> >>> source release.
> >>>
> >>> L.5 All dependencies must be checked for their license and the license
> >> must
> >>> be ASL 2.0 compatible (
> >> http://www.apache.org/legal/resolved.html#category-x)
> >>> * The LICENSE and NOTICE files in the root directory refer to
> >> dependencies
> >>> in the source release, i.e., files in the git repository (such as
> fonts,
> >>> css, JavaScript, images)
> >>> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer
> to
> >>> the binary distribution and mention all of Flink's Maven dependencies
> as
> >>> well
> >>>
> >>> L.6 Check that all POM files point to the same version (mostly relevant
> >> to
> >>> examine quickstart artifact files)
> >>>
> >>> L.7 Read the README.md file
> >>>
> >>>
> >>> Functional
> >>> ========
> >>>
> >>> F.1 Run the start-local.sh/start-local-streaming.sh,
> >>> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh
> scripts
> >> and
> >>> verify that the processes come up
> >>>
> >>> F.2 Examine the *.out files (should be empty) and the log files (should
> >>> contain no exceptions)
> >>> * Test for Linux, OS X, Windows (for Windows as far as possible, not
> all
> >>> scripts exist)
> >>> * Shutdown and verify there are no exceptions in the log output (after
> >>> shutdown)
> >>> * Check all start+submission scripts for paths with and without spaces
> >>> (./bin/* scripts are quite fragile for paths with spaces)
> >>>
> >>> F.3 local mode (start-local.sh, see criteria below)
> >>> F.4 cluster mode (start-cluster.sh, see criteria below)
> >>> F.5 multi-node cluster (can simulate locally by starting two
> >> taskmanagers,
> >>> see criteria below)
> >>>
> >>> Criteria for F.3 F.4 F.5
> >>> ----------------------------
> >>> * Verify that the examples are running from both ./bin/flink and from
> the
> >>> web-based job submission tool
> >>> * flink-conf.yml should define more than one task slot
> >>> * Results of job are produced and correct
> >>> ** Check also that the examples are running with the build-in data and
> >>> external sources.
> >>> * Examine the log output - no error messages should be encountered
> >>> ** Web interface shows progress and finished job in history
> >>>
> >>>
> >>> F.6 Test on a cluster with HDFS.
> >>> * Check that a good amount of input splits is read locally (JobManager
> >> log
> >>> reveals local assignments)
> >>>
> >>> F.7 Test against a Kafka installation
> >>>
> >>> F.8 Test the ./bin/flink command line client
> >>> * Test "info" option, paste the JSON into the plan visualizer HTML
> file,
> >>> check that plan is rendered
> >>> * Test the parallelism flag (-p) to override the configured default
> >>> parallelism
> >>>
> >>> F.9 Verify the plan visualizer with different browsers/operating
> systems
> >>>
> >>> F.10 Verify that the quickstarts for scala and java are working with
> the
> >>> staging repository for both IntelliJ and Eclipse.
> >>> * In particular the dependencies of the quickstart project need to be
> set
> >>> correctly and the QS project needs to build from the staging repository
> >>> (replace the snapshot repo URL with the staging repo URL)
> >>> * The dependency tree of the QuickStart project must not contain any
> >>> dependencies we shade away upstream (guava, netty, ...)
> >>>
> >>> F.11 Run examples on a YARN cluster
> >>>
> >>> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
> >>>
> >>> F.13 Run an example with the RemoteEnvironment against a cluster
> started
> >>> from the shell script
> >>>
> >>> F.14 Run manual Tests in "flink-tests" module.
> >>> * Marked with the @Ignore interface.
> >>>
> >>>
> >>> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
> >>
> >>
> >>
> >>
> >>
> >>
>
>
>
>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Chiwan Park <ch...@icloud.com>.
Hi. I have a problem running `mvn clean verify` command.
TaskManagerFailsWithSlotSharingITCase hangs in Oracle JDK 7 (1.7.0_80). But in Oracle JDK 8 the test case doesn’t hang.
I’ve investigated about this problem but I cannot found the bug.
Regards,
Chiwan Park
> On Jun 9, 2015, at 2:11 AM, Márton Balassi <ba...@gmail.com> wrote:
>
> Added F7 Running against Kafka cluster for me in the doc. Doing it
> tomorrow.
>
> On Mon, Jun 8, 2015 at 7:00 PM, Chiwan Park <ch...@icloud.com> wrote:
>
>> Hi. I’m very excited about preparing a new major release. :)
>> I just picked two tests. I will report status as soon as possible.
>>
>> Regards,
>> Chiwan Park
>>
>>> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Hi everyone!
>>>
>>> As previously discussed, the Flink developer community is very eager to
>> get
>>> out a new major release. Apache Flink 0.9.0 will contain lots of new
>>> features and many bugfixes. This time, I'll try to coordinate the release
>>> process. Feel free to correct me if I'm doing something wrong because I
>>> don't no any better :)
>>>
>>> To release a great version of Flink to the public, I'd like to ask
>> everyone
>>> to test the release candidate. Recently, Flink has received a lot of
>>> attention. The expectations are quite high. Only through thorough testing
>>> we will be able to satisfy all the Flink users out there.
>>>
>>> Below is a list from the Wiki that we use to ensure the legal and
>>> functional aspects of a release [1]. What I would like you to do is pick
>> at
>>> least one of the tasks, put your name as assignee in the link below, and
>>> report back once you verified it. That way, I hope we can quickly and
>>> thoroughly test the release candidate.
>>>
>>>
>> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
>>>
>>> Best,
>>> Max
>>>
>>> Git branch: release-0.9-rc1
>>> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
>>> Maven artifacts:
>>> https://repository.apache.org/content/repositories/orgapacheflink-1037/
>>> PGP public key for verifying the signatures:
>>> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
>>>
>>>
>>> Legal
>>> ====
>>>
>>> L.1 Check if checksums and GPG files match the corresponding release
>> files
>>>
>>> L.2 Verify that the source archives do NOT contains any binaries
>>>
>>> L.3 Check if the source release is building properly with Maven
>> (including
>>> license header check (default) and checkstyle). Also the tests should be
>>> executed (mvn clean verify)
>>>
>>> L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
>>> source release.
>>>
>>> L.5 All dependencies must be checked for their license and the license
>> must
>>> be ASL 2.0 compatible (
>> http://www.apache.org/legal/resolved.html#category-x)
>>> * The LICENSE and NOTICE files in the root directory refer to
>> dependencies
>>> in the source release, i.e., files in the git repository (such as fonts,
>>> css, JavaScript, images)
>>> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
>>> the binary distribution and mention all of Flink's Maven dependencies as
>>> well
>>>
>>> L.6 Check that all POM files point to the same version (mostly relevant
>> to
>>> examine quickstart artifact files)
>>>
>>> L.7 Read the README.md file
>>>
>>>
>>> Functional
>>> ========
>>>
>>> F.1 Run the start-local.sh/start-local-streaming.sh,
>>> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts
>> and
>>> verify that the processes come up
>>>
>>> F.2 Examine the *.out files (should be empty) and the log files (should
>>> contain no exceptions)
>>> * Test for Linux, OS X, Windows (for Windows as far as possible, not all
>>> scripts exist)
>>> * Shutdown and verify there are no exceptions in the log output (after
>>> shutdown)
>>> * Check all start+submission scripts for paths with and without spaces
>>> (./bin/* scripts are quite fragile for paths with spaces)
>>>
>>> F.3 local mode (start-local.sh, see criteria below)
>>> F.4 cluster mode (start-cluster.sh, see criteria below)
>>> F.5 multi-node cluster (can simulate locally by starting two
>> taskmanagers,
>>> see criteria below)
>>>
>>> Criteria for F.3 F.4 F.5
>>> ----------------------------
>>> * Verify that the examples are running from both ./bin/flink and from the
>>> web-based job submission tool
>>> * flink-conf.yml should define more than one task slot
>>> * Results of job are produced and correct
>>> ** Check also that the examples are running with the build-in data and
>>> external sources.
>>> * Examine the log output - no error messages should be encountered
>>> ** Web interface shows progress and finished job in history
>>>
>>>
>>> F.6 Test on a cluster with HDFS.
>>> * Check that a good amount of input splits is read locally (JobManager
>> log
>>> reveals local assignments)
>>>
>>> F.7 Test against a Kafka installation
>>>
>>> F.8 Test the ./bin/flink command line client
>>> * Test "info" option, paste the JSON into the plan visualizer HTML file,
>>> check that plan is rendered
>>> * Test the parallelism flag (-p) to override the configured default
>>> parallelism
>>>
>>> F.9 Verify the plan visualizer with different browsers/operating systems
>>>
>>> F.10 Verify that the quickstarts for scala and java are working with the
>>> staging repository for both IntelliJ and Eclipse.
>>> * In particular the dependencies of the quickstart project need to be set
>>> correctly and the QS project needs to build from the staging repository
>>> (replace the snapshot repo URL with the staging repo URL)
>>> * The dependency tree of the QuickStart project must not contain any
>>> dependencies we shade away upstream (guava, netty, ...)
>>>
>>> F.11 Run examples on a YARN cluster
>>>
>>> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
>>>
>>> F.13 Run an example with the RemoteEnvironment against a cluster started
>>> from the shell script
>>>
>>> F.14 Run manual Tests in "flink-tests" module.
>>> * Marked with the @Ignore interface.
>>>
>>>
>>> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
>>
>>
>>
>>
>>
>>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Márton Balassi <ba...@gmail.com>.
Added F7 Running against Kafka cluster for me in the doc. Doing it
tomorrow.
On Mon, Jun 8, 2015 at 7:00 PM, Chiwan Park <ch...@icloud.com> wrote:
> Hi. I’m very excited about preparing a new major release. :)
> I just picked two tests. I will report status as soon as possible.
>
> Regards,
> Chiwan Park
>
> > On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
> >
> > Hi everyone!
> >
> > As previously discussed, the Flink developer community is very eager to
> get
> > out a new major release. Apache Flink 0.9.0 will contain lots of new
> > features and many bugfixes. This time, I'll try to coordinate the release
> > process. Feel free to correct me if I'm doing something wrong because I
> > don't no any better :)
> >
> > To release a great version of Flink to the public, I'd like to ask
> everyone
> > to test the release candidate. Recently, Flink has received a lot of
> > attention. The expectations are quite high. Only through thorough testing
> > we will be able to satisfy all the Flink users out there.
> >
> > Below is a list from the Wiki that we use to ensure the legal and
> > functional aspects of a release [1]. What I would like you to do is pick
> at
> > least one of the tasks, put your name as assignee in the link below, and
> > report back once you verified it. That way, I hope we can quickly and
> > thoroughly test the release candidate.
> >
> >
> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
> >
> > Best,
> > Max
> >
> > Git branch: release-0.9-rc1
> > Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
> > Maven artifacts:
> > https://repository.apache.org/content/repositories/orgapacheflink-1037/
> > PGP public key for verifying the signatures:
> > http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
> >
> >
> > Legal
> > ====
> >
> > L.1 Check if checksums and GPG files match the corresponding release
> files
> >
> > L.2 Verify that the source archives do NOT contains any binaries
> >
> > L.3 Check if the source release is building properly with Maven
> (including
> > license header check (default) and checkstyle). Also the tests should be
> > executed (mvn clean verify)
> >
> > L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
> > source release.
> >
> > L.5 All dependencies must be checked for their license and the license
> must
> > be ASL 2.0 compatible (
> http://www.apache.org/legal/resolved.html#category-x)
> > * The LICENSE and NOTICE files in the root directory refer to
> dependencies
> > in the source release, i.e., files in the git repository (such as fonts,
> > css, JavaScript, images)
> > * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
> > the binary distribution and mention all of Flink's Maven dependencies as
> > well
> >
> > L.6 Check that all POM files point to the same version (mostly relevant
> to
> > examine quickstart artifact files)
> >
> > L.7 Read the README.md file
> >
> >
> > Functional
> > ========
> >
> > F.1 Run the start-local.sh/start-local-streaming.sh,
> > start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts
> and
> > verify that the processes come up
> >
> > F.2 Examine the *.out files (should be empty) and the log files (should
> > contain no exceptions)
> > * Test for Linux, OS X, Windows (for Windows as far as possible, not all
> > scripts exist)
> > * Shutdown and verify there are no exceptions in the log output (after
> > shutdown)
> > * Check all start+submission scripts for paths with and without spaces
> > (./bin/* scripts are quite fragile for paths with spaces)
> >
> > F.3 local mode (start-local.sh, see criteria below)
> > F.4 cluster mode (start-cluster.sh, see criteria below)
> > F.5 multi-node cluster (can simulate locally by starting two
> taskmanagers,
> > see criteria below)
> >
> > Criteria for F.3 F.4 F.5
> > ----------------------------
> > * Verify that the examples are running from both ./bin/flink and from the
> > web-based job submission tool
> > * flink-conf.yml should define more than one task slot
> > * Results of job are produced and correct
> > ** Check also that the examples are running with the build-in data and
> > external sources.
> > * Examine the log output - no error messages should be encountered
> > ** Web interface shows progress and finished job in history
> >
> >
> > F.6 Test on a cluster with HDFS.
> > * Check that a good amount of input splits is read locally (JobManager
> log
> > reveals local assignments)
> >
> > F.7 Test against a Kafka installation
> >
> > F.8 Test the ./bin/flink command line client
> > * Test "info" option, paste the JSON into the plan visualizer HTML file,
> > check that plan is rendered
> > * Test the parallelism flag (-p) to override the configured default
> > parallelism
> >
> > F.9 Verify the plan visualizer with different browsers/operating systems
> >
> > F.10 Verify that the quickstarts for scala and java are working with the
> > staging repository for both IntelliJ and Eclipse.
> > * In particular the dependencies of the quickstart project need to be set
> > correctly and the QS project needs to build from the staging repository
> > (replace the snapshot repo URL with the staging repo URL)
> > * The dependency tree of the QuickStart project must not contain any
> > dependencies we shade away upstream (guava, netty, ...)
> >
> > F.11 Run examples on a YARN cluster
> >
> > F.12 Run all examples from the IDE (Eclipse & IntelliJ)
> >
> > F.13 Run an example with the RemoteEnvironment against a cluster started
> > from the shell script
> >
> > F.14 Run manual Tests in "flink-tests" module.
> > * Marked with the @Ignore interface.
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing
>
>
>
>
>
>
Re: Testing Apache Flink 0.9.0-rc1
Posted by Chiwan Park <ch...@icloud.com>.
Hi. I’m very excited about preparing a new major release. :)
I just picked two tests. I will report status as soon as possible.
Regards,
Chiwan Park
> On Jun 9, 2015, at 1:52 AM, Maximilian Michels <mx...@apache.org> wrote:
>
> Hi everyone!
>
> As previously discussed, the Flink developer community is very eager to get
> out a new major release. Apache Flink 0.9.0 will contain lots of new
> features and many bugfixes. This time, I'll try to coordinate the release
> process. Feel free to correct me if I'm doing something wrong because I
> don't no any better :)
>
> To release a great version of Flink to the public, I'd like to ask everyone
> to test the release candidate. Recently, Flink has received a lot of
> attention. The expectations are quite high. Only through thorough testing
> we will be able to satisfy all the Flink users out there.
>
> Below is a list from the Wiki that we use to ensure the legal and
> functional aspects of a release [1]. What I would like you to do is pick at
> least one of the tasks, put your name as assignee in the link below, and
> report back once you verified it. That way, I hope we can quickly and
> thoroughly test the release candidate.
>
> https://docs.google.com/document/d/1BhyMPTpAUYA8dG8-vJ3gSAmBUAa0PBSRkxIBPsZxkLs/edit
>
> Best,
> Max
>
> Git branch: release-0.9-rc1
> Release binaries: http://people.apache.org/~mxm/flink-0.9.0-rc1/
> Maven artifacts:
> https://repository.apache.org/content/repositories/orgapacheflink-1037/
> PGP public key for verifying the signatures:
> http://pgp.mit.edu/pks/lookup?op=vindex&search=0xDE976D18C2909CBF
>
>
> Legal
> ====
>
> L.1 Check if checksums and GPG files match the corresponding release files
>
> L.2 Verify that the source archives do NOT contains any binaries
>
> L.3 Check if the source release is building properly with Maven (including
> license header check (default) and checkstyle). Also the tests should be
> executed (mvn clean verify)
>
> L.4 Verify that the LICENSE and NOTICE file is correct for the binary and
> source release.
>
> L.5 All dependencies must be checked for their license and the license must
> be ASL 2.0 compatible (http://www.apache.org/legal/resolved.html#category-x)
> * The LICENSE and NOTICE files in the root directory refer to dependencies
> in the source release, i.e., files in the git repository (such as fonts,
> css, JavaScript, images)
> * The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to
> the binary distribution and mention all of Flink's Maven dependencies as
> well
>
> L.6 Check that all POM files point to the same version (mostly relevant to
> examine quickstart artifact files)
>
> L.7 Read the README.md file
>
>
> Functional
> ========
>
> F.1 Run the start-local.sh/start-local-streaming.sh,
> start-cluster.sh/start-cluster-streaming.sh, start-webclient.sh scripts and
> verify that the processes come up
>
> F.2 Examine the *.out files (should be empty) and the log files (should
> contain no exceptions)
> * Test for Linux, OS X, Windows (for Windows as far as possible, not all
> scripts exist)
> * Shutdown and verify there are no exceptions in the log output (after
> shutdown)
> * Check all start+submission scripts for paths with and without spaces
> (./bin/* scripts are quite fragile for paths with spaces)
>
> F.3 local mode (start-local.sh, see criteria below)
> F.4 cluster mode (start-cluster.sh, see criteria below)
> F.5 multi-node cluster (can simulate locally by starting two taskmanagers,
> see criteria below)
>
> Criteria for F.3 F.4 F.5
> ----------------------------
> * Verify that the examples are running from both ./bin/flink and from the
> web-based job submission tool
> * flink-conf.yml should define more than one task slot
> * Results of job are produced and correct
> ** Check also that the examples are running with the build-in data and
> external sources.
> * Examine the log output - no error messages should be encountered
> ** Web interface shows progress and finished job in history
>
>
> F.6 Test on a cluster with HDFS.
> * Check that a good amount of input splits is read locally (JobManager log
> reveals local assignments)
>
> F.7 Test against a Kafka installation
>
> F.8 Test the ./bin/flink command line client
> * Test "info" option, paste the JSON into the plan visualizer HTML file,
> check that plan is rendered
> * Test the parallelism flag (-p) to override the configured default
> parallelism
>
> F.9 Verify the plan visualizer with different browsers/operating systems
>
> F.10 Verify that the quickstarts for scala and java are working with the
> staging repository for both IntelliJ and Eclipse.
> * In particular the dependencies of the quickstart project need to be set
> correctly and the QS project needs to build from the staging repository
> (replace the snapshot repo URL with the staging repo URL)
> * The dependency tree of the QuickStart project must not contain any
> dependencies we shade away upstream (guava, netty, ...)
>
> F.11 Run examples on a YARN cluster
>
> F.12 Run all examples from the IDE (Eclipse & IntelliJ)
>
> F.13 Run an example with the RemoteEnvironment against a cluster started
> from the shell script
>
> F.14 Run manual Tests in "flink-tests" module.
> * Marked with the @Ignore interface.
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/Releasing