You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Márton Balassi <ba...@gmail.com> on 2016/02/25 15:36:59 UTC
Inconvenient (unforeseen?) consequences of PR #1683
Recent changes to the build [1] where many libraries got their core
dependencies (the ones included in the flink-dist fat jar) moved to the
provided scope.
The reasoning was that when submitting to the Flink cluster the application
already has these dependencies, while when a user writes a program against
these libraries she will include the core dependencies explicitly anyway.
There is one other case of usage however, namely when someone is trying to
run an application defined in these libraries depending on the core jars.
To give an example if you were to run the Gelly ConnectedComponents example
[2] from an IDE after importing Flink (or running with java -jar without
including the flink fat jar in the classpath) you would receive the
following class not found exception as per the current master:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/flink/api/common/ProgramDescription
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
Where the class missing in runtime is in flink-core, a transitive
dependency of the jars moved to the provided scope.
Funny thing is we have tests in place to run our examples, but those add
test scope dependencies re-adding the missing classes, so it is never
discovered.
I agree with the original purpose of PR #1683, but also think that the
current state makes for very inconvenient user experience.
I would like to open a discussion on how and when to resolve the issue
given the release of 1.0.0.
1. Is it a release blocker?
2. Should the change be reverted or is it sufficient to have proper
documentation around it? Maybe a maven profile for explicitly for
developing Flink without the provided scope?
Note that the issue was originally reported by Gábor Gévay.
[1] https://github.com/apache/flink/pull/1683
[2]
https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
Best,
Marton
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Vasiliki Kalavri <va...@gmail.com>.
/me likes this idea :)
On 29 February 2016 at 12:38, Stephan Ewen <se...@apache.org> wrote:
> How about adding to "flink-gelly-examples" that it packages the examples in
> JAR files. We can then add them even to the "examples/gelly" folder in the
> binary distribution.
>
> On Mon, Feb 29, 2016 at 12:24 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com> wrote:
>
> > In my opinion, the fat jar solution is easier than having to copy the
> Gelly
> > jar to all task managers.
> > I would be in favor of including flink-gelly in the flink-examples
> module,
> > but we can also simply document the current behavior nicely.
> >
> > -V.
> >
> > On 29 February 2016 at 12:15, Till Rohrmann <tr...@apache.org>
> wrote:
> >
> > > Good catch :-) I mean we could also change the behaviour to include
> > > flink-gelly in the flink-gelly-examples module.
> > >
> > > On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
> > > vasilikikalavri@gmail.com> wrote:
> > >
> > > > Thanks Till! Then, we'd better update the docs, too. Currently both
> > links
> > > > to examples and cluster execution instructions are broken.
> > > > I'll create an issue.
> > > >
> > > > -V.
> > > >
> > > > On 29 February 2016 at 11:54, Till Rohrmann <tr...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Vasia,
> > > > >
> > > > > that is because you're missing the flink-gelly dependency. If you
> > just
> > > > > build flink-gelly-examples, then it won't contain flink-gelly
> because
> > > it
> > > > is
> > > > > not a fat jar. You either have to install flink-gelly on your
> cluster
> > > or
> > > > > package it in the final user jar.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > > > > vasilikikalavri@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi squirrels,
> > > > > >
> > > > > > sorry I've been slow to respond to this, but I'm now testing RC1
> > and
> > > > I'm
> > > > > a
> > > > > > bit confused with this change.
> > > > > >
> > > > > > So far, the easier way to run a Gelly example on a cluster was to
> > > > package
> > > > > > and submit the Gelly jar.
> > > > > > Now, since the flink-gelly project doesn't contain the examples
> > > > anymore,
> > > > > I
> > > > > > tried running a Gelly example by the packaging
> flink-gelly-examples
> > > jar
> > > > > > (mvn package). However, this gives me a ClassNotFoundException.
> > > > > > e.g. for SingleSourceShortestPaths, the following:
> > > > > >
> > > > > > java.lang.RuntimeException: Could not look up the main(String[])
> > > method
> > > > > > from the class
> > > > org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > > > > > at
> > > >
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > > > > at
> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > > > > Caused by: java.lang.NoClassDefFoundError:
> > > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > > > > at java.lang.Class.getMethod0(Class.java:2774)
> > > > > > at java.lang.Class.getMethod(Class.java:1663)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > > > > ... 5 more
> > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > org.apache.flink.graph.spargel.MessagingFunction
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > > > > ... 10 more
> > > > > >
> > > > > >
> > > > > > What am I missing here?
> > > > > >
> > > > > > Thanks!
> > > > > > -Vasia.
> > > > > >
> > > > > > On 26 February 2016 at 14:10, Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks, +1.
> > > > > > >
> > > > > > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <
> sewen@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I think it is a release blocker.
> > > > > > > >
> > > > > > > > That was a change to make it possible to use Flink with SBT
> > (and
> > > > make
> > > > > > it
> > > > > > > > easier to use it with Maven).
> > > > > > > > Just had an offline chat with Till, and we suggest the
> > following
> > > > > > > solution:
> > > > > > > >
> > > > > > > > All libraries should be split into two projects:
> > > > > > > > - library
> > > > > > > > - library-examples
> > > > > > > >
> > > > > > > > The "library" project has the core flink dependencies as
> > > "provided"
> > > > > > > > The "library-examples" project has both the "library" and the
> > > core
> > > > > > flink
> > > > > > > > dependencies with scope "compile"
> > > > > > > >
> > > > > > > > That way the example should run in the IDE out of the cox,
> and
> > > > users
> > > > > > that
> > > > > > > > reference the libraries will still get the correct packaging
> > > > (include
> > > > > > the
> > > > > > > > library in the user jar, but not additionally the core flink
> > > jars).
> > > > > > > >
> > > > > > > > Greetings,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > > > > > balassi.marton@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Issued JIRA ticket 3511 to make it referable in other
> > > > discussions.
> > > > > > [1]
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > > > > > >
> > > > > > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > > > > > > balassi.marton@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Recent changes to the build [1] where many libraries got
> > > their
> > > > > core
> > > > > > > > > > dependencies (the ones included in the flink-dist fat
> jar)
> > > > moved
> > > > > to
> > > > > > > the
> > > > > > > > > > provided scope.
> > > > > > > > > >
> > > > > > > > > > The reasoning was that when submitting to the Flink
> cluster
> > > the
> > > > > > > > > > application already has these dependencies, while when a
> > user
> > > > > > writes
> > > > > > > a
> > > > > > > > > > program against these libraries she will include the core
> > > > > > > dependencies
> > > > > > > > > > explicitly anyway.
> > > > > > > > > >
> > > > > > > > > > There is one other case of usage however, namely when
> > someone
> > > > is
> > > > > > > trying
> > > > > > > > > to
> > > > > > > > > > run an application defined in these libraries depending
> on
> > > the
> > > > > core
> > > > > > > > jars.
> > > > > > > > > > To give an example if you were to run the Gelly
> > > > > ConnectedComponents
> > > > > > > > > example
> > > > > > > > > > [2] from an IDE after importing Flink (or running with
> java
> > > > -jar
> > > > > > > > without
> > > > > > > > > > including the flink fat jar in the classpath) you would
> > > receive
> > > > > the
> > > > > > > > > > following class not found exception as per the current
> > > master:
> > > > > > > > > >
> > > > > > > > > > Exception in thread "main"
> java.lang.NoClassDefFoundError:
> > > > > > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > > > > > > at
> java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > > > > > at
> > > java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > > > > > at
> > java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > > > > > at java.security.AccessController.doPrivileged(Native
> > Method)
> > > > > > > > > > at
> > java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > > > > at
> > > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > > > > at
> > > > > > >
> > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > > > > > >
> > > > > > > > > > Where the class missing in runtime is in flink-core, a
> > > > transitive
> > > > > > > > > > dependency of the jars moved to the provided scope.
> > > > > > > > > >
> > > > > > > > > > Funny thing is we have tests in place to run our
> examples,
> > > but
> > > > > > those
> > > > > > > > add
> > > > > > > > > > test scope dependencies re-adding the missing classes, so
> > it
> > > is
> > > > > > never
> > > > > > > > > > discovered.
> > > > > > > > > >
> > > > > > > > > > I agree with the original purpose of PR #1683, but also
> > think
> > > > > that
> > > > > > > the
> > > > > > > > > > current state makes for very inconvenient user
> experience.
> > > > > > > > > >
> > > > > > > > > > I would like to open a discussion on how and when to
> > resolve
> > > > the
> > > > > > > issue
> > > > > > > > > > given the release of 1.0.0.
> > > > > > > > > >
> > > > > > > > > > 1. Is it a release blocker?
> > > > > > > > > > 2. Should the change be reverted or is it sufficient to
> > have
> > > > > proper
> > > > > > > > > > documentation around it? Maybe a maven profile for
> > explicitly
> > > > for
> > > > > > > > > > developing Flink without the provided scope?
> > > > > > > > > >
> > > > > > > > > > Note that the issue was originally reported by Gábor
> Gévay.
> > > > > > > > > >
> > > > > > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > > > > > [2]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Marton
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Stephan Ewen <se...@apache.org>.
How about adding to "flink-gelly-examples" that it packages the examples in
JAR files. We can then add them even to the "examples/gelly" folder in the
binary distribution.
On Mon, Feb 29, 2016 at 12:24 PM, Vasiliki Kalavri <
vasilikikalavri@gmail.com> wrote:
> In my opinion, the fat jar solution is easier than having to copy the Gelly
> jar to all task managers.
> I would be in favor of including flink-gelly in the flink-examples module,
> but we can also simply document the current behavior nicely.
>
> -V.
>
> On 29 February 2016 at 12:15, Till Rohrmann <tr...@apache.org> wrote:
>
> > Good catch :-) I mean we could also change the behaviour to include
> > flink-gelly in the flink-gelly-examples module.
> >
> > On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
> > vasilikikalavri@gmail.com> wrote:
> >
> > > Thanks Till! Then, we'd better update the docs, too. Currently both
> links
> > > to examples and cluster execution instructions are broken.
> > > I'll create an issue.
> > >
> > > -V.
> > >
> > > On 29 February 2016 at 11:54, Till Rohrmann <tr...@apache.org>
> > wrote:
> > >
> > > > Hi Vasia,
> > > >
> > > > that is because you're missing the flink-gelly dependency. If you
> just
> > > > build flink-gelly-examples, then it won't contain flink-gelly because
> > it
> > > is
> > > > not a fat jar. You either have to install flink-gelly on your cluster
> > or
> > > > package it in the final user jar.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > > > vasilikikalavri@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi squirrels,
> > > > >
> > > > > sorry I've been slow to respond to this, but I'm now testing RC1
> and
> > > I'm
> > > > a
> > > > > bit confused with this change.
> > > > >
> > > > > So far, the easier way to run a Gelly example on a cluster was to
> > > package
> > > > > and submit the Gelly jar.
> > > > > Now, since the flink-gelly project doesn't contain the examples
> > > anymore,
> > > > I
> > > > > tried running a Gelly example by the packaging flink-gelly-examples
> > jar
> > > > > (mvn package). However, this gives me a ClassNotFoundException.
> > > > > e.g. for SingleSourceShortestPaths, the following:
> > > > >
> > > > > java.lang.RuntimeException: Could not look up the main(String[])
> > method
> > > > > from the class
> > > org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > > > > at
> > > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > > > Caused by: java.lang.NoClassDefFoundError:
> > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > > > at java.lang.Class.getMethod0(Class.java:2774)
> > > > > at java.lang.Class.getMethod(Class.java:1663)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > > > ... 5 more
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > org.apache.flink.graph.spargel.MessagingFunction
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > > > ... 10 more
> > > > >
> > > > >
> > > > > What am I missing here?
> > > > >
> > > > > Thanks!
> > > > > -Vasia.
> > > > >
> > > > > On 26 February 2016 at 14:10, Márton Balassi <
> > balassi.marton@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks, +1.
> > > > > >
> > > > > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi!
> > > > > > >
> > > > > > > I think it is a release blocker.
> > > > > > >
> > > > > > > That was a change to make it possible to use Flink with SBT
> (and
> > > make
> > > > > it
> > > > > > > easier to use it with Maven).
> > > > > > > Just had an offline chat with Till, and we suggest the
> following
> > > > > > solution:
> > > > > > >
> > > > > > > All libraries should be split into two projects:
> > > > > > > - library
> > > > > > > - library-examples
> > > > > > >
> > > > > > > The "library" project has the core flink dependencies as
> > "provided"
> > > > > > > The "library-examples" project has both the "library" and the
> > core
> > > > > flink
> > > > > > > dependencies with scope "compile"
> > > > > > >
> > > > > > > That way the example should run in the IDE out of the cox, and
> > > users
> > > > > that
> > > > > > > reference the libraries will still get the correct packaging
> > > (include
> > > > > the
> > > > > > > library in the user jar, but not additionally the core flink
> > jars).
> > > > > > >
> > > > > > > Greetings,
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > > > > balassi.marton@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Issued JIRA ticket 3511 to make it referable in other
> > > discussions.
> > > > > [1]
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > > > > >
> > > > > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > > > > > balassi.marton@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Recent changes to the build [1] where many libraries got
> > their
> > > > core
> > > > > > > > > dependencies (the ones included in the flink-dist fat jar)
> > > moved
> > > > to
> > > > > > the
> > > > > > > > > provided scope.
> > > > > > > > >
> > > > > > > > > The reasoning was that when submitting to the Flink cluster
> > the
> > > > > > > > > application already has these dependencies, while when a
> user
> > > > > writes
> > > > > > a
> > > > > > > > > program against these libraries she will include the core
> > > > > > dependencies
> > > > > > > > > explicitly anyway.
> > > > > > > > >
> > > > > > > > > There is one other case of usage however, namely when
> someone
> > > is
> > > > > > trying
> > > > > > > > to
> > > > > > > > > run an application defined in these libraries depending on
> > the
> > > > core
> > > > > > > jars.
> > > > > > > > > To give an example if you were to run the Gelly
> > > > ConnectedComponents
> > > > > > > > example
> > > > > > > > > [2] from an IDE after importing Flink (or running with java
> > > -jar
> > > > > > > without
> > > > > > > > > including the flink fat jar in the classpath) you would
> > receive
> > > > the
> > > > > > > > > following class not found exception as per the current
> > master:
> > > > > > > > >
> > > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > > > > at
> > > > > > > >
> > > > >
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > > > > at
> > java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > > > > at
> java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > > > > at java.security.AccessController.doPrivileged(Native
> Method)
> > > > > > > > > at
> java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > > > at
> > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > > > at
> > > > > >
> > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > > > > >
> > > > > > > > > Where the class missing in runtime is in flink-core, a
> > > transitive
> > > > > > > > > dependency of the jars moved to the provided scope.
> > > > > > > > >
> > > > > > > > > Funny thing is we have tests in place to run our examples,
> > but
> > > > > those
> > > > > > > add
> > > > > > > > > test scope dependencies re-adding the missing classes, so
> it
> > is
> > > > > never
> > > > > > > > > discovered.
> > > > > > > > >
> > > > > > > > > I agree with the original purpose of PR #1683, but also
> think
> > > > that
> > > > > > the
> > > > > > > > > current state makes for very inconvenient user experience.
> > > > > > > > >
> > > > > > > > > I would like to open a discussion on how and when to
> resolve
> > > the
> > > > > > issue
> > > > > > > > > given the release of 1.0.0.
> > > > > > > > >
> > > > > > > > > 1. Is it a release blocker?
> > > > > > > > > 2. Should the change be reverted or is it sufficient to
> have
> > > > proper
> > > > > > > > > documentation around it? Maybe a maven profile for
> explicitly
> > > for
> > > > > > > > > developing Flink without the provided scope?
> > > > > > > > >
> > > > > > > > > Note that the issue was originally reported by Gábor Gévay.
> > > > > > > > >
> > > > > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Marton
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Vasiliki Kalavri <va...@gmail.com>.
In my opinion, the fat jar solution is easier than having to copy the Gelly
jar to all task managers.
I would be in favor of including flink-gelly in the flink-examples module,
but we can also simply document the current behavior nicely.
-V.
On 29 February 2016 at 12:15, Till Rohrmann <tr...@apache.org> wrote:
> Good catch :-) I mean we could also change the behaviour to include
> flink-gelly in the flink-gelly-examples module.
>
> On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com> wrote:
>
> > Thanks Till! Then, we'd better update the docs, too. Currently both links
> > to examples and cluster execution instructions are broken.
> > I'll create an issue.
> >
> > -V.
> >
> > On 29 February 2016 at 11:54, Till Rohrmann <tr...@apache.org>
> wrote:
> >
> > > Hi Vasia,
> > >
> > > that is because you're missing the flink-gelly dependency. If you just
> > > build flink-gelly-examples, then it won't contain flink-gelly because
> it
> > is
> > > not a fat jar. You either have to install flink-gelly on your cluster
> or
> > > package it in the final user jar.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > > vasilikikalavri@gmail.com
> > > > wrote:
> > >
> > > > Hi squirrels,
> > > >
> > > > sorry I've been slow to respond to this, but I'm now testing RC1 and
> > I'm
> > > a
> > > > bit confused with this change.
> > > >
> > > > So far, the easier way to run a Gelly example on a cluster was to
> > package
> > > > and submit the Gelly jar.
> > > > Now, since the flink-gelly project doesn't contain the examples
> > anymore,
> > > I
> > > > tried running a Gelly example by the packaging flink-gelly-examples
> jar
> > > > (mvn package). However, this gives me a ClassNotFoundException.
> > > > e.g. for SingleSourceShortestPaths, the following:
> > > >
> > > > java.lang.RuntimeException: Could not look up the main(String[])
> method
> > > > from the class
> > org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > > > at
> > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > > Caused by: java.lang.NoClassDefFoundError:
> > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > > at java.lang.Class.getMethod0(Class.java:2774)
> > > > at java.lang.Class.getMethod(Class.java:1663)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > > ... 5 more
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.flink.graph.spargel.MessagingFunction
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > > ... 10 more
> > > >
> > > >
> > > > What am I missing here?
> > > >
> > > > Thanks!
> > > > -Vasia.
> > > >
> > > > On 26 February 2016 at 14:10, Márton Balassi <
> balassi.marton@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks, +1.
> > > > >
> > > > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi!
> > > > > >
> > > > > > I think it is a release blocker.
> > > > > >
> > > > > > That was a change to make it possible to use Flink with SBT (and
> > make
> > > > it
> > > > > > easier to use it with Maven).
> > > > > > Just had an offline chat with Till, and we suggest the following
> > > > > solution:
> > > > > >
> > > > > > All libraries should be split into two projects:
> > > > > > - library
> > > > > > - library-examples
> > > > > >
> > > > > > The "library" project has the core flink dependencies as
> "provided"
> > > > > > The "library-examples" project has both the "library" and the
> core
> > > > flink
> > > > > > dependencies with scope "compile"
> > > > > >
> > > > > > That way the example should run in the IDE out of the cox, and
> > users
> > > > that
> > > > > > reference the libraries will still get the correct packaging
> > (include
> > > > the
> > > > > > library in the user jar, but not additionally the core flink
> jars).
> > > > > >
> > > > > > Greetings,
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > > > balassi.marton@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Issued JIRA ticket 3511 to make it referable in other
> > discussions.
> > > > [1]
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > > > >
> > > > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > > > > balassi.marton@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Recent changes to the build [1] where many libraries got
> their
> > > core
> > > > > > > > dependencies (the ones included in the flink-dist fat jar)
> > moved
> > > to
> > > > > the
> > > > > > > > provided scope.
> > > > > > > >
> > > > > > > > The reasoning was that when submitting to the Flink cluster
> the
> > > > > > > > application already has these dependencies, while when a user
> > > > writes
> > > > > a
> > > > > > > > program against these libraries she will include the core
> > > > > dependencies
> > > > > > > > explicitly anyway.
> > > > > > > >
> > > > > > > > There is one other case of usage however, namely when someone
> > is
> > > > > trying
> > > > > > > to
> > > > > > > > run an application defined in these libraries depending on
> the
> > > core
> > > > > > jars.
> > > > > > > > To give an example if you were to run the Gelly
> > > ConnectedComponents
> > > > > > > example
> > > > > > > > [2] from an IDE after importing Flink (or running with java
> > -jar
> > > > > > without
> > > > > > > > including the flink fat jar in the classpath) you would
> receive
> > > the
> > > > > > > > following class not found exception as per the current
> master:
> > > > > > > >
> > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > > > at
> > > > > > >
> > > >
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > > > at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > > at
> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > > at
> > > > >
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > > > >
> > > > > > > > Where the class missing in runtime is in flink-core, a
> > transitive
> > > > > > > > dependency of the jars moved to the provided scope.
> > > > > > > >
> > > > > > > > Funny thing is we have tests in place to run our examples,
> but
> > > > those
> > > > > > add
> > > > > > > > test scope dependencies re-adding the missing classes, so it
> is
> > > > never
> > > > > > > > discovered.
> > > > > > > >
> > > > > > > > I agree with the original purpose of PR #1683, but also think
> > > that
> > > > > the
> > > > > > > > current state makes for very inconvenient user experience.
> > > > > > > >
> > > > > > > > I would like to open a discussion on how and when to resolve
> > the
> > > > > issue
> > > > > > > > given the release of 1.0.0.
> > > > > > > >
> > > > > > > > 1. Is it a release blocker?
> > > > > > > > 2. Should the change be reverted or is it sufficient to have
> > > proper
> > > > > > > > documentation around it? Maybe a maven profile for explicitly
> > for
> > > > > > > > developing Flink without the provided scope?
> > > > > > > >
> > > > > > > > Note that the issue was originally reported by Gábor Gévay.
> > > > > > > >
> > > > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Marton
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Till Rohrmann <tr...@apache.org>.
Good catch :-) I mean we could also change the behaviour to include
flink-gelly in the flink-gelly-examples module.
On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
vasilikikalavri@gmail.com> wrote:
> Thanks Till! Then, we'd better update the docs, too. Currently both links
> to examples and cluster execution instructions are broken.
> I'll create an issue.
>
> -V.
>
> On 29 February 2016 at 11:54, Till Rohrmann <tr...@apache.org> wrote:
>
> > Hi Vasia,
> >
> > that is because you're missing the flink-gelly dependency. If you just
> > build flink-gelly-examples, then it won't contain flink-gelly because it
> is
> > not a fat jar. You either have to install flink-gelly on your cluster or
> > package it in the final user jar.
> >
> > Cheers,
> > Till
> >
> > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > vasilikikalavri@gmail.com
> > > wrote:
> >
> > > Hi squirrels,
> > >
> > > sorry I've been slow to respond to this, but I'm now testing RC1 and
> I'm
> > a
> > > bit confused with this change.
> > >
> > > So far, the easier way to run a Gelly example on a cluster was to
> package
> > > and submit the Gelly jar.
> > > Now, since the flink-gelly project doesn't contain the examples
> anymore,
> > I
> > > tried running a Gelly example by the packaging flink-gelly-examples jar
> > > (mvn package). However, this gives me a ClassNotFoundException.
> > > e.g. for SingleSourceShortestPaths, the following:
> > >
> > > java.lang.RuntimeException: Could not look up the main(String[]) method
> > > from the class
> org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > org/apache/flink/graph/spargel/MessagingFunction
> > > at
> > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > at
> > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > > at
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > at
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > Caused by: java.lang.NoClassDefFoundError:
> > > org/apache/flink/graph/spargel/MessagingFunction
> > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > at java.lang.Class.getMethod0(Class.java:2774)
> > > at java.lang.Class.getMethod(Class.java:1663)
> > > at
> > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > ... 5 more
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.flink.graph.spargel.MessagingFunction
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > ... 10 more
> > >
> > >
> > > What am I missing here?
> > >
> > > Thanks!
> > > -Vasia.
> > >
> > > On 26 February 2016 at 14:10, Márton Balassi <balassi.marton@gmail.com
> >
> > > wrote:
> > >
> > > > Thanks, +1.
> > > >
> > > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > I think it is a release blocker.
> > > > >
> > > > > That was a change to make it possible to use Flink with SBT (and
> make
> > > it
> > > > > easier to use it with Maven).
> > > > > Just had an offline chat with Till, and we suggest the following
> > > > solution:
> > > > >
> > > > > All libraries should be split into two projects:
> > > > > - library
> > > > > - library-examples
> > > > >
> > > > > The "library" project has the core flink dependencies as "provided"
> > > > > The "library-examples" project has both the "library" and the core
> > > flink
> > > > > dependencies with scope "compile"
> > > > >
> > > > > That way the example should run in the IDE out of the cox, and
> users
> > > that
> > > > > reference the libraries will still get the correct packaging
> (include
> > > the
> > > > > library in the user jar, but not additionally the core flink jars).
> > > > >
> > > > > Greetings,
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > > balassi.marton@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Issued JIRA ticket 3511 to make it referable in other
> discussions.
> > > [1]
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > > >
> > > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > > > balassi.marton@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Recent changes to the build [1] where many libraries got their
> > core
> > > > > > > dependencies (the ones included in the flink-dist fat jar)
> moved
> > to
> > > > the
> > > > > > > provided scope.
> > > > > > >
> > > > > > > The reasoning was that when submitting to the Flink cluster the
> > > > > > > application already has these dependencies, while when a user
> > > writes
> > > > a
> > > > > > > program against these libraries she will include the core
> > > > dependencies
> > > > > > > explicitly anyway.
> > > > > > >
> > > > > > > There is one other case of usage however, namely when someone
> is
> > > > trying
> > > > > > to
> > > > > > > run an application defined in these libraries depending on the
> > core
> > > > > jars.
> > > > > > > To give an example if you were to run the Gelly
> > ConnectedComponents
> > > > > > example
> > > > > > > [2] from an IDE after importing Flink (or running with java
> -jar
> > > > > without
> > > > > > > including the flink fat jar in the classpath) you would receive
> > the
> > > > > > > following class not found exception as per the current master:
> > > > > > >
> > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > > at
> > > > > >
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > at
> > > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > > >
> > > > > > > Where the class missing in runtime is in flink-core, a
> transitive
> > > > > > > dependency of the jars moved to the provided scope.
> > > > > > >
> > > > > > > Funny thing is we have tests in place to run our examples, but
> > > those
> > > > > add
> > > > > > > test scope dependencies re-adding the missing classes, so it is
> > > never
> > > > > > > discovered.
> > > > > > >
> > > > > > > I agree with the original purpose of PR #1683, but also think
> > that
> > > > the
> > > > > > > current state makes for very inconvenient user experience.
> > > > > > >
> > > > > > > I would like to open a discussion on how and when to resolve
> the
> > > > issue
> > > > > > > given the release of 1.0.0.
> > > > > > >
> > > > > > > 1. Is it a release blocker?
> > > > > > > 2. Should the change be reverted or is it sufficient to have
> > proper
> > > > > > > documentation around it? Maybe a maven profile for explicitly
> for
> > > > > > > developing Flink without the provided scope?
> > > > > > >
> > > > > > > Note that the issue was originally reported by Gábor Gévay.
> > > > > > >
> > > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > > [2]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Marton
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Vasiliki Kalavri <va...@gmail.com>.
Thanks Till! Then, we'd better update the docs, too. Currently both links
to examples and cluster execution instructions are broken.
I'll create an issue.
-V.
On 29 February 2016 at 11:54, Till Rohrmann <tr...@apache.org> wrote:
> Hi Vasia,
>
> that is because you're missing the flink-gelly dependency. If you just
> build flink-gelly-examples, then it won't contain flink-gelly because it is
> not a fat jar. You either have to install flink-gelly on your cluster or
> package it in the final user jar.
>
> Cheers,
> Till
>
> On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com
> > wrote:
>
> > Hi squirrels,
> >
> > sorry I've been slow to respond to this, but I'm now testing RC1 and I'm
> a
> > bit confused with this change.
> >
> > So far, the easier way to run a Gelly example on a cluster was to package
> > and submit the Gelly jar.
> > Now, since the flink-gelly project doesn't contain the examples anymore,
> I
> > tried running a Gelly example by the packaging flink-gelly-examples jar
> > (mvn package). However, this gives me a ClassNotFoundException.
> > e.g. for SingleSourceShortestPaths, the following:
> >
> > java.lang.RuntimeException: Could not look up the main(String[]) method
> > from the class org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > org/apache/flink/graph/spargel/MessagingFunction
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > at
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/graph/spargel/MessagingFunction
> > at java.lang.Class.getDeclaredMethods0(Native Method)
> > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > at java.lang.Class.getMethod0(Class.java:2774)
> > at java.lang.Class.getMethod(Class.java:1663)
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > ... 5 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.graph.spargel.MessagingFunction
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > ... 10 more
> >
> >
> > What am I missing here?
> >
> > Thanks!
> > -Vasia.
> >
> > On 26 February 2016 at 14:10, Márton Balassi <ba...@gmail.com>
> > wrote:
> >
> > > Thanks, +1.
> > >
> > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > Hi!
> > > >
> > > > I think it is a release blocker.
> > > >
> > > > That was a change to make it possible to use Flink with SBT (and make
> > it
> > > > easier to use it with Maven).
> > > > Just had an offline chat with Till, and we suggest the following
> > > solution:
> > > >
> > > > All libraries should be split into two projects:
> > > > - library
> > > > - library-examples
> > > >
> > > > The "library" project has the core flink dependencies as "provided"
> > > > The "library-examples" project has both the "library" and the core
> > flink
> > > > dependencies with scope "compile"
> > > >
> > > > That way the example should run in the IDE out of the cox, and users
> > that
> > > > reference the libraries will still get the correct packaging (include
> > the
> > > > library in the user jar, but not additionally the core flink jars).
> > > >
> > > > Greetings,
> > > > Stephan
> > > >
> > > >
> > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > balassi.marton@gmail.com>
> > > > wrote:
> > > >
> > > > > Issued JIRA ticket 3511 to make it referable in other discussions.
> > [1]
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > >
> > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > > balassi.marton@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Recent changes to the build [1] where many libraries got their
> core
> > > > > > dependencies (the ones included in the flink-dist fat jar) moved
> to
> > > the
> > > > > > provided scope.
> > > > > >
> > > > > > The reasoning was that when submitting to the Flink cluster the
> > > > > > application already has these dependencies, while when a user
> > writes
> > > a
> > > > > > program against these libraries she will include the core
> > > dependencies
> > > > > > explicitly anyway.
> > > > > >
> > > > > > There is one other case of usage however, namely when someone is
> > > trying
> > > > > to
> > > > > > run an application defined in these libraries depending on the
> core
> > > > jars.
> > > > > > To give an example if you were to run the Gelly
> ConnectedComponents
> > > > > example
> > > > > > [2] from an IDE after importing Flink (or running with java -jar
> > > > without
> > > > > > including the flink fat jar in the classpath) you would receive
> the
> > > > > > following class not found exception as per the current master:
> > > > > >
> > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > at
> > > > >
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > at
> > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > >
> > > > > > Where the class missing in runtime is in flink-core, a transitive
> > > > > > dependency of the jars moved to the provided scope.
> > > > > >
> > > > > > Funny thing is we have tests in place to run our examples, but
> > those
> > > > add
> > > > > > test scope dependencies re-adding the missing classes, so it is
> > never
> > > > > > discovered.
> > > > > >
> > > > > > I agree with the original purpose of PR #1683, but also think
> that
> > > the
> > > > > > current state makes for very inconvenient user experience.
> > > > > >
> > > > > > I would like to open a discussion on how and when to resolve the
> > > issue
> > > > > > given the release of 1.0.0.
> > > > > >
> > > > > > 1. Is it a release blocker?
> > > > > > 2. Should the change be reverted or is it sufficient to have
> proper
> > > > > > documentation around it? Maybe a maven profile for explicitly for
> > > > > > developing Flink without the provided scope?
> > > > > >
> > > > > > Note that the issue was originally reported by Gábor Gévay.
> > > > > >
> > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Marton
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Till Rohrmann <tr...@apache.org>.
Hi Vasia,
that is because you're missing the flink-gelly dependency. If you just
build flink-gelly-examples, then it won't contain flink-gelly because it is
not a fat jar. You either have to install flink-gelly on your cluster or
package it in the final user jar.
Cheers,
Till
On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <vasilikikalavri@gmail.com
> wrote:
> Hi squirrels,
>
> sorry I've been slow to respond to this, but I'm now testing RC1 and I'm a
> bit confused with this change.
>
> So far, the easier way to run a Gelly example on a cluster was to package
> and submit the Gelly jar.
> Now, since the flink-gelly project doesn't contain the examples anymore, I
> tried running a Gelly example by the packaging flink-gelly-examples jar
> (mvn package). However, this gives me a ClassNotFoundException.
> e.g. for SingleSourceShortestPaths, the following:
>
> java.lang.RuntimeException: Could not look up the main(String[]) method
> from the class org.apache.flink.graph.examples.SingleSourceShortestPaths:
> org/apache/flink/graph/spargel/MessagingFunction
> at
>
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> at
>
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/graph/spargel/MessagingFunction
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> at java.lang.Class.getMethod0(Class.java:2774)
> at java.lang.Class.getMethod(Class.java:1663)
> at
>
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.graph.spargel.MessagingFunction
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 10 more
>
>
> What am I missing here?
>
> Thanks!
> -Vasia.
>
> On 26 February 2016 at 14:10, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Thanks, +1.
> >
> > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi!
> > >
> > > I think it is a release blocker.
> > >
> > > That was a change to make it possible to use Flink with SBT (and make
> it
> > > easier to use it with Maven).
> > > Just had an offline chat with Till, and we suggest the following
> > solution:
> > >
> > > All libraries should be split into two projects:
> > > - library
> > > - library-examples
> > >
> > > The "library" project has the core flink dependencies as "provided"
> > > The "library-examples" project has both the "library" and the core
> flink
> > > dependencies with scope "compile"
> > >
> > > That way the example should run in the IDE out of the cox, and users
> that
> > > reference the libraries will still get the correct packaging (include
> the
> > > library in the user jar, but not additionally the core flink jars).
> > >
> > > Greetings,
> > > Stephan
> > >
> > >
> > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > Issued JIRA ticket 3511 to make it referable in other discussions.
> [1]
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > >
> > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > > balassi.marton@gmail.com>
> > > > wrote:
> > > >
> > > > > Recent changes to the build [1] where many libraries got their core
> > > > > dependencies (the ones included in the flink-dist fat jar) moved to
> > the
> > > > > provided scope.
> > > > >
> > > > > The reasoning was that when submitting to the Flink cluster the
> > > > > application already has these dependencies, while when a user
> writes
> > a
> > > > > program against these libraries she will include the core
> > dependencies
> > > > > explicitly anyway.
> > > > >
> > > > > There is one other case of usage however, namely when someone is
> > trying
> > > > to
> > > > > run an application defined in these libraries depending on the core
> > > jars.
> > > > > To give an example if you were to run the Gelly ConnectedComponents
> > > > example
> > > > > [2] from an IDE after importing Flink (or running with java -jar
> > > without
> > > > > including the flink fat jar in the classpath) you would receive the
> > > > > following class not found exception as per the current master:
> > > > >
> > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > org/apache/flink/api/common/ProgramDescription
> > > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > at
> > > >
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > at java.lang.Class.forName0(Native Method)
> > > > > at java.lang.Class.forName(Class.java:264)
> > > > > at
> > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > >
> > > > > Where the class missing in runtime is in flink-core, a transitive
> > > > > dependency of the jars moved to the provided scope.
> > > > >
> > > > > Funny thing is we have tests in place to run our examples, but
> those
> > > add
> > > > > test scope dependencies re-adding the missing classes, so it is
> never
> > > > > discovered.
> > > > >
> > > > > I agree with the original purpose of PR #1683, but also think that
> > the
> > > > > current state makes for very inconvenient user experience.
> > > > >
> > > > > I would like to open a discussion on how and when to resolve the
> > issue
> > > > > given the release of 1.0.0.
> > > > >
> > > > > 1. Is it a release blocker?
> > > > > 2. Should the change be reverted or is it sufficient to have proper
> > > > > documentation around it? Maybe a maven profile for explicitly for
> > > > > developing Flink without the provided scope?
> > > > >
> > > > > Note that the issue was originally reported by Gábor Gévay.
> > > > >
> > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > >
> > > > > Best,
> > > > >
> > > > > Marton
> > > > >
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Vasiliki Kalavri <va...@gmail.com>.
Hi squirrels,
sorry I've been slow to respond to this, but I'm now testing RC1 and I'm a
bit confused with this change.
So far, the easier way to run a Gelly example on a cluster was to package
and submit the Gelly jar.
Now, since the flink-gelly project doesn't contain the examples anymore, I
tried running a Gelly example by the packaging flink-gelly-examples jar
(mvn package). However, this gives me a ClassNotFoundException.
e.g. for SingleSourceShortestPaths, the following:
java.lang.RuntimeException: Could not look up the main(String[]) method
from the class org.apache.flink.graph.examples.SingleSourceShortestPaths:
org/apache/flink/graph/spargel/MessagingFunction
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
at
org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/graph/spargel/MessagingFunction
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
at java.lang.Class.getMethod0(Class.java:2774)
at java.lang.Class.getMethod(Class.java:1663)
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
... 5 more
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.graph.spargel.MessagingFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 10 more
What am I missing here?
Thanks!
-Vasia.
On 26 February 2016 at 14:10, Márton Balassi <ba...@gmail.com>
wrote:
> Thanks, +1.
>
> On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi!
> >
> > I think it is a release blocker.
> >
> > That was a change to make it possible to use Flink with SBT (and make it
> > easier to use it with Maven).
> > Just had an offline chat with Till, and we suggest the following
> solution:
> >
> > All libraries should be split into two projects:
> > - library
> > - library-examples
> >
> > The "library" project has the core flink dependencies as "provided"
> > The "library-examples" project has both the "library" and the core flink
> > dependencies with scope "compile"
> >
> > That way the example should run in the IDE out of the cox, and users that
> > reference the libraries will still get the correct packaging (include the
> > library in the user jar, but not additionally the core flink jars).
> >
> > Greetings,
> > Stephan
> >
> >
> > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> > > Issued JIRA ticket 3511 to make it referable in other discussions. [1]
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > >
> > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > Recent changes to the build [1] where many libraries got their core
> > > > dependencies (the ones included in the flink-dist fat jar) moved to
> the
> > > > provided scope.
> > > >
> > > > The reasoning was that when submitting to the Flink cluster the
> > > > application already has these dependencies, while when a user writes
> a
> > > > program against these libraries she will include the core
> dependencies
> > > > explicitly anyway.
> > > >
> > > > There is one other case of usage however, namely when someone is
> trying
> > > to
> > > > run an application defined in these libraries depending on the core
> > jars.
> > > > To give an example if you were to run the Gelly ConnectedComponents
> > > example
> > > > [2] from an IDE after importing Flink (or running with java -jar
> > without
> > > > including the flink fat jar in the classpath) you would receive the
> > > > following class not found exception as per the current master:
> > > >
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > org/apache/flink/api/common/ProgramDescription
> > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > at
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > at java.lang.Class.forName0(Native Method)
> > > > at java.lang.Class.forName(Class.java:264)
> > > > at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > >
> > > > Where the class missing in runtime is in flink-core, a transitive
> > > > dependency of the jars moved to the provided scope.
> > > >
> > > > Funny thing is we have tests in place to run our examples, but those
> > add
> > > > test scope dependencies re-adding the missing classes, so it is never
> > > > discovered.
> > > >
> > > > I agree with the original purpose of PR #1683, but also think that
> the
> > > > current state makes for very inconvenient user experience.
> > > >
> > > > I would like to open a discussion on how and when to resolve the
> issue
> > > > given the release of 1.0.0.
> > > >
> > > > 1. Is it a release blocker?
> > > > 2. Should the change be reverted or is it sufficient to have proper
> > > > documentation around it? Maybe a maven profile for explicitly for
> > > > developing Flink without the provided scope?
> > > >
> > > > Note that the issue was originally reported by Gábor Gévay.
> > > >
> > > > [1] https://github.com/apache/flink/pull/1683
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > >
> > > > Best,
> > > >
> > > > Marton
> > > >
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Márton Balassi <ba...@gmail.com>.
Thanks, +1.
On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> I think it is a release blocker.
>
> That was a change to make it possible to use Flink with SBT (and make it
> easier to use it with Maven).
> Just had an offline chat with Till, and we suggest the following solution:
>
> All libraries should be split into two projects:
> - library
> - library-examples
>
> The "library" project has the core flink dependencies as "provided"
> The "library-examples" project has both the "library" and the core flink
> dependencies with scope "compile"
>
> That way the example should run in the IDE out of the cox, and users that
> reference the libraries will still get the correct packaging (include the
> library in the user jar, but not additionally the core flink jars).
>
> Greetings,
> Stephan
>
>
> On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Issued JIRA ticket 3511 to make it referable in other discussions. [1]
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-3511
> >
> > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> > > Recent changes to the build [1] where many libraries got their core
> > > dependencies (the ones included in the flink-dist fat jar) moved to the
> > > provided scope.
> > >
> > > The reasoning was that when submitting to the Flink cluster the
> > > application already has these dependencies, while when a user writes a
> > > program against these libraries she will include the core dependencies
> > > explicitly anyway.
> > >
> > > There is one other case of usage however, namely when someone is trying
> > to
> > > run an application defined in these libraries depending on the core
> jars.
> > > To give an example if you were to run the Gelly ConnectedComponents
> > example
> > > [2] from an IDE after importing Flink (or running with java -jar
> without
> > > including the flink fat jar in the classpath) you would receive the
> > > following class not found exception as per the current master:
> > >
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > org/apache/flink/api/common/ProgramDescription
> > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > at java.lang.Class.forName0(Native Method)
> > > at java.lang.Class.forName(Class.java:264)
> > > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > >
> > > Where the class missing in runtime is in flink-core, a transitive
> > > dependency of the jars moved to the provided scope.
> > >
> > > Funny thing is we have tests in place to run our examples, but those
> add
> > > test scope dependencies re-adding the missing classes, so it is never
> > > discovered.
> > >
> > > I agree with the original purpose of PR #1683, but also think that the
> > > current state makes for very inconvenient user experience.
> > >
> > > I would like to open a discussion on how and when to resolve the issue
> > > given the release of 1.0.0.
> > >
> > > 1. Is it a release blocker?
> > > 2. Should the change be reverted or is it sufficient to have proper
> > > documentation around it? Maybe a maven profile for explicitly for
> > > developing Flink without the provided scope?
> > >
> > > Note that the issue was originally reported by Gábor Gévay.
> > >
> > > [1] https://github.com/apache/flink/pull/1683
> > > [2]
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > >
> > > Best,
> > >
> > > Marton
> > >
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Stephan Ewen <se...@apache.org>.
Hi!
I think it is a release blocker.
That was a change to make it possible to use Flink with SBT (and make it
easier to use it with Maven).
Just had an offline chat with Till, and we suggest the following solution:
All libraries should be split into two projects:
- library
- library-examples
The "library" project has the core flink dependencies as "provided"
The "library-examples" project has both the "library" and the core flink
dependencies with scope "compile"
That way the example should run in the IDE out of the cox, and users that
reference the libraries will still get the correct packaging (include the
library in the user jar, but not additionally the core flink jars).
Greetings,
Stephan
On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <ba...@gmail.com>
wrote:
> Issued JIRA ticket 3511 to make it referable in other discussions. [1]
>
> [1] https://issues.apache.org/jira/browse/FLINK-3511
>
> On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Recent changes to the build [1] where many libraries got their core
> > dependencies (the ones included in the flink-dist fat jar) moved to the
> > provided scope.
> >
> > The reasoning was that when submitting to the Flink cluster the
> > application already has these dependencies, while when a user writes a
> > program against these libraries she will include the core dependencies
> > explicitly anyway.
> >
> > There is one other case of usage however, namely when someone is trying
> to
> > run an application defined in these libraries depending on the core jars.
> > To give an example if you were to run the Gelly ConnectedComponents
> example
> > [2] from an IDE after importing Flink (or running with java -jar without
> > including the flink fat jar in the classpath) you would receive the
> > following class not found exception as per the current master:
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/flink/api/common/ProgramDescription
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:264)
> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> >
> > Where the class missing in runtime is in flink-core, a transitive
> > dependency of the jars moved to the provided scope.
> >
> > Funny thing is we have tests in place to run our examples, but those add
> > test scope dependencies re-adding the missing classes, so it is never
> > discovered.
> >
> > I agree with the original purpose of PR #1683, but also think that the
> > current state makes for very inconvenient user experience.
> >
> > I would like to open a discussion on how and when to resolve the issue
> > given the release of 1.0.0.
> >
> > 1. Is it a release blocker?
> > 2. Should the change be reverted or is it sufficient to have proper
> > documentation around it? Maybe a maven profile for explicitly for
> > developing Flink without the provided scope?
> >
> > Note that the issue was originally reported by Gábor Gévay.
> >
> > [1] https://github.com/apache/flink/pull/1683
> > [2]
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> >
> > Best,
> >
> > Marton
> >
>
Re: Inconvenient (unforeseen?) consequences of PR #1683
Posted by Márton Balassi <ba...@gmail.com>.
Issued JIRA ticket 3511 to make it referable in other discussions. [1]
[1] https://issues.apache.org/jira/browse/FLINK-3511
On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <ba...@gmail.com>
wrote:
> Recent changes to the build [1] where many libraries got their core
> dependencies (the ones included in the flink-dist fat jar) moved to the
> provided scope.
>
> The reasoning was that when submitting to the Flink cluster the
> application already has these dependencies, while when a user writes a
> program against these libraries she will include the core dependencies
> explicitly anyway.
>
> There is one other case of usage however, namely when someone is trying to
> run an application defined in these libraries depending on the core jars.
> To give an example if you were to run the Gelly ConnectedComponents example
> [2] from an IDE after importing Flink (or running with java -jar without
> including the flink fat jar in the classpath) you would receive the
> following class not found exception as per the current master:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/flink/api/common/ProgramDescription
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
>
> Where the class missing in runtime is in flink-core, a transitive
> dependency of the jars moved to the provided scope.
>
> Funny thing is we have tests in place to run our examples, but those add
> test scope dependencies re-adding the missing classes, so it is never
> discovered.
>
> I agree with the original purpose of PR #1683, but also think that the
> current state makes for very inconvenient user experience.
>
> I would like to open a discussion on how and when to resolve the issue
> given the release of 1.0.0.
>
> 1. Is it a release blocker?
> 2. Should the change be reverted or is it sufficient to have proper
> documentation around it? Maybe a maven profile for explicitly for
> developing Flink without the provided scope?
>
> Note that the issue was originally reported by Gábor Gévay.
>
> [1] https://github.com/apache/flink/pull/1683
> [2]
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
>
> Best,
>
> Marton
>