You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2015/03/17 10:46:58 UTC
Re: Restructuring the maven projects
To not let this discussion die, here is a concrete JIRA and a proposed
layout to restructure to.
What remains to be discusses is whether we want to keep the Scala/Java APIs
for batch/streaming in separate projects or in one project.
Also, we need to find a good time to do this, when we are low on pull
requests...
https://issues.apache.org/jira/browse/FLINK-1712
Proposed Layout:
- flink-hadoop (shaded fat jar)
- Core (Core and Java and Scala)
- Streaming (core + java + scala)
- Runtime
- Client (Client + Optimizer)
- Examples (Java + Scala + Streaming Java + Streaming Scala)
- Tests (test-utils (compile) and tests (test))
- Quickstarts
- Quickstart Java
- Quickstart Scala
- connectors / Input/Output Formats
- Avro
- HBase
- HadoopCompartibility
- HCatalogue
- JDBC
- kafka
- rabbit
- ...
- staging
- Gelly
- Gilbert (ML)
- spargel (deprecated)
- expression API
- contrib
- yarn
- dist
- yarn tests
- java 8
On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <he...@gmail.com>
wrote:
> Thanks Marton, having 2 threads discussing same thing can be confusing.
>
> - Henry
>
> On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
> wrote:
> > Let us consider this thread the standard for the restructure, it is
> > perfectly in line with the wishes I have posted.
> >
> > +1 for keeping the 'flink-' prefix.
> >
> > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <he...@gmail.com>
> > wrote:
> >
> >> @Robert, and @Stephan, sure I am ok with it, thanks for the responses.
> >>
> >> - Henry
> >>
> >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org> wrote:
> >> > I think this works well together with Marton's restructuring.
> >> >
> >> > I would vote to keep the "flink-" prefix, because it guarantees that
> the
> >> > produced jars are prefixed with "flink-". Otherwise, we will have to
> >> start
> >> > configuring a lot...
> >> >
> >> > Greetings,
> >> > Stephan
> >> >
> >> >
> >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> henry.saputra@gmail.com>
> >> > wrote:
> >> >
> >> >> Will this conflict with Marton's restructuring proposal which happens
> >> >> in another thread (see "Project restructure" thread in the dev@
> list).
> >> >>
> >> >> Since we are doing refactoring, may I suggest that we also remove
> >> >> "flink-" prefix since maven group name will indicate it is part of
> >> >> Flink.
> >> >>
> >> >> - Henry
> >> >>
> >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >> >> > Hi everyone!
> >> >> >
> >> >> > I think that by now, quite a bit of the maven project structure
> can be
> >> >> > improved to get rid of some legacy artifacts. Especially the
> >> >> "flink-addons"
> >> >> > project seems to be a catch-all place for various projects.
> >> >> >
> >> >> > Here is a suggestion what we could do:
> >> >> >
> >> >> > 1) Move "flink-yarn" to the root.
> >> >> >
> >> >> > 2) Move "flink-streaming" to the root (this is planned anyways for
> the
> >> >> next
> >> >> > release)
> >> >> >
> >> >> > 3) Create a project "flink-connectors", which will contain "avro",
> >> >> "jdbc",
> >> >> > and "hbase". Should we have them as separate sub-projects, or as
> one
> >> >> > project?
> >> >> >
> >> >> > 4) Consolidate the examples into a single project "flink-examples",
> >> where
> >> >> > Java, Scala, Streaming examples exist in different packages.
> >> >> >
> >> >> >
> >> >> > Greetings,
> >> >> > Stephan
> >> >>
> >>
>
Re: Restructuring the maven projects
Posted by Stephan Ewen <se...@apache.org>.
Optimizer and Compiler can stay separate, of course.
I was just thinking that we have a lot of projects and this would be a
simple way to get rid of one. But it actually is intuitive to me as well to
keep them separate.
On Wed, Mar 18, 2015 at 1:23 AM, Fabian Hueske <fh...@gmail.com> wrote:
> I agree that it's a good idea to move the APIs into one module.
>
> But why should we merge client and compiler (optimizer) and the examples
> into one module?
> I think modules with clearly separated responsibilities can also help new
> contributors to navigate the code.
>
> 2015-03-17 16:16 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
> > The good thing about the API projects is that there is no dependency from
> > Java code to Scala code. I think that caused most of the issues.
> >
> > We may still want to keep it separate. I am not fully decided on this
> > yet...
> >
> > Stephan
> >
> > On Tue, Mar 17, 2015 at 3:52 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > Thanks for bringing this up, Till. You are right, but I think the main
> > > issue was that tight interaction between Java and Scala was
> problematic.
> > I
> > > am not sure whether this is such a big problem for the the APIs.
> > >
> > > On Tue, Mar 17, 2015 at 3:21 PM, Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > Putting the Scala and Java API into the same module means that we'll
> > have
> > > > more mixed Java/Scala projects, right? I just want to check if
> everyone
> > > is
> > > > aware of it considering our latest experiences with these kind of
> > > modules.
> > > >
> > > > On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uc...@apache.org> wrote:
> > > >
> > > > > +1 I like the proposed structure.
> > > > >
> > > > > The only thing I was wondering about is whether to name "core" =>
> > > > "batch"?
> > > > >
> > > > > On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <
> > > > balassi.marton@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1 for the proposed structure.
> > > > > >
> > > > > > I have no explicit preference for having batch and streaming
> scala
> > > > > together
> > > > > > or separated. That said streaming scala is considerably thin, it
> > does
> > > > not
> > > > > > really require an own maven submodule.
> > > > > >
> > > > > > Marked an older JIRA for the same issue as duplicate. [1]
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-1340
> > > > > >
> > > > > > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > To not let this discussion die, here is a concrete JIRA and a
> > > > proposed
> > > > > > > layout to restructure to.
> > > > > > >
> > > > > > > What remains to be discusses is whether we want to keep the
> > > > Scala/Java
> > > > > > APIs
> > > > > > > for batch/streaming in separate projects or in one project.
> > > > > > >
> > > > > > > Also, we need to find a good time to do this, when we are low
> on
> > > pull
> > > > > > > requests...
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1712
> > > > > > >
> > > > > > > Proposed Layout:
> > > > > > >
> > > > > > > - flink-hadoop (shaded fat jar)
> > > > > > >
> > > > > > > - Core (Core and Java and Scala)
> > > > > > > - Streaming (core + java + scala)
> > > > > > > - Runtime
> > > > > > > - Client (Client + Optimizer)
> > > > > > >
> > > > > > > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > > > > > > - Tests (test-utils (compile) and tests (test))
> > > > > > >
> > > > > > > - Quickstarts
> > > > > > > - Quickstart Java
> > > > > > > - Quickstart Scala
> > > > > > >
> > > > > > > - connectors / Input/Output Formats
> > > > > > > - Avro
> > > > > > > - HBase
> > > > > > > - HadoopCompartibility
> > > > > > > - HCatalogue
> > > > > > > - JDBC
> > > > > > > - kafka
> > > > > > > - rabbit
> > > > > > > - ...
> > > > > > >
> > > > > > > - staging
> > > > > > > - Gelly
> > > > > > > - Gilbert (ML)
> > > > > > > - spargel (deprecated)
> > > > > > > - expression API
> > > > > > >
> > > > > > > - contrib
> > > > > > >
> > > > > > > - yarn
> > > > > > >
> > > > > > > - dist
> > > > > > >
> > > > > > > - yarn tests
> > > > > > >
> > > > > > > - java 8
> > > > > > >
> > > > > > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <
> > > > henry.saputra@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Marton, having 2 threads discussing same thing can be
> > > > > confusing.
> > > > > > > >
> > > > > > > > - Henry
> > > > > > > >
> > > > > > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <
> > > > mbalassi@apache.org>
> > > > > > > > wrote:
> > > > > > > > > Let us consider this thread the standard for the
> restructure,
> > > it
> > > > is
> > > > > > > > > perfectly in line with the wishes I have posted.
> > > > > > > > >
> > > > > > > > > +1 for keeping the 'flink-' prefix.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > > > > > > henry.saputra@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> @Robert, and @Stephan, sure I am ok with it, thanks for
> the
> > > > > > responses.
> > > > > > > > >>
> > > > > > > > >> - Henry
> > > > > > > > >>
> > > > > > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > > >> > I think this works well together with Marton's
> > > restructuring.
> > > > > > > > >> >
> > > > > > > > >> > I would vote to keep the "flink-" prefix, because it
> > > > guarantees
> > > > > > that
> > > > > > > > the
> > > > > > > > >> > produced jars are prefixed with "flink-". Otherwise, we
> > will
> > > > > have
> > > > > > to
> > > > > > > > >> start
> > > > > > > > >> > configuring a lot...
> > > > > > > > >> >
> > > > > > > > >> > Greetings,
> > > > > > > > >> > Stephan
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > > > > > > henry.saputra@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Will this conflict with Marton's restructuring proposal
> > > which
> > > > > > > happens
> > > > > > > > >> >> in another thread (see "Project restructure" thread in
> > the
> > > > dev@
> > > > > > > > list).
> > > > > > > > >> >>
> > > > > > > > >> >> Since we are doing refactoring, may I suggest that we
> > also
> > > > > remove
> > > > > > > > >> >> "flink-" prefix since maven group name will indicate it
> > is
> > > > part
> > > > > > of
> > > > > > > > >> >> Flink.
> > > > > > > > >> >>
> > > > > > > > >> >> - Henry
> > > > > > > > >> >>
> > > > > > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <
> > > > sewen@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >> > Hi everyone!
> > > > > > > > >> >> >
> > > > > > > > >> >> > I think that by now, quite a bit of the maven project
> > > > > structure
> > > > > > > > can be
> > > > > > > > >> >> > improved to get rid of some legacy artifacts.
> > Especially
> > > > the
> > > > > > > > >> >> "flink-addons"
> > > > > > > > >> >> > project seems to be a catch-all place for various
> > > projects.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Here is a suggestion what we could do:
> > > > > > > > >> >> >
> > > > > > > > >> >> > 1) Move "flink-yarn" to the root.
> > > > > > > > >> >> >
> > > > > > > > >> >> > 2) Move "flink-streaming" to the root (this is
> planned
> > > > > anyways
> > > > > > > for
> > > > > > > > the
> > > > > > > > >> >> next
> > > > > > > > >> >> > release)
> > > > > > > > >> >> >
> > > > > > > > >> >> > 3) Create a project "flink-connectors", which will
> > > contain
> > > > > > > "avro",
> > > > > > > > >> >> "jdbc",
> > > > > > > > >> >> > and "hbase". Should we have them as separate
> > > sub-projects,
> > > > or
> > > > > > as
> > > > > > > > one
> > > > > > > > >> >> > project?
> > > > > > > > >> >> >
> > > > > > > > >> >> > 4) Consolidate the examples into a single project
> > > > > > > "flink-examples",
> > > > > > > > >> where
> > > > > > > > >> >> > Java, Scala, Streaming examples exist in different
> > > > packages.
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > Greetings,
> > > > > > > > >> >> > Stephan
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Restructuring the maven projects
Posted by Fabian Hueske <fh...@gmail.com>.
I agree that it's a good idea to move the APIs into one module.
But why should we merge client and compiler (optimizer) and the examples
into one module?
I think modules with clearly separated responsibilities can also help new
contributors to navigate the code.
2015-03-17 16:16 GMT+01:00 Stephan Ewen <se...@apache.org>:
> The good thing about the API projects is that there is no dependency from
> Java code to Scala code. I think that caused most of the issues.
>
> We may still want to keep it separate. I am not fully decided on this
> yet...
>
> Stephan
>
> On Tue, Mar 17, 2015 at 3:52 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > Thanks for bringing this up, Till. You are right, but I think the main
> > issue was that tight interaction between Java and Scala was problematic.
> I
> > am not sure whether this is such a big problem for the the APIs.
> >
> > On Tue, Mar 17, 2015 at 3:21 PM, Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Putting the Scala and Java API into the same module means that we'll
> have
> > > more mixed Java/Scala projects, right? I just want to check if everyone
> > is
> > > aware of it considering our latest experiences with these kind of
> > modules.
> > >
> > > On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > > +1 I like the proposed structure.
> > > >
> > > > The only thing I was wondering about is whether to name "core" =>
> > > "batch"?
> > > >
> > > > On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > +1 for the proposed structure.
> > > > >
> > > > > I have no explicit preference for having batch and streaming scala
> > > > together
> > > > > or separated. That said streaming scala is considerably thin, it
> does
> > > not
> > > > > really require an own maven submodule.
> > > > >
> > > > > Marked an older JIRA for the same issue as duplicate. [1]
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-1340
> > > > >
> > > > > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > To not let this discussion die, here is a concrete JIRA and a
> > > proposed
> > > > > > layout to restructure to.
> > > > > >
> > > > > > What remains to be discusses is whether we want to keep the
> > > Scala/Java
> > > > > APIs
> > > > > > for batch/streaming in separate projects or in one project.
> > > > > >
> > > > > > Also, we need to find a good time to do this, when we are low on
> > pull
> > > > > > requests...
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/FLINK-1712
> > > > > >
> > > > > > Proposed Layout:
> > > > > >
> > > > > > - flink-hadoop (shaded fat jar)
> > > > > >
> > > > > > - Core (Core and Java and Scala)
> > > > > > - Streaming (core + java + scala)
> > > > > > - Runtime
> > > > > > - Client (Client + Optimizer)
> > > > > >
> > > > > > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > > > > > - Tests (test-utils (compile) and tests (test))
> > > > > >
> > > > > > - Quickstarts
> > > > > > - Quickstart Java
> > > > > > - Quickstart Scala
> > > > > >
> > > > > > - connectors / Input/Output Formats
> > > > > > - Avro
> > > > > > - HBase
> > > > > > - HadoopCompartibility
> > > > > > - HCatalogue
> > > > > > - JDBC
> > > > > > - kafka
> > > > > > - rabbit
> > > > > > - ...
> > > > > >
> > > > > > - staging
> > > > > > - Gelly
> > > > > > - Gilbert (ML)
> > > > > > - spargel (deprecated)
> > > > > > - expression API
> > > > > >
> > > > > > - contrib
> > > > > >
> > > > > > - yarn
> > > > > >
> > > > > > - dist
> > > > > >
> > > > > > - yarn tests
> > > > > >
> > > > > > - java 8
> > > > > >
> > > > > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <
> > > henry.saputra@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Marton, having 2 threads discussing same thing can be
> > > > confusing.
> > > > > > >
> > > > > > > - Henry
> > > > > > >
> > > > > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <
> > > mbalassi@apache.org>
> > > > > > > wrote:
> > > > > > > > Let us consider this thread the standard for the restructure,
> > it
> > > is
> > > > > > > > perfectly in line with the wishes I have posted.
> > > > > > > >
> > > > > > > > +1 for keeping the 'flink-' prefix.
> > > > > > > >
> > > > > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > > > > > henry.saputra@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> > > > > responses.
> > > > > > > >>
> > > > > > > >> - Henry
> > > > > > > >>
> > > > > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <
> > sewen@apache.org>
> > > > > > wrote:
> > > > > > > >> > I think this works well together with Marton's
> > restructuring.
> > > > > > > >> >
> > > > > > > >> > I would vote to keep the "flink-" prefix, because it
> > > guarantees
> > > > > that
> > > > > > > the
> > > > > > > >> > produced jars are prefixed with "flink-". Otherwise, we
> will
> > > > have
> > > > > to
> > > > > > > >> start
> > > > > > > >> > configuring a lot...
> > > > > > > >> >
> > > > > > > >> > Greetings,
> > > > > > > >> > Stephan
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > > > > > henry.saputra@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Will this conflict with Marton's restructuring proposal
> > which
> > > > > > happens
> > > > > > > >> >> in another thread (see "Project restructure" thread in
> the
> > > dev@
> > > > > > > list).
> > > > > > > >> >>
> > > > > > > >> >> Since we are doing refactoring, may I suggest that we
> also
> > > > remove
> > > > > > > >> >> "flink-" prefix since maven group name will indicate it
> is
> > > part
> > > > > of
> > > > > > > >> >> Flink.
> > > > > > > >> >>
> > > > > > > >> >> - Henry
> > > > > > > >> >>
> > > > > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <
> > > sewen@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >> >> > Hi everyone!
> > > > > > > >> >> >
> > > > > > > >> >> > I think that by now, quite a bit of the maven project
> > > > structure
> > > > > > > can be
> > > > > > > >> >> > improved to get rid of some legacy artifacts.
> Especially
> > > the
> > > > > > > >> >> "flink-addons"
> > > > > > > >> >> > project seems to be a catch-all place for various
> > projects.
> > > > > > > >> >> >
> > > > > > > >> >> > Here is a suggestion what we could do:
> > > > > > > >> >> >
> > > > > > > >> >> > 1) Move "flink-yarn" to the root.
> > > > > > > >> >> >
> > > > > > > >> >> > 2) Move "flink-streaming" to the root (this is planned
> > > > anyways
> > > > > > for
> > > > > > > the
> > > > > > > >> >> next
> > > > > > > >> >> > release)
> > > > > > > >> >> >
> > > > > > > >> >> > 3) Create a project "flink-connectors", which will
> > contain
> > > > > > "avro",
> > > > > > > >> >> "jdbc",
> > > > > > > >> >> > and "hbase". Should we have them as separate
> > sub-projects,
> > > or
> > > > > as
> > > > > > > one
> > > > > > > >> >> > project?
> > > > > > > >> >> >
> > > > > > > >> >> > 4) Consolidate the examples into a single project
> > > > > > "flink-examples",
> > > > > > > >> where
> > > > > > > >> >> > Java, Scala, Streaming examples exist in different
> > > packages.
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > Greetings,
> > > > > > > >> >> > Stephan
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Restructuring the maven projects
Posted by Stephan Ewen <se...@apache.org>.
The good thing about the API projects is that there is no dependency from
Java code to Scala code. I think that caused most of the issues.
We may still want to keep it separate. I am not fully decided on this yet...
Stephan
On Tue, Mar 17, 2015 at 3:52 PM, Ufuk Celebi <uc...@apache.org> wrote:
> Thanks for bringing this up, Till. You are right, but I think the main
> issue was that tight interaction between Java and Scala was problematic. I
> am not sure whether this is such a big problem for the the APIs.
>
> On Tue, Mar 17, 2015 at 3:21 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Putting the Scala and Java API into the same module means that we'll have
> > more mixed Java/Scala projects, right? I just want to check if everyone
> is
> > aware of it considering our latest experiences with these kind of
> modules.
> >
> > On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > +1 I like the proposed structure.
> > >
> > > The only thing I was wondering about is whether to name "core" =>
> > "batch"?
> > >
> > > On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <
> > balassi.marton@gmail.com
> > > >
> > > wrote:
> > >
> > > > +1 for the proposed structure.
> > > >
> > > > I have no explicit preference for having batch and streaming scala
> > > together
> > > > or separated. That said streaming scala is considerably thin, it does
> > not
> > > > really require an own maven submodule.
> > > >
> > > > Marked an older JIRA for the same issue as duplicate. [1]
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-1340
> > > >
> > > > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > To not let this discussion die, here is a concrete JIRA and a
> > proposed
> > > > > layout to restructure to.
> > > > >
> > > > > What remains to be discusses is whether we want to keep the
> > Scala/Java
> > > > APIs
> > > > > for batch/streaming in separate projects or in one project.
> > > > >
> > > > > Also, we need to find a good time to do this, when we are low on
> pull
> > > > > requests...
> > > > >
> > > > > https://issues.apache.org/jira/browse/FLINK-1712
> > > > >
> > > > > Proposed Layout:
> > > > >
> > > > > - flink-hadoop (shaded fat jar)
> > > > >
> > > > > - Core (Core and Java and Scala)
> > > > > - Streaming (core + java + scala)
> > > > > - Runtime
> > > > > - Client (Client + Optimizer)
> > > > >
> > > > > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > > > > - Tests (test-utils (compile) and tests (test))
> > > > >
> > > > > - Quickstarts
> > > > > - Quickstart Java
> > > > > - Quickstart Scala
> > > > >
> > > > > - connectors / Input/Output Formats
> > > > > - Avro
> > > > > - HBase
> > > > > - HadoopCompartibility
> > > > > - HCatalogue
> > > > > - JDBC
> > > > > - kafka
> > > > > - rabbit
> > > > > - ...
> > > > >
> > > > > - staging
> > > > > - Gelly
> > > > > - Gilbert (ML)
> > > > > - spargel (deprecated)
> > > > > - expression API
> > > > >
> > > > > - contrib
> > > > >
> > > > > - yarn
> > > > >
> > > > > - dist
> > > > >
> > > > > - yarn tests
> > > > >
> > > > > - java 8
> > > > >
> > > > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <
> > henry.saputra@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Marton, having 2 threads discussing same thing can be
> > > confusing.
> > > > > >
> > > > > > - Henry
> > > > > >
> > > > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <
> > mbalassi@apache.org>
> > > > > > wrote:
> > > > > > > Let us consider this thread the standard for the restructure,
> it
> > is
> > > > > > > perfectly in line with the wishes I have posted.
> > > > > > >
> > > > > > > +1 for keeping the 'flink-' prefix.
> > > > > > >
> > > > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > > > > henry.saputra@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> > > > responses.
> > > > > > >>
> > > > > > >> - Henry
> > > > > > >>
> > > > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <
> sewen@apache.org>
> > > > > wrote:
> > > > > > >> > I think this works well together with Marton's
> restructuring.
> > > > > > >> >
> > > > > > >> > I would vote to keep the "flink-" prefix, because it
> > guarantees
> > > > that
> > > > > > the
> > > > > > >> > produced jars are prefixed with "flink-". Otherwise, we will
> > > have
> > > > to
> > > > > > >> start
> > > > > > >> > configuring a lot...
> > > > > > >> >
> > > > > > >> > Greetings,
> > > > > > >> > Stephan
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > > > > henry.saputra@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> Will this conflict with Marton's restructuring proposal
> which
> > > > > happens
> > > > > > >> >> in another thread (see "Project restructure" thread in the
> > dev@
> > > > > > list).
> > > > > > >> >>
> > > > > > >> >> Since we are doing refactoring, may I suggest that we also
> > > remove
> > > > > > >> >> "flink-" prefix since maven group name will indicate it is
> > part
> > > > of
> > > > > > >> >> Flink.
> > > > > > >> >>
> > > > > > >> >> - Henry
> > > > > > >> >>
> > > > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <
> > sewen@apache.org
> > > >
> > > > > > wrote:
> > > > > > >> >> > Hi everyone!
> > > > > > >> >> >
> > > > > > >> >> > I think that by now, quite a bit of the maven project
> > > structure
> > > > > > can be
> > > > > > >> >> > improved to get rid of some legacy artifacts. Especially
> > the
> > > > > > >> >> "flink-addons"
> > > > > > >> >> > project seems to be a catch-all place for various
> projects.
> > > > > > >> >> >
> > > > > > >> >> > Here is a suggestion what we could do:
> > > > > > >> >> >
> > > > > > >> >> > 1) Move "flink-yarn" to the root.
> > > > > > >> >> >
> > > > > > >> >> > 2) Move "flink-streaming" to the root (this is planned
> > > anyways
> > > > > for
> > > > > > the
> > > > > > >> >> next
> > > > > > >> >> > release)
> > > > > > >> >> >
> > > > > > >> >> > 3) Create a project "flink-connectors", which will
> contain
> > > > > "avro",
> > > > > > >> >> "jdbc",
> > > > > > >> >> > and "hbase". Should we have them as separate
> sub-projects,
> > or
> > > > as
> > > > > > one
> > > > > > >> >> > project?
> > > > > > >> >> >
> > > > > > >> >> > 4) Consolidate the examples into a single project
> > > > > "flink-examples",
> > > > > > >> where
> > > > > > >> >> > Java, Scala, Streaming examples exist in different
> > packages.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > Greetings,
> > > > > > >> >> > Stephan
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Restructuring the maven projects
Posted by Ufuk Celebi <uc...@apache.org>.
Thanks for bringing this up, Till. You are right, but I think the main
issue was that tight interaction between Java and Scala was problematic. I
am not sure whether this is such a big problem for the the APIs.
On Tue, Mar 17, 2015 at 3:21 PM, Till Rohrmann <tr...@apache.org> wrote:
> Putting the Scala and Java API into the same module means that we'll have
> more mixed Java/Scala projects, right? I just want to check if everyone is
> aware of it considering our latest experiences with these kind of modules.
>
> On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > +1 I like the proposed structure.
> >
> > The only thing I was wondering about is whether to name "core" =>
> "batch"?
> >
> > On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <
> balassi.marton@gmail.com
> > >
> > wrote:
> >
> > > +1 for the proposed structure.
> > >
> > > I have no explicit preference for having batch and streaming scala
> > together
> > > or separated. That said streaming scala is considerably thin, it does
> not
> > > really require an own maven submodule.
> > >
> > > Marked an older JIRA for the same issue as duplicate. [1]
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-1340
> > >
> > > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > To not let this discussion die, here is a concrete JIRA and a
> proposed
> > > > layout to restructure to.
> > > >
> > > > What remains to be discusses is whether we want to keep the
> Scala/Java
> > > APIs
> > > > for batch/streaming in separate projects or in one project.
> > > >
> > > > Also, we need to find a good time to do this, when we are low on pull
> > > > requests...
> > > >
> > > > https://issues.apache.org/jira/browse/FLINK-1712
> > > >
> > > > Proposed Layout:
> > > >
> > > > - flink-hadoop (shaded fat jar)
> > > >
> > > > - Core (Core and Java and Scala)
> > > > - Streaming (core + java + scala)
> > > > - Runtime
> > > > - Client (Client + Optimizer)
> > > >
> > > > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > > > - Tests (test-utils (compile) and tests (test))
> > > >
> > > > - Quickstarts
> > > > - Quickstart Java
> > > > - Quickstart Scala
> > > >
> > > > - connectors / Input/Output Formats
> > > > - Avro
> > > > - HBase
> > > > - HadoopCompartibility
> > > > - HCatalogue
> > > > - JDBC
> > > > - kafka
> > > > - rabbit
> > > > - ...
> > > >
> > > > - staging
> > > > - Gelly
> > > > - Gilbert (ML)
> > > > - spargel (deprecated)
> > > > - expression API
> > > >
> > > > - contrib
> > > >
> > > > - yarn
> > > >
> > > > - dist
> > > >
> > > > - yarn tests
> > > >
> > > > - java 8
> > > >
> > > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <
> henry.saputra@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks Marton, having 2 threads discussing same thing can be
> > confusing.
> > > > >
> > > > > - Henry
> > > > >
> > > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <
> mbalassi@apache.org>
> > > > > wrote:
> > > > > > Let us consider this thread the standard for the restructure, it
> is
> > > > > > perfectly in line with the wishes I have posted.
> > > > > >
> > > > > > +1 for keeping the 'flink-' prefix.
> > > > > >
> > > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > > > henry.saputra@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> > > responses.
> > > > > >>
> > > > > >> - Henry
> > > > > >>
> > > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >> > I think this works well together with Marton's restructuring.
> > > > > >> >
> > > > > >> > I would vote to keep the "flink-" prefix, because it
> guarantees
> > > that
> > > > > the
> > > > > >> > produced jars are prefixed with "flink-". Otherwise, we will
> > have
> > > to
> > > > > >> start
> > > > > >> > configuring a lot...
> > > > > >> >
> > > > > >> > Greetings,
> > > > > >> > Stephan
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > > > henry.saputra@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Will this conflict with Marton's restructuring proposal which
> > > > happens
> > > > > >> >> in another thread (see "Project restructure" thread in the
> dev@
> > > > > list).
> > > > > >> >>
> > > > > >> >> Since we are doing refactoring, may I suggest that we also
> > remove
> > > > > >> >> "flink-" prefix since maven group name will indicate it is
> part
> > > of
> > > > > >> >> Flink.
> > > > > >> >>
> > > > > >> >> - Henry
> > > > > >> >>
> > > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <
> sewen@apache.org
> > >
> > > > > wrote:
> > > > > >> >> > Hi everyone!
> > > > > >> >> >
> > > > > >> >> > I think that by now, quite a bit of the maven project
> > structure
> > > > > can be
> > > > > >> >> > improved to get rid of some legacy artifacts. Especially
> the
> > > > > >> >> "flink-addons"
> > > > > >> >> > project seems to be a catch-all place for various projects.
> > > > > >> >> >
> > > > > >> >> > Here is a suggestion what we could do:
> > > > > >> >> >
> > > > > >> >> > 1) Move "flink-yarn" to the root.
> > > > > >> >> >
> > > > > >> >> > 2) Move "flink-streaming" to the root (this is planned
> > anyways
> > > > for
> > > > > the
> > > > > >> >> next
> > > > > >> >> > release)
> > > > > >> >> >
> > > > > >> >> > 3) Create a project "flink-connectors", which will contain
> > > > "avro",
> > > > > >> >> "jdbc",
> > > > > >> >> > and "hbase". Should we have them as separate sub-projects,
> or
> > > as
> > > > > one
> > > > > >> >> > project?
> > > > > >> >> >
> > > > > >> >> > 4) Consolidate the examples into a single project
> > > > "flink-examples",
> > > > > >> where
> > > > > >> >> > Java, Scala, Streaming examples exist in different
> packages.
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > Greetings,
> > > > > >> >> > Stephan
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Restructuring the maven projects
Posted by Till Rohrmann <tr...@apache.org>.
Putting the Scala and Java API into the same module means that we'll have
more mixed Java/Scala projects, right? I just want to check if everyone is
aware of it considering our latest experiences with these kind of modules.
On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uc...@apache.org> wrote:
> +1 I like the proposed structure.
>
> The only thing I was wondering about is whether to name "core" => "batch"?
>
> On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <balassi.marton@gmail.com
> >
> wrote:
>
> > +1 for the proposed structure.
> >
> > I have no explicit preference for having batch and streaming scala
> together
> > or separated. That said streaming scala is considerably thin, it does not
> > really require an own maven submodule.
> >
> > Marked an older JIRA for the same issue as duplicate. [1]
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-1340
> >
> > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > To not let this discussion die, here is a concrete JIRA and a proposed
> > > layout to restructure to.
> > >
> > > What remains to be discusses is whether we want to keep the Scala/Java
> > APIs
> > > for batch/streaming in separate projects or in one project.
> > >
> > > Also, we need to find a good time to do this, when we are low on pull
> > > requests...
> > >
> > > https://issues.apache.org/jira/browse/FLINK-1712
> > >
> > > Proposed Layout:
> > >
> > > - flink-hadoop (shaded fat jar)
> > >
> > > - Core (Core and Java and Scala)
> > > - Streaming (core + java + scala)
> > > - Runtime
> > > - Client (Client + Optimizer)
> > >
> > > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > > - Tests (test-utils (compile) and tests (test))
> > >
> > > - Quickstarts
> > > - Quickstart Java
> > > - Quickstart Scala
> > >
> > > - connectors / Input/Output Formats
> > > - Avro
> > > - HBase
> > > - HadoopCompartibility
> > > - HCatalogue
> > > - JDBC
> > > - kafka
> > > - rabbit
> > > - ...
> > >
> > > - staging
> > > - Gelly
> > > - Gilbert (ML)
> > > - spargel (deprecated)
> > > - expression API
> > >
> > > - contrib
> > >
> > > - yarn
> > >
> > > - dist
> > >
> > > - yarn tests
> > >
> > > - java 8
> > >
> > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <henry.saputra@gmail.com
> >
> > > wrote:
> > >
> > > > Thanks Marton, having 2 threads discussing same thing can be
> confusing.
> > > >
> > > > - Henry
> > > >
> > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
> > > > wrote:
> > > > > Let us consider this thread the standard for the restructure, it is
> > > > > perfectly in line with the wishes I have posted.
> > > > >
> > > > > +1 for keeping the 'flink-' prefix.
> > > > >
> > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > > henry.saputra@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> > responses.
> > > > >>
> > > > >> - Henry
> > > > >>
> > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >> > I think this works well together with Marton's restructuring.
> > > > >> >
> > > > >> > I would vote to keep the "flink-" prefix, because it guarantees
> > that
> > > > the
> > > > >> > produced jars are prefixed with "flink-". Otherwise, we will
> have
> > to
> > > > >> start
> > > > >> > configuring a lot...
> > > > >> >
> > > > >> > Greetings,
> > > > >> > Stephan
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > > henry.saputra@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Will this conflict with Marton's restructuring proposal which
> > > happens
> > > > >> >> in another thread (see "Project restructure" thread in the dev@
> > > > list).
> > > > >> >>
> > > > >> >> Since we are doing refactoring, may I suggest that we also
> remove
> > > > >> >> "flink-" prefix since maven group name will indicate it is part
> > of
> > > > >> >> Flink.
> > > > >> >>
> > > > >> >> - Henry
> > > > >> >>
> > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > >> >> > Hi everyone!
> > > > >> >> >
> > > > >> >> > I think that by now, quite a bit of the maven project
> structure
> > > > can be
> > > > >> >> > improved to get rid of some legacy artifacts. Especially the
> > > > >> >> "flink-addons"
> > > > >> >> > project seems to be a catch-all place for various projects.
> > > > >> >> >
> > > > >> >> > Here is a suggestion what we could do:
> > > > >> >> >
> > > > >> >> > 1) Move "flink-yarn" to the root.
> > > > >> >> >
> > > > >> >> > 2) Move "flink-streaming" to the root (this is planned
> anyways
> > > for
> > > > the
> > > > >> >> next
> > > > >> >> > release)
> > > > >> >> >
> > > > >> >> > 3) Create a project "flink-connectors", which will contain
> > > "avro",
> > > > >> >> "jdbc",
> > > > >> >> > and "hbase". Should we have them as separate sub-projects, or
> > as
> > > > one
> > > > >> >> > project?
> > > > >> >> >
> > > > >> >> > 4) Consolidate the examples into a single project
> > > "flink-examples",
> > > > >> where
> > > > >> >> > Java, Scala, Streaming examples exist in different packages.
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Greetings,
> > > > >> >> > Stephan
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
Re: Restructuring the maven projects
Posted by Ufuk Celebi <uc...@apache.org>.
+1 I like the proposed structure.
The only thing I was wondering about is whether to name "core" => "batch"?
On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <ba...@gmail.com>
wrote:
> +1 for the proposed structure.
>
> I have no explicit preference for having batch and streaming scala together
> or separated. That said streaming scala is considerably thin, it does not
> really require an own maven submodule.
>
> Marked an older JIRA for the same issue as duplicate. [1]
>
> [1] https://issues.apache.org/jira/browse/FLINK-1340
>
> On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org> wrote:
>
> > To not let this discussion die, here is a concrete JIRA and a proposed
> > layout to restructure to.
> >
> > What remains to be discusses is whether we want to keep the Scala/Java
> APIs
> > for batch/streaming in separate projects or in one project.
> >
> > Also, we need to find a good time to do this, when we are low on pull
> > requests...
> >
> > https://issues.apache.org/jira/browse/FLINK-1712
> >
> > Proposed Layout:
> >
> > - flink-hadoop (shaded fat jar)
> >
> > - Core (Core and Java and Scala)
> > - Streaming (core + java + scala)
> > - Runtime
> > - Client (Client + Optimizer)
> >
> > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > - Tests (test-utils (compile) and tests (test))
> >
> > - Quickstarts
> > - Quickstart Java
> > - Quickstart Scala
> >
> > - connectors / Input/Output Formats
> > - Avro
> > - HBase
> > - HadoopCompartibility
> > - HCatalogue
> > - JDBC
> > - kafka
> > - rabbit
> > - ...
> >
> > - staging
> > - Gelly
> > - Gilbert (ML)
> > - spargel (deprecated)
> > - expression API
> >
> > - contrib
> >
> > - yarn
> >
> > - dist
> >
> > - yarn tests
> >
> > - java 8
> >
> > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <he...@gmail.com>
> > wrote:
> >
> > > Thanks Marton, having 2 threads discussing same thing can be confusing.
> > >
> > > - Henry
> > >
> > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
> > > wrote:
> > > > Let us consider this thread the standard for the restructure, it is
> > > > perfectly in line with the wishes I have posted.
> > > >
> > > > +1 for keeping the 'flink-' prefix.
> > > >
> > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> > henry.saputra@gmail.com>
> > > > wrote:
> > > >
> > > >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> responses.
> > > >>
> > > >> - Henry
> > > >>
> > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >> > I think this works well together with Marton's restructuring.
> > > >> >
> > > >> > I would vote to keep the "flink-" prefix, because it guarantees
> that
> > > the
> > > >> > produced jars are prefixed with "flink-". Otherwise, we will have
> to
> > > >> start
> > > >> > configuring a lot...
> > > >> >
> > > >> > Greetings,
> > > >> > Stephan
> > > >> >
> > > >> >
> > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > > henry.saputra@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Will this conflict with Marton's restructuring proposal which
> > happens
> > > >> >> in another thread (see "Project restructure" thread in the dev@
> > > list).
> > > >> >>
> > > >> >> Since we are doing refactoring, may I suggest that we also remove
> > > >> >> "flink-" prefix since maven group name will indicate it is part
> of
> > > >> >> Flink.
> > > >> >>
> > > >> >> - Henry
> > > >> >>
> > > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > >> >> > Hi everyone!
> > > >> >> >
> > > >> >> > I think that by now, quite a bit of the maven project structure
> > > can be
> > > >> >> > improved to get rid of some legacy artifacts. Especially the
> > > >> >> "flink-addons"
> > > >> >> > project seems to be a catch-all place for various projects.
> > > >> >> >
> > > >> >> > Here is a suggestion what we could do:
> > > >> >> >
> > > >> >> > 1) Move "flink-yarn" to the root.
> > > >> >> >
> > > >> >> > 2) Move "flink-streaming" to the root (this is planned anyways
> > for
> > > the
> > > >> >> next
> > > >> >> > release)
> > > >> >> >
> > > >> >> > 3) Create a project "flink-connectors", which will contain
> > "avro",
> > > >> >> "jdbc",
> > > >> >> > and "hbase". Should we have them as separate sub-projects, or
> as
> > > one
> > > >> >> > project?
> > > >> >> >
> > > >> >> > 4) Consolidate the examples into a single project
> > "flink-examples",
> > > >> where
> > > >> >> > Java, Scala, Streaming examples exist in different packages.
> > > >> >> >
> > > >> >> >
> > > >> >> > Greetings,
> > > >> >> > Stephan
> > > >> >>
> > > >>
> > >
> >
>
Re: Restructuring the maven projects
Posted by Márton Balassi <ba...@gmail.com>.
+1 for the proposed structure.
I have no explicit preference for having batch and streaming scala together
or separated. That said streaming scala is considerably thin, it does not
really require an own maven submodule.
Marked an older JIRA for the same issue as duplicate. [1]
[1] https://issues.apache.org/jira/browse/FLINK-1340
On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <se...@apache.org> wrote:
> To not let this discussion die, here is a concrete JIRA and a proposed
> layout to restructure to.
>
> What remains to be discusses is whether we want to keep the Scala/Java APIs
> for batch/streaming in separate projects or in one project.
>
> Also, we need to find a good time to do this, when we are low on pull
> requests...
>
> https://issues.apache.org/jira/browse/FLINK-1712
>
> Proposed Layout:
>
> - flink-hadoop (shaded fat jar)
>
> - Core (Core and Java and Scala)
> - Streaming (core + java + scala)
> - Runtime
> - Client (Client + Optimizer)
>
> - Examples (Java + Scala + Streaming Java + Streaming Scala)
> - Tests (test-utils (compile) and tests (test))
>
> - Quickstarts
> - Quickstart Java
> - Quickstart Scala
>
> - connectors / Input/Output Formats
> - Avro
> - HBase
> - HadoopCompartibility
> - HCatalogue
> - JDBC
> - kafka
> - rabbit
> - ...
>
> - staging
> - Gelly
> - Gilbert (ML)
> - spargel (deprecated)
> - expression API
>
> - contrib
>
> - yarn
>
> - dist
>
> - yarn tests
>
> - java 8
>
> On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <he...@gmail.com>
> wrote:
>
> > Thanks Marton, having 2 threads discussing same thing can be confusing.
> >
> > - Henry
> >
> > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
> > wrote:
> > > Let us consider this thread the standard for the restructure, it is
> > > perfectly in line with the wishes I have posted.
> > >
> > > +1 for keeping the 'flink-' prefix.
> > >
> > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> henry.saputra@gmail.com>
> > > wrote:
> > >
> > >> @Robert, and @Stephan, sure I am ok with it, thanks for the responses.
> > >>
> > >> - Henry
> > >>
> > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org>
> wrote:
> > >> > I think this works well together with Marton's restructuring.
> > >> >
> > >> > I would vote to keep the "flink-" prefix, because it guarantees that
> > the
> > >> > produced jars are prefixed with "flink-". Otherwise, we will have to
> > >> start
> > >> > configuring a lot...
> > >> >
> > >> > Greetings,
> > >> > Stephan
> > >> >
> > >> >
> > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> > henry.saputra@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Will this conflict with Marton's restructuring proposal which
> happens
> > >> >> in another thread (see "Project restructure" thread in the dev@
> > list).
> > >> >>
> > >> >> Since we are doing refactoring, may I suggest that we also remove
> > >> >> "flink-" prefix since maven group name will indicate it is part of
> > >> >> Flink.
> > >> >>
> > >> >> - Henry
> > >> >>
> > >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <se...@apache.org>
> > wrote:
> > >> >> > Hi everyone!
> > >> >> >
> > >> >> > I think that by now, quite a bit of the maven project structure
> > can be
> > >> >> > improved to get rid of some legacy artifacts. Especially the
> > >> >> "flink-addons"
> > >> >> > project seems to be a catch-all place for various projects.
> > >> >> >
> > >> >> > Here is a suggestion what we could do:
> > >> >> >
> > >> >> > 1) Move "flink-yarn" to the root.
> > >> >> >
> > >> >> > 2) Move "flink-streaming" to the root (this is planned anyways
> for
> > the
> > >> >> next
> > >> >> > release)
> > >> >> >
> > >> >> > 3) Create a project "flink-connectors", which will contain
> "avro",
> > >> >> "jdbc",
> > >> >> > and "hbase". Should we have them as separate sub-projects, or as
> > one
> > >> >> > project?
> > >> >> >
> > >> >> > 4) Consolidate the examples into a single project
> "flink-examples",
> > >> where
> > >> >> > Java, Scala, Streaming examples exist in different packages.
> > >> >> >
> > >> >> >
> > >> >> > Greetings,
> > >> >> > Stephan
> > >> >>
> > >>
> >
>
Re: Restructuring the maven projects
Posted by Stephan Ewen <se...@apache.org>.
I am reworking the web frontend a bit (PR may come in a bit) and was
thinking the same thing.
Would also allow to vastly reduce the runtime dependencies, since all the
webserver stuff would be out...
On Sun, Mar 29, 2015 at 9:11 PM, Henry Saputra <he...@gmail.com>
wrote:
> Also looks like runtime getting too big.
>
> Thoughts about moving the web info frontend to separate maven module?
>
> - Henry
>
>
> On Tue, Mar 17, 2015 at 2:46 AM, Stephan Ewen <se...@apache.org> wrote:
> > To not let this discussion die, here is a concrete JIRA and a proposed
> > layout to restructure to.
> >
> > What remains to be discusses is whether we want to keep the Scala/Java
> APIs
> > for batch/streaming in separate projects or in one project.
> >
> > Also, we need to find a good time to do this, when we are low on pull
> > requests...
> >
> > https://issues.apache.org/jira/browse/FLINK-1712
> >
> > Proposed Layout:
> >
> > - flink-hadoop (shaded fat jar)
> >
> > - Core (Core and Java and Scala)
> > - Streaming (core + java + scala)
> > - Runtime
> > - Client (Client + Optimizer)
> >
> > - Examples (Java + Scala + Streaming Java + Streaming Scala)
> > - Tests (test-utils (compile) and tests (test))
> >
> > - Quickstarts
> > - Quickstart Java
> > - Quickstart Scala
> >
> > - connectors / Input/Output Formats
> > - Avro
> > - HBase
> > - HadoopCompartibility
> > - HCatalogue
> > - JDBC
> > - kafka
> > - rabbit
> > - ...
> >
> > - staging
> > - Gelly
> > - Gilbert (ML)
> > - spargel (deprecated)
> > - expression API
> >
> > - contrib
> >
> > - yarn
> >
> > - dist
> >
> > - yarn tests
> >
> > - java 8
> >
> > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <he...@gmail.com>
> > wrote:
> >
> >> Thanks Marton, having 2 threads discussing same thing can be confusing.
> >>
> >> - Henry
> >>
> >> On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
> >> wrote:
> >> > Let us consider this thread the standard for the restructure, it is
> >> > perfectly in line with the wishes I have posted.
> >> >
> >> > +1 for keeping the 'flink-' prefix.
> >> >
> >> > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <
> henry.saputra@gmail.com>
> >> > wrote:
> >> >
> >> >> @Robert, and @Stephan, sure I am ok with it, thanks for the
> responses.
> >> >>
> >> >> - Henry
> >> >>
> >> >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >> >> > I think this works well together with Marton's restructuring.
> >> >> >
> >> >> > I would vote to keep the "flink-" prefix, because it guarantees
> that
> >> the
> >> >> > produced jars are prefixed with "flink-". Otherwise, we will have
> to
> >> >> start
> >> >> > configuring a lot...
> >> >> >
> >> >> > Greetings,
> >> >> > Stephan
> >> >> >
> >> >> >
> >> >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
> >> henry.saputra@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Will this conflict with Marton's restructuring proposal which
> happens
> >> >> >> in another thread (see "Project restructure" thread in the dev@
> >> list).
> >> >> >>
> >> >> >> Since we are doing refactoring, may I suggest that we also remove
> >> >> >> "flink-" prefix since maven group name will indicate it is part of
> >> >> >> Flink.
> >> >> >>
> >> >> >> - Henry
> >> >> >>
> >> >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <se...@apache.org>
> >> wrote:
> >> >> >> > Hi everyone!
> >> >> >> >
> >> >> >> > I think that by now, quite a bit of the maven project structure
> >> can be
> >> >> >> > improved to get rid of some legacy artifacts. Especially the
> >> >> >> "flink-addons"
> >> >> >> > project seems to be a catch-all place for various projects.
> >> >> >> >
> >> >> >> > Here is a suggestion what we could do:
> >> >> >> >
> >> >> >> > 1) Move "flink-yarn" to the root.
> >> >> >> >
> >> >> >> > 2) Move "flink-streaming" to the root (this is planned anyways
> for
> >> the
> >> >> >> next
> >> >> >> > release)
> >> >> >> >
> >> >> >> > 3) Create a project "flink-connectors", which will contain
> "avro",
> >> >> >> "jdbc",
> >> >> >> > and "hbase". Should we have them as separate sub-projects, or as
> >> one
> >> >> >> > project?
> >> >> >> >
> >> >> >> > 4) Consolidate the examples into a single project
> "flink-examples",
> >> >> where
> >> >> >> > Java, Scala, Streaming examples exist in different packages.
> >> >> >> >
> >> >> >> >
> >> >> >> > Greetings,
> >> >> >> > Stephan
> >> >> >>
> >> >>
> >>
>
Re: Restructuring the maven projects
Posted by Henry Saputra <he...@gmail.com>.
Also looks like runtime getting too big.
Thoughts about moving the web info frontend to separate maven module?
- Henry
On Tue, Mar 17, 2015 at 2:46 AM, Stephan Ewen <se...@apache.org> wrote:
> To not let this discussion die, here is a concrete JIRA and a proposed
> layout to restructure to.
>
> What remains to be discusses is whether we want to keep the Scala/Java APIs
> for batch/streaming in separate projects or in one project.
>
> Also, we need to find a good time to do this, when we are low on pull
> requests...
>
> https://issues.apache.org/jira/browse/FLINK-1712
>
> Proposed Layout:
>
> - flink-hadoop (shaded fat jar)
>
> - Core (Core and Java and Scala)
> - Streaming (core + java + scala)
> - Runtime
> - Client (Client + Optimizer)
>
> - Examples (Java + Scala + Streaming Java + Streaming Scala)
> - Tests (test-utils (compile) and tests (test))
>
> - Quickstarts
> - Quickstart Java
> - Quickstart Scala
>
> - connectors / Input/Output Formats
> - Avro
> - HBase
> - HadoopCompartibility
> - HCatalogue
> - JDBC
> - kafka
> - rabbit
> - ...
>
> - staging
> - Gelly
> - Gilbert (ML)
> - spargel (deprecated)
> - expression API
>
> - contrib
>
> - yarn
>
> - dist
>
> - yarn tests
>
> - java 8
>
> On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <he...@gmail.com>
> wrote:
>
>> Thanks Marton, having 2 threads discussing same thing can be confusing.
>>
>> - Henry
>>
>> On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <mb...@apache.org>
>> wrote:
>> > Let us consider this thread the standard for the restructure, it is
>> > perfectly in line with the wishes I have posted.
>> >
>> > +1 for keeping the 'flink-' prefix.
>> >
>> > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra <he...@gmail.com>
>> > wrote:
>> >
>> >> @Robert, and @Stephan, sure I am ok with it, thanks for the responses.
>> >>
>> >> - Henry
>> >>
>> >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen <se...@apache.org> wrote:
>> >> > I think this works well together with Marton's restructuring.
>> >> >
>> >> > I would vote to keep the "flink-" prefix, because it guarantees that
>> the
>> >> > produced jars are prefixed with "flink-". Otherwise, we will have to
>> >> start
>> >> > configuring a lot...
>> >> >
>> >> > Greetings,
>> >> > Stephan
>> >> >
>> >> >
>> >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry Saputra <
>> henry.saputra@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Will this conflict with Marton's restructuring proposal which happens
>> >> >> in another thread (see "Project restructure" thread in the dev@
>> list).
>> >> >>
>> >> >> Since we are doing refactoring, may I suggest that we also remove
>> >> >> "flink-" prefix since maven group name will indicate it is part of
>> >> >> Flink.
>> >> >>
>> >> >> - Henry
>> >> >>
>> >> >> On Fri, Jan 2, 2015 at 6:52 AM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >> >> > Hi everyone!
>> >> >> >
>> >> >> > I think that by now, quite a bit of the maven project structure
>> can be
>> >> >> > improved to get rid of some legacy artifacts. Especially the
>> >> >> "flink-addons"
>> >> >> > project seems to be a catch-all place for various projects.
>> >> >> >
>> >> >> > Here is a suggestion what we could do:
>> >> >> >
>> >> >> > 1) Move "flink-yarn" to the root.
>> >> >> >
>> >> >> > 2) Move "flink-streaming" to the root (this is planned anyways for
>> the
>> >> >> next
>> >> >> > release)
>> >> >> >
>> >> >> > 3) Create a project "flink-connectors", which will contain "avro",
>> >> >> "jdbc",
>> >> >> > and "hbase". Should we have them as separate sub-projects, or as
>> one
>> >> >> > project?
>> >> >> >
>> >> >> > 4) Consolidate the examples into a single project "flink-examples",
>> >> where
>> >> >> > Java, Scala, Streaming examples exist in different packages.
>> >> >> >
>> >> >> >
>> >> >> > Greetings,
>> >> >> > Stephan
>> >> >>
>> >>
>>