You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Maximilian Michels <mx...@apache.org> on 2016/01/21 17:51:44 UTC
Re: Scala 2.10/2.11 Maven dependencies
https://issues.apache.org/jira/browse/FLINK-2940
There is now a pending pull request: https://github.com/apache/flink/pull/1529
As I was working on the changes, I discovered we have some more
modules which have a Scala dependency which could be avoided. Namely
these are "flink-java8", and "flink-streaming-java". The latter
probably affects a lot of users. So it would be nice to make it
Scala-free.
Do you think that would be feasible?
Cheers,
Max
On Mon, Nov 2, 2015 at 9:01 PM, Fabian Hueske <fh...@gmail.com> wrote:
> +1 for that.
>
> 2015-11-02 20:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> +1 for Max' suggestion to fix that for 1.0 (next release).
>>
>> Hot fixing of this thing so short before a release is a bit risky in my
>> opinion. It is easy to make errors (overlooking something, error not
>> visible because of cached older dependencies, ...), it happened more than
>> once during version upgrades, maven project re-organizations, etc.
>>
>> Doing it after 0.10 and having a few week to let it sink in and errors
>> surface would probably be much safer...
>>
>> On Mon, Nov 2, 2015 at 5:19 AM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Ah OK. Sorry, I misunderstood your intention.
>> >
>> > 2015-11-02 14:07 GMT+01:00 Maximilian Michels <mx...@apache.org>:
>> >
>> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
>> > artifacts
>> > > > (and others that depend on flink-java and have no other Scala
>> > dependency)
>> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
>> > >
>> > >
>> > > My suggestion was to keep the Scala unsuffixed Scala 2.10 and add a
>> > > suffix for Scala 2.11. That's the way we are currently doing it (also
>> > > deployed on Maven like this). For the next release after 0.10, we can
>> > > do it properly.
>> > >
>> > > On Mon, Nov 2, 2015 at 1:30 PM, Chiwan Park <ch...@apache.org>
>> > wrote:
>> > > > If we choose selective Scala version suffix for artifacts, we have to
>> > > tell which artifacts have the version suffix to newcomers. Some
>> artifacts
>> > > such as "flink-java”, "flink-streaming-java" are easily recognized. But
>> > > IMO, knowing whether artifacts such as "flink-ml", "flink-clients",
>> > > "flink-table" have the version suffix or not is difficult for
>> newcomers.
>> > > >
>> > > > This is why we are adding the version suffix to all Scala 2.11
>> > artifacts
>> > > currently. For Scala 2.10 artifacts, we aren’t adding the version
>> suffix
>> > > for Flink with Java users.
>> > > >
>> > > > I’m for adding the version suffix to Scala 2.10 artifacts also. But
>> I’m
>> > > not sure that removing the version suffix from Java-only artifacts
>> would
>> > be
>> > > good. As I said above, It seems difficult for newcomers.
>> > > >
>> > > > Regards,
>> > > > Chiwan Park
>> > > >
>> > > > On November 2, 2015 at 8:19:15 PM, Fabian Hueske (fhueske@gmail.com)
>> > > wrote:
>> > > >
>> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
>> > artifacts
>> > > > (and others that depend on flink-java and have no other Scala
>> > dependency)
>> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
>> > > >
>> > > > Do we want that?
>> > > >
>> > > > 2015-11-02 11:37 GMT+01:00 Maximilian Michels <mx...@apache.org>:
>> > > >
>> > > >> I'm for leaving it as-is and renaming all artifacts which depend on
>> > > >> Scala for the release following 0.10.
>> > > >>
>> > > >> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fh...@gmail.com>
>> > > wrote:
>> > > >> > OK, let me try to summarize the discussion (and please correct me
>> > if I
>> > > >> got
>> > > >> > something wrong).
>> > > >> >
>> > > >> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have
>> > to
>> > > >> > release 2.11 artifacts for the 0.10.0 release version as well.
>> > > >> >
>> > > >> > 2) Everybody agrees to appropriately tag all artifacts that have a
>> > > >> > (transitive) Scala dependency. ATM, that would also include
>> > flink-java
>> > > >> > which is a bit awkward. The Scala dependency in flink-java
>> > originates
>> > > >> from
>> > > >> > the Chill library which is used to obtain a Kryo serializer which
>> is
>> > > >> > initialized with serializers for Scala classes. We could resolve
>> > this
>> > > >> issue
>> > > >> > by providing Java and Scala specific implementations of the Kryo
>> > > >> > serializers and have KryoTypeInfos for Java and Scala.
>> > > >> >
>> > > >> > The question to answer right now is, do we want to have
>> "correctly"
>> > > >> labeled
>> > > >> > artifacts for the next 0.10.0 release or do we defer that for 1.0?
>> > > >> > If we want to solve it for 0.10.0 we need to cancel the current RC
>> > and
>> > > >> > provide a fix to remove the Scala dependency in flink-java, IMO.
>> > > >> >
>> > > >> > Opinions?
>> > > >> >
>> > > >> > Cheers, Fabian
>> > > >> >
>> > > >> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <se...@apache.org>:
>> > > >> >
>> > > >> >> +1 for the approach discusses here, and for removing Scala
>> > > dependencies
>> > > >> >> from modules that can be Scala independent.
>> > > >> >>
>> > > >> >> It would be nice if pure Java users would not see any Scala
>> > > versioning
>> > > >> (on
>> > > >> >> flink-core, flink-java, later also flink-sreaming-java). I guess
>> > for
>> > > any
>> > > >> >> runtime-related parts (including flink-client and currently all
>> > > >> streaming
>> > > >> >> projects), we need the Scala versions...
>> > > >> >>
>> > > >> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <
>> mxm@apache.org
>> > >
>> > > >> wrote:
>> > > >> >>
>> > > >> >> > Good point. Didn't know that. We can still add them for the
>> > > release.
>> > > >> >> >
>> > > >> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
>> > > >> >> > <al...@gmail.com> wrote:
>> > > >> >> > > My two cents - there are already Maven artifacts deployed for
>> > > 2.11
>> > > >> in
>> > > >> >> the
>> > > >> >> > > SNAPSHOT repository. I think it might be confusing if they
>> > > suddenly
>> > > >> >> > > disappear for the stable release.
>> > > >> >> > >
>> > > >> >> > >
>> > > >> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <
>> mxm@apache.org
>> > >:
>> > > >> >> > >
>> > > >> >> > >> Seems like we agree that we need artifacts for different
>> > > versions
>> > > >> of
>> > > >> >> > Scala
>> > > >> >> > >> on Maven. There also seems to be a preference for including
>> > the
>> > > >> >> version
>> > > >> >> > in
>> > > >> >> > >> the artifact name.
>> > > >> >> > >>
>> > > >> >> > >> I've created an issue and marked it to be resolved for 1.0.
>> > For
>> > > the
>> > > >> >> 0.10
>> > > >> >> > >> release, we will have binaries but no Maven artifacts. The
>> > > biggest
>> > > >> >> > >> challenge I see is to remove Scala from as many modules as
>> > > >> possible.
>> > > >> >> For
>> > > >> >> > >> example, flink-java depends on Scala at the moment..
>> > > >> >> > >>
>> > > >> >> > >> https://issues.apache.org/jira/browse/FLINK-2940
>> > > >> >> > >>
>> > > >> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz IV <
>> > > >> >> > fkautz@redhat.com>
>> > > >> >> > >> wrote:
>> > > >> >> > >>
>> > > >> >> > >> > No idea if I get a vote ;) Nevertheless, +1 to have
>> binaries
>> > > for
>> > > >> >> both
>> > > >> >> > >> > versions in Maven and explicitly "scala versioned".
>> > > >> >> > >> >
>> > > >> >> > >> > Some background on this for those not as familiar with
>> scala
>> > > >> >> > versioning:
>> > > >> >> > >> >
>> > > >> >> > >> > It's considered best practice to label what version of
>> > scala a
>> > > >> >> library
>> > > >> >> > >> > uses in the artifact id.
>> > > >> >> > >> >
>> > > >> >> > >> > The reason is compiled scala code is only compatible with
>> > the
>> > > >> major
>> > > >> >> > >> > version of scala it was compiled for. For example, a
>> library
>> > > >> >> > compatible
>> > > >> >> > >> > with 2.10 is not compatible with 2.11. The same will be
>> true
>> > > with
>> > > >> >> 2.12
>> > > >> >> > >> once
>> > > >> >> > >> > it is released. Mixing versions will result in undefined
>> > > behavior
>> > > >> >> > which
>> > > >> >> > >> > will likely manifest itself as runtime exceptions.
>> > > >> >> > >> >
>> > > >> >> > >> > The convention to fix this problem is for all published
>> > > >> libraries to
>> > > >> >> > >> > specify the version of scala they are compatible with.
>> > Leaving
>> > > >> out
>> > > >> >> the
>> > > >> >> > >> > scala version in a library is akin to saying "We don't
>> > depend
>> > > on
>> > > >> >> scala
>> > > >> >> > >> for
>> > > >> >> > >> > this library, so feel free to use whatever you want." Sbt
>> > > users
>> > > >> will
>> > > >> >> > >> > typically specify the version of scala they use and
>> tooling
>> > is
>> > > >> built
>> > > >> >> > >> around
>> > > >> >> > >> > ensuring consistency with the "%%" operator.
>> > > >> >> > >> >
>> > > >> >> > >> > E.g.
>> > > >> >> > >> >
>> > > >> >> > >> > scalaVersion := "2.11.4"
>> > > >> >> > >> >
>> > > >> >> > >> > // this resolves to to artifactID: "scalacheck_2.11"
>> > > >> >> > >> > libraryDependencies += "org.scalacheck" %% "scalacheck" %
>> > > >> "1.12.0" %
>> > > >> >> > >> "test"
>> > > >> >> > >> >
>> > > >> >> > >> > The most important part of this is that the scala version
>> is
>> > > >> >> explicit
>> > > >> >> > >> > which eliminates the problem for downstream users.
>> > > >> >> > >> >
>> > > >> >> > >> > Cheers,
>> > > >> >> > >> > Frederick
>> > > >> >> > >> >
>> > > >> >> > >> >
>> > > >> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote:
>> > > >> >> > >> >
>> > > >> >> > >> >> +1 to have binaries for both versions in Maven and as
>> build
>> > > to
>> > > >> >> > download.
>> > > >> >> > >> >>
>> > > >> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis <
>> > > >> >> > >> >> theodoros.vasiloudis@gmail.com>:
>> > > >> >> > >> >>
>> > > >> >> > >> >> +1 for having binaries, I'm working on a Spark
>> application
>> > > >> >> currently
>> > > >> >> > >> with
>> > > >> >> > >> >>> Scala 2.11 and having to rebuild everything when
>> deploying
>> > > >> e.g. to
>> > > >> >> > EC2
>> > > >> >> > >> >>> is a
>> > > >> >> > >> >>> pain.
>> > > >> >> > >> >>>
>> > > >> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk Celebi <
>> > > uce@apache.org>
>> > > >> >> > wrote:
>> > > >> >> > >> >>>
>> > > >> >> > >> >>> I agree with Till, but is this something you want to
>> > > address in
>> > > >> >> this
>> > > >> >> > >> >>>> release already?
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>> I would postpone it to 1.0.0.
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>> – Ufuk
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>> On 26 Oct 2015, at 16:17, Till Rohrmann <
>> > > trohrmann@apache.org
>> > > >> >
>> > > >> >> > wrote:
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>>> I would be in favor of deploying also Scala 2.11
>> > > artifacts to
>> > > >> >> > Maven
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> since
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> more and more people will try out Flink with Scala
>> 2.11.
>> > > >> Having
>> > > >> >> the
>> > > >> >> > >> >>>>> dependencies in the Maven repository makes it
>> > considerably
>> > > >> >> easier
>> > > >> >> > for
>> > > >> >> > >> >>>>> people to get their Flink jobs running.
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>>> Furthermore, I observed that people are not aware that
>> > our
>> > > >> >> > deployed
>> > > >> >> > >> >>>>> artifacts, e.g. flink-runtime, are built with Scala
>> > 2.10.
>> > > As
>> > > >> a
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> consequence,
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> they mix flink dependencies with other dependencies
>> > > pulling
>> > > >> in
>> > > >> >> > Scala
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> 2.11
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> and then they wonder that the program crashes. It would
>> > be,
>> > > >> imho,
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> clearer
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> if all our dependencies which depend on a specific
>> Scala
>> > > >> version
>> > > >> >> > would
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> have
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> the corresponding Scala suffix appended.
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>>> Adding the 2.10 suffix would also spare us the hassle
>> of
>> > > >> >> upgrading
>> > > >> >> > >> to a
>> > > >> >> > >> >>>>> newer Scala version in the future, because then the
>> > > artifacts
>> > > >> >> > >> wouldn't
>> > > >> >> > >> >>>>> share the same artifact name.
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>>> Cheers,
>> > > >> >> > >> >>>>> Till
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>>> On Mon, Oct 26, 2015 at 4:04 PM, Maximilian Michels <
>> > > >> >> > mxm@apache.org>
>> > > >> >> > >> >>>>>
>> > > >> >> > >> >>>> wrote:
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> Hi Flinksters,
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> We have recently committed an easy way to change
>> > Flink's
>> > > >> Scala
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> version.
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> The
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> question arises now whether we should ship Scala 2.11
>> as
>> > > >> >> binaries
>> > > >> >> > and
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> via
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> Maven. For the rc0, I created all binaries twice, for
>> > > Scala
>> > > >> 2.10
>> > > >> >> > and
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> 2.11.
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> However, I didn't create Maven artifacts. This follows
>> > our
>> > > >> >> current
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> shipping
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> strategy where we only ship Hadoop1 and Hadoop 2.3.0
>> > Maven
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> dependencies
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> but
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> additionally Hadoop 2.4, 2.6, 2.7 as binaries.
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> Should we also upload Maven dependencies for Scala
>> > 2.11?
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> If so, the next question arises: What version pattern
>> > > >> should we
>> > > >> >> > have
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> for
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> the Flink Scala 2.11 dependencies? For Hadoop, we
>> append
>> > > >> -hadoop1
>> > > >> >> > to
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> the
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> VERSION, e.g. artifactID=flink-core,
>> > version=0.9.1-hadoop1.
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> However, it is common practice to append the suffix
>> to
>> > > the
>> > > >> >> > >> artifactID
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> of
>> > > >> >> > >> >>>
>> > > >> >> > >> >>>> the Maven dependency, e.g. artifactID=flink-core_2.11,
>> > > >> >> > version=0.9.1.
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> This
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> has mostly historic reasons but is widely used.
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> Whatever naming pattern we choose, it should be
>> > > consistent.
>> > > >> I
>> > > >> >> > would
>> > > >> >> > >> be
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>> in
>> > > >> >> > >> >>>>
>> > > >> >> > >> >>>>> favor of changing our artifact names to contain the
>> > Hadoop
>> > > >> and
>> > > >> >> > Scala
>> > > >> >> > >> >>>>>> version. This would also imply that all Scala
>> dependent
>> > > >> Maven
>> > > >> >> > >> modules
>> > > >> >> > >> >>>>>> receive a Scala suffix (also the default Scala 2.10
>> > > >> modules).
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>> Cheers,
>> > > >> >> > >> >>>>>> Max
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>>>
>> > > >> >> > >> >>>>
>> > > >> >> > >> >
>> > > >> >> > >>
>> > > >> >> >
>> > > >> >>
>> > > >>
>> > >
>> >
>>
Re: Scala 2.10/2.11 Maven dependencies
Posted by Till Rohrmann <tr...@apache.org>.
+1
On Mon, Jan 25, 2016 at 3:36 PM, Stephan Ewen <se...@apache.org> wrote:
> +1
>
> On Mon, Jan 25, 2016 at 11:41 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > > On 25 Jan 2016, at 11:39, Maximilian Michels <mx...@apache.org> wrote:
> > >
> > > I won't have the time to finish the refactoring. Also, it will be
> > > pretty painful with all the large streaming pull requests being merged
> > > at the moment. If there are no objections, I would like to merge the
> > > Scala suffix changes with "flink-streaming-java" being Scala
> > > dependent. It will improve the experience for users in the long run.
> > >
> > > After merging, I'll announce the new Scala suffixed modules. In
> > > addition, we could deploy an empty Maven module to overwrite the
> > > current snapshot version of flink-streaming-java. That would prevent
> > > conflicts with different snapshot versions, e.g. combining
> > > flink-streaming-java (instead of flink-streaming-java_2.11) with
> > > flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> > > problem anymore.
> > >
> > > Any objections?
> >
> > Sounds good to me!
> >
> > – Ufuk
> >
> >
>
Re: Scala 2.10/2.11 Maven dependencies
Posted by Stephan Ewen <se...@apache.org>.
+1
On Mon, Jan 25, 2016 at 11:41 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > On 25 Jan 2016, at 11:39, Maximilian Michels <mx...@apache.org> wrote:
> >
> > I won't have the time to finish the refactoring. Also, it will be
> > pretty painful with all the large streaming pull requests being merged
> > at the moment. If there are no objections, I would like to merge the
> > Scala suffix changes with "flink-streaming-java" being Scala
> > dependent. It will improve the experience for users in the long run.
> >
> > After merging, I'll announce the new Scala suffixed modules. In
> > addition, we could deploy an empty Maven module to overwrite the
> > current snapshot version of flink-streaming-java. That would prevent
> > conflicts with different snapshot versions, e.g. combining
> > flink-streaming-java (instead of flink-streaming-java_2.11) with
> > flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> > problem anymore.
> >
> > Any objections?
>
> Sounds good to me!
>
> – Ufuk
>
>
Re: Scala 2.10/2.11 Maven dependencies
Posted by Ufuk Celebi <uc...@apache.org>.
> On 25 Jan 2016, at 11:39, Maximilian Michels <mx...@apache.org> wrote:
>
> I won't have the time to finish the refactoring. Also, it will be
> pretty painful with all the large streaming pull requests being merged
> at the moment. If there are no objections, I would like to merge the
> Scala suffix changes with "flink-streaming-java" being Scala
> dependent. It will improve the experience for users in the long run.
>
> After merging, I'll announce the new Scala suffixed modules. In
> addition, we could deploy an empty Maven module to overwrite the
> current snapshot version of flink-streaming-java. That would prevent
> conflicts with different snapshot versions, e.g. combining
> flink-streaming-java (instead of flink-streaming-java_2.11) with
> flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> problem anymore.
>
> Any objections?
Sounds good to me!
– Ufuk
Re: Scala 2.10/2.11 Maven dependencies
Posted by Maximilian Michels <mx...@apache.org>.
I won't have the time to finish the refactoring. Also, it will be
pretty painful with all the large streaming pull requests being merged
at the moment. If there are no objections, I would like to merge the
Scala suffix changes with "flink-streaming-java" being Scala
dependent. It will improve the experience for users in the long run.
After merging, I'll announce the new Scala suffixed modules. In
addition, we could deploy an empty Maven module to overwrite the
current snapshot version of flink-streaming-java. That would prevent
conflicts with different snapshot versions, e.g. combining
flink-streaming-java (instead of flink-streaming-java_2.11) with
flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
problem anymore.
Any objections?
On Fri, Jan 22, 2016 at 6:30 PM, Maximilian Michels <mx...@apache.org> wrote:
> +1 for a big notice once we merge this.
>
> I would like to have a suffix-free "flink-streaming-java". However,
> I'm having a hard time to refactor the streaming-java code to get rid
> of Scala. The streaming API depends on "flink-clients" and
> "flink-runtime" which both inherently depend on Scala. Unfortunately,
> the streaming API is very much chained with the runtime which makes it
> hard to move classes from flink-streaming-java to flink-runtime.
>
> I think the best solution would be to have similar abstraction for the
> streaming API as we have in batch. That is, to generate a Plan
> independently of the runtime which is later turned into a JobGraph. In
> streaming, we have the StreamGraph but it is not runtime independent.
> It's a bit of a mess.
>
> Another option I see is to shade away Scala. I did this just to see
> how large the binaries would be. It would let "flink-streaming-java"
> grow from 3,1 MB (without shading) to 59,7 MB (with shading). That
> seems a bit too large for users although it would get rid of the Scala
> suffix without refactoring.
>
> On Thu, Jan 21, 2016 at 11:33 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>> On 21 Jan 2016, at 17:51, Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> https://issues.apache.org/jira/browse/FLINK-2940
>>>
>>> There is now a pending pull request: https://github.com/apache/flink/pull/1529
>>>
>>> As I was working on the changes, I discovered we have some more
>>> modules which have a Scala dependency which could be avoided. Namely
>>> these are "flink-java8", and "flink-streaming-java". The latter
>>> probably affects a lot of users. So it would be nice to make it
>>> Scala-free.
>>>
>>> Do you think that would be feasible?
>>
>> Thank you very much for taking care of this. :)
>>
>> I would like to have it changed as well before merging for the same reason Fabian brought up in the PR.
>>
>> Since this will be quite a breaking change, we should consider adding a big notice to the 1.0-SNAPSHOT docs regarding this (basically on the landing page) for the few weeks until the release.
>>
>> – Ufuk
>>
Re: Scala 2.10/2.11 Maven dependencies
Posted by Maximilian Michels <mx...@apache.org>.
+1 for a big notice once we merge this.
I would like to have a suffix-free "flink-streaming-java". However,
I'm having a hard time to refactor the streaming-java code to get rid
of Scala. The streaming API depends on "flink-clients" and
"flink-runtime" which both inherently depend on Scala. Unfortunately,
the streaming API is very much chained with the runtime which makes it
hard to move classes from flink-streaming-java to flink-runtime.
I think the best solution would be to have similar abstraction for the
streaming API as we have in batch. That is, to generate a Plan
independently of the runtime which is later turned into a JobGraph. In
streaming, we have the StreamGraph but it is not runtime independent.
It's a bit of a mess.
Another option I see is to shade away Scala. I did this just to see
how large the binaries would be. It would let "flink-streaming-java"
grow from 3,1 MB (without shading) to 59,7 MB (with shading). That
seems a bit too large for users although it would get rid of the Scala
suffix without refactoring.
On Thu, Jan 21, 2016 at 11:33 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
>> On 21 Jan 2016, at 17:51, Maximilian Michels <mx...@apache.org> wrote:
>>
>> https://issues.apache.org/jira/browse/FLINK-2940
>>
>> There is now a pending pull request: https://github.com/apache/flink/pull/1529
>>
>> As I was working on the changes, I discovered we have some more
>> modules which have a Scala dependency which could be avoided. Namely
>> these are "flink-java8", and "flink-streaming-java". The latter
>> probably affects a lot of users. So it would be nice to make it
>> Scala-free.
>>
>> Do you think that would be feasible?
>
> Thank you very much for taking care of this. :)
>
> I would like to have it changed as well before merging for the same reason Fabian brought up in the PR.
>
> Since this will be quite a breaking change, we should consider adding a big notice to the 1.0-SNAPSHOT docs regarding this (basically on the landing page) for the few weeks until the release.
>
> – Ufuk
>
Re: Scala 2.10/2.11 Maven dependencies
Posted by Ufuk Celebi <uc...@apache.org>.
> On 21 Jan 2016, at 17:51, Maximilian Michels <mx...@apache.org> wrote:
>
> https://issues.apache.org/jira/browse/FLINK-2940
>
> There is now a pending pull request: https://github.com/apache/flink/pull/1529
>
> As I was working on the changes, I discovered we have some more
> modules which have a Scala dependency which could be avoided. Namely
> these are "flink-java8", and "flink-streaming-java". The latter
> probably affects a lot of users. So it would be nice to make it
> Scala-free.
>
> Do you think that would be feasible?
Thank you very much for taking care of this. :)
I would like to have it changed as well before merging for the same reason Fabian brought up in the PR.
Since this will be quite a breaking change, we should consider adding a big notice to the 1.0-SNAPSHOT docs regarding this (basically on the landing page) for the few weeks until the release.
– Ufuk