Upgrading Guava

[Julian Hyde <jh...@apache.org>]

Michael Mior would like to upgrade Guava from 14 to 15 so that we can use Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier versions. So, how to please everyone?

I have logged https://issues.apache.org/jira/browse/CALCITE-1111 <https://issues.apache.org/jira/browse/CALCITE-1111> and am suggesting solving the problem by making Guava what Maven calls a “provided” dependency. Do you think that might work?

If this works, we might try the same approach with Jackson. (Jackson is currently shaded in Avatica but not relocated.)

Julian

Re: Upgrading Guava

[Julian Hyde <jh...@apache.org>]

I was thinking of testing on the minimum (13.0.1), the default (18.0) and the maximum (19.0) and no other versions.

If this needs further discussion let’s continue at https://issues.apache.org/jira/browse/CALCITE-1111.

> On Mar 1, 2016, at 2:16 PM, Josh Elser <jo...@gmail.com> wrote:
> 
> Julian Hyde wrote:
>> I can see the wisdom of including shaded Jackson and Guava in
>> Avatica's uber jar. But for core Calcite, if as you say, applications
>> can simply override with their own version of various libraries, I
>> would prefer to not shade. I think we should test Calcite on a range
>> of Guava versions (say 11 to 19) and ship by default with a modern one
>> (say 18).
>> 
>> The same goes for Jackson, although I would not be inclined to support
>> old versions for quite so long.
> 
> That's fair. My recommendations were made with the assumption that we wouldn't want to take on that burden in Calcite itself (we know what we test with, we use that, and we insulate users from even knowing that's happening).
> 
> Testing against some ranges of versions would also be acceptable, IMO.

Re: Upgrading Guava

[Josh Elser <jo...@gmail.com>]

Julian Hyde wrote:
> I can see the wisdom of including shaded Jackson and Guava in
> Avatica's uber jar. But for core Calcite, if as you say, applications
> can simply override with their own version of various libraries, I
> would prefer to not shade. I think we should test Calcite on a range
> of Guava versions (say 11 to 19) and ship by default with a modern one
> (say 18).
>
> The same goes for Jackson, although I would not be inclined to support
> old versions for quite so long.

That's fair. My recommendations were made with the assumption that we 
wouldn't want to take on that burden in Calcite itself (we know what we 
test with, we use that, and we insulate users from even knowing that's 
happening).

Testing against some ranges of versions would also be acceptable, IMO.

Re: Upgrading Guava

[Julian Hyde <jh...@apache.org>]

Regarding including Guava classes/interfaces in Calcite's APIs. I
agree that we shouldn't wantonly include classes in APIs, we do
include the immutable collection classes in APIs, and there is value
in that. I very much doubt that the immutable collections will ever be
removed from Guava.

In my opinion, immutable collections should have been part of the JDK,
just as they are part of Scala's standard library. I treat Guava as an
extension to the JDK, and just as we can run on any version of Java
we're given, we should be able to run on any version of Guava that we
are given.

I can see the wisdom of including shaded Jackson and Guava in
Avatica's uber jar. But for core Calcite, if as you say, applications
can simply override with their own version of various libraries, I
would prefer to not shade. I think we should test Calcite on a range
of Guava versions (say 11 to 19) and ship by default with a modern one
(say 18).

The same goes for Jackson, although I would not be inclined to support
old versions for quite so long.

Julian


On Mon, Feb 29, 2016 at 8:56 PM, Jacques Nadeau <ja...@apache.org> wrote:
> I'm all for having a shaded version of the dependencies as long as
> consumers can choose whether or not they want to use the shaded version.
> Having a bunch of copies of common dependencies like Guava, Jackson,
> Protobuf, Netty, etc can make distribution tarballs for other projects
> massive.
>
> On Mon, Feb 29, 2016 at 8:39 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> I'd prefer to rely on shading instead of making it as provided. As I
>> understand it, using the provided scope doesn't really help us any more
>> than a downstream project overriding the version of Guava Calcite depends
>> upon. If downstream projects provide their own version of Guava, we have no
>> guarantee that the provided version will satisfy our needs (Guava has a
>> tendency to remove APIs in the release after they've been deprecated).
>>
>> With my history in dealing with Hadoop and friends, I'm of the opinion
>> that we should not expose these dependencies to our downstream users at
>> all. Shade+relocate, downstream projects include whatever version of Guava
>> they'd like. If we want to make sure we keep the size of the jars down, we
>> can still provide non-shaded artifacts with the express understanding that
>> users will need to provide a specific version.
>>
>> This recommendation also requires that we don't expose Guava classes via
>> any Calcite APIs (as we can't be 100% certain such classes won't disappear).
>>
>>
>> Julian Hyde wrote:
>>
>>> Michael Mior would like to upgrade Guava from 14 to 15 so that we can use
>>> Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier
>>> versions. So, how to please everyone?
>>>
>>> I have logged https://issues.apache.org/jira/browse/CALCITE-1111<
>>> https://issues.apache.org/jira/browse/CALCITE-1111>  and am suggesting
>>> solving the problem by making Guava what Maven calls a “provided”
>>> dependency. Do you think that might work?
>>>
>>> If this works, we might try the same approach with Jackson. (Jackson is
>>> currently shaded in Avatica but not relocated.)
>>>
>>> Julian
>>>
>>>
>>>

Re: Upgrading Guava

[Jacques Nadeau <ja...@apache.org>]

I'm all for having a shaded version of the dependencies as long as
consumers can choose whether or not they want to use the shaded version.
Having a bunch of copies of common dependencies like Guava, Jackson,
Protobuf, Netty, etc can make distribution tarballs for other projects
massive.

On Mon, Feb 29, 2016 at 8:39 PM, Josh Elser <jo...@gmail.com> wrote:

> I'd prefer to rely on shading instead of making it as provided. As I
> understand it, using the provided scope doesn't really help us any more
> than a downstream project overriding the version of Guava Calcite depends
> upon. If downstream projects provide their own version of Guava, we have no
> guarantee that the provided version will satisfy our needs (Guava has a
> tendency to remove APIs in the release after they've been deprecated).
>
> With my history in dealing with Hadoop and friends, I'm of the opinion
> that we should not expose these dependencies to our downstream users at
> all. Shade+relocate, downstream projects include whatever version of Guava
> they'd like. If we want to make sure we keep the size of the jars down, we
> can still provide non-shaded artifacts with the express understanding that
> users will need to provide a specific version.
>
> This recommendation also requires that we don't expose Guava classes via
> any Calcite APIs (as we can't be 100% certain such classes won't disappear).
>
>
> Julian Hyde wrote:
>
>> Michael Mior would like to upgrade Guava from 14 to 15 so that we can use
>> Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier
>> versions. So, how to please everyone?
>>
>> I have logged https://issues.apache.org/jira/browse/CALCITE-1111<
>> https://issues.apache.org/jira/browse/CALCITE-1111>  and am suggesting
>> solving the problem by making Guava what Maven calls a “provided”
>> dependency. Do you think that might work?
>>
>> If this works, we might try the same approach with Jackson. (Jackson is
>> currently shaded in Avatica but not relocated.)
>>
>> Julian
>>
>>
>>

Re: Upgrading Guava

[Josh Elser <jo...@gmail.com>]

I'd prefer to rely on shading instead of making it as provided. As I 
understand it, using the provided scope doesn't really help us any more 
than a downstream project overriding the version of Guava Calcite 
depends upon. If downstream projects provide their own version of Guava, 
we have no guarantee that the provided version will satisfy our needs 
(Guava has a tendency to remove APIs in the release after they've been 
deprecated).

With my history in dealing with Hadoop and friends, I'm of the opinion 
that we should not expose these dependencies to our downstream users at 
all. Shade+relocate, downstream projects include whatever version of 
Guava they'd like. If we want to make sure we keep the size of the jars 
down, we can still provide non-shaded artifacts with the express 
understanding that users will need to provide a specific version.

This recommendation also requires that we don't expose Guava classes via 
any Calcite APIs (as we can't be 100% certain such classes won't disappear).

Julian Hyde wrote:
> Michael Mior would like to upgrade Guava from 14 to 15 so that we can use Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier versions. So, how to please everyone?
>
> I have logged https://issues.apache.org/jira/browse/CALCITE-1111<https://issues.apache.org/jira/browse/CALCITE-1111>  and am suggesting solving the problem by making Guava what Maven calls a “provided” dependency. Do you think that might work?
>
> If this works, we might try the same approach with Jackson. (Jackson is currently shaded in Avatica but not relocated.)
>
> Julian
>
>

Re: Upgrading Guava

[James Taylor <ja...@apache.org>]

We've had a rough time with Guava versions among all our dependencies:
HBase (and in turn HDFS) which requires guava 12, Tephra which required
guava 13 (but now works with 12 too, thanks to some clever copy/pasting of
code), and our own usage (not to mention requirements internally at
SFDC). So the less specific you can be, the better. If I had to do it over
again, I'd not use Guava at all. Nick said it best:
https://twitter.com/xefyr/status/694571680950284288



On Mon, Feb 29, 2016 at 2:20 PM, Julian Hyde <jh...@apache.org> wrote:

> Michael Mior would like to upgrade Guava from 14 to 15 so that we can use
> Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier
> versions. So, how to please everyone?
>
> I have logged https://issues.apache.org/jira/browse/CALCITE-1111 <
> https://issues.apache.org/jira/browse/CALCITE-1111> and am suggesting
> solving the problem by making Guava what Maven calls a “provided”
> dependency. Do you think that might work?
>
> If this works, we might try the same approach with Jackson. (Jackson is
> currently shaded in Avatica but not relocated.)
>
> Julian
>
>

Re: Upgrading Guava

[Jacques Nadeau <ja...@apache.org>]

We just upgraded Drill to Guava 18. We did this by adding this class:

https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/test/java/org/apache/drill/hbase/GuavaPatcher.java

I think it depends on the particular projects but the items patched in that
class solved our issues with using HBase with Guava 18. (Note that these
were only needed for the HBase server, no changes were needed for the HBase
client. Also note that we didn't run a full test suite with Guava on this
but it allowed the launch and query of HBase.)

So the question is really what interfaces each tool is using.



On Mon, Feb 29, 2016 at 6:38 PM, hongbin ma <ma...@apache.org> wrote:

> also interested in this topic, as we have the similar issue in Kylin :)
>
> On Tue, Mar 1, 2016 at 6:20 AM, Julian Hyde <jh...@apache.org> wrote:
>
> > Michael Mior would like to upgrade Guava from 14 to 15 so that we can use
> > Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier
> > versions. So, how to please everyone?
> >
> > I have logged https://issues.apache.org/jira/browse/CALCITE-1111 <
> > https://issues.apache.org/jira/browse/CALCITE-1111> and am suggesting
> > solving the problem by making Guava what Maven calls a “provided”
> > dependency. Do you think that might work?
> >
> > If this works, we might try the same approach with Jackson. (Jackson is
> > currently shaded in Avatica but not relocated.)
> >
> > Julian
> >
> >
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: Upgrading Guava

[hongbin ma <ma...@apache.org>]

also interested in this topic, as we have the similar issue in Kylin :)

On Tue, Mar 1, 2016 at 6:20 AM, Julian Hyde <jh...@apache.org> wrote:

> Michael Mior would like to upgrade Guava from 14 to 15 so that we can use
> Cassandra 3.0. But I know some projects (e.g. Hive) depend on earlier
> versions. So, how to please everyone?
>
> I have logged https://issues.apache.org/jira/browse/CALCITE-1111 <
> https://issues.apache.org/jira/browse/CALCITE-1111> and am suggesting
> solving the problem by making Guava what Maven calls a “provided”
> dependency. Do you think that might work?
>
> If this works, we might try the same approach with Jackson. (Jackson is
> currently shaded in Avatica but not relocated.)
>
> Julian
>
>


-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone