You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Gian Merlino <gi...@apache.org> on 2023/01/04 06:15:00 UTC
Re: Calcite jars with shaded guava
Thank you! I've built a copy of main and have started working on integrating it into Druid. I'll let the group know if anything interesting comes up. There is also a good chunk of work to do here, as it's been a while since we updated our version of Calcite.
On 2022/12/11 18:43:08 Julian Hyde wrote:
> The bug is now fixed. (I started from your PR, but I had to do quite a lot of extra work. Despite what I said earlier, there were several places where Calcite was using APIs added between Guava 16 and 19.)
>
> > On Dec 9, 2022, at 1:30 PM, Gian Merlino <gi...@apache.org> wrote:
> >
> > Hmm. In that case I'm not sure why we came to the belief that newer Calcites could not be used with Guava 16.0.1. Maybe it was from reading the Calcite release notes rather than actually testing it. It would be great if this just works.
> >
> > I raised this PR: https://github.com/apache/calcite/pull/2999 and this JIRA: https://issues.apache.org/jira/browse/CALCITE-5428.
> >
> >> On 2022/12/09 20:54:46 Julian Hyde wrote:
> >> I ran with '-Dguava.version=16.0.1' and it seems to just work.
> >>
> >> If you want this to be the new minimum, please create a Jira case and
> >> PR. Changes will be needed to main.yml (GitHub CI), comments in
> >> gradle.properties, and history.md (release notes for the upcoming
> >> release).
> >>
> >> Julian
> >>
> >>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino <gi...@apache.org> wrote:
> >>>
> >>> If Calcite ends up dropping the minimum to 16, that would certainly be helpful, and appreciated, since it would mean Druid can upgrade Calcite without needing to also tackle the Hadoop 2 problem. Please let us know if it seems like that would be in the cards!
> >>>
> >>> In case Calcite sticks on Guava 19+, I asked on the Druid issue (https://github.com/apache/druid/issues/13532) to see if anyone's tried Hadoop 2 with Guava 19. I'm not sure if we've tried that specific combination; if not, maybe that would work for us. If that doesn't work out, I think our best remaining idea would be to build our own version of Calcite that uses a relocated Guava.
> >>>
> >>> On 2022/12/08 20:49:17 Julian Hyde wrote:
> >>>> If it helps, it wouldn’t be impossible for Calcite to move its minimum Guava version down to 16. (If we rely on APIs introduced after 16 we can create a facade for those APIs.)
> >>>>
> >>>> I am frustrated that the Hadoop folks get locked into a particular version of Guava (and JDK… don’t get me started on that one) and in Calcite I would like to do the opposite.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>> On Dec 8, 2022, at 9:24 AM, Gian Merlino <gi...@apache.org> wrote:
> >>>>>
> >>>>> It pains me to admit that the reason Druid still uses Guava 16 is that we still support Hadoop 2, which relies on APIs that have been removed in newer versions of Guava. From time to time it is proposed that the project drops support for Hadoop 2 in order to be free to move up Guava versions, and also achieve various other benefits. There are generally concerns from people that are stuck with Hadoop 2 for various reasons. The most recent thread is here: https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope at some point we decide to bite the bullet and do it, but the community has not yet come to that decision.
> >>>>>
> >>>>> In the meantime, we're stuck on an older Calcite version, and really want to update it, so we're looking into options. There's a GitHub issue here with our thoughts: https://github.com/apache/druid/issues/13532
> >>>>>
> >>>>> On 2022/03/30 17:55:44 Julian Hyde wrote:
> >>>>>> Wow, I had no idea that anyone was on a version of Guava that old.
> >>>>>>
> >>>>>> I checked our history of Guava versions:
> >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1590 <https://issues.apache.org/jira/browse/CALCITE-1590> (Calcite 1.12) we changed the default from 19 to 20, but still supported Guava 14.0.1 and higher.
> >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1715 <https://issues.apache.org/jira/browse/CALCITE-1715> (also Calcite 1.12) we changed the default version from 20 to 19
> >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-2191 <https://issues.apache.org/jira/browse/CALCITE-2191> (Calcite 1.16) we moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] about upgrading Druid’s version of Guava
> >>>>>>
> >>>>>> Foolish of me to assume that 4 years later Druid would have done something.
> >>>>>>
> >>>>>> Julian
> >>>>>>
> >>>>>> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> >>>>>>
> >>>>>>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal <ab...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> >>>>>>> version being used in druid is 16. So simple exclusion is unlikely to work.
> >>>>>>> I will still give it a try. If it doesn't, we will shade the jars on our
> >>>>>>> end.
> >>>>>>>
> >>>>>>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> >>>>>>> alessandro.solimando@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hello Abhishek,
> >>>>>>>> maybe I am re-stating what Stamatis suggested without realising, but since
> >>>>>>>> Calcite works well with a broad range of guava versions, can't you simply
> >>>>>>>> exclude guava when you include it in druid?
> >>>>>>>>
> >>>>>>>> At that point if the guava version of druid is in the list of those
> >>>>>>>> supported by Calcite (>= 19 IIRC) all should work fine.
> >>>>>>>>
> >>>>>>>> I feel like Stamatis concerning shading.
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Alessandro
> >>>>>>>>
> >>>>>>>> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis <za...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Abhishek,
> >>>>>>>>>
> >>>>>>>>> Calcite supports multiple versions of Guava so in principle it should
> >>>>>>>> work
> >>>>>>>>> without problems with older versions.
> >>>>>>>>> Instead of shading you could possibly select explicitly the version
> >>>>>>>>> that you want to use via maven/gradle.
> >>>>>>>>>
> >>>>>>>>> I share your pain but shading is quite complex to get right so I am not
> >>>>>>>> in
> >>>>>>>>> favor of putting this burden on Calcite.
> >>>>>>>>>
> >>>>>>>>> I've seen it used in many projects and it rarely works as expected. Most
> >>>>>>>> of
> >>>>>>>>> the time it creates additional problems that need to be resolved.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Stamatis
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
>
Re: Calcite jars with shaded guava
Posted by Gian Merlino <gi...@apache.org>.
Still working on this and gradually making progress. So far, I ran into these three things along the way:
https://issues.apache.org/jira/browse/CALCITE-5477
https://issues.apache.org/jira/browse/CALCITE-5478
https://issues.apache.org/jira/browse/CALCITE-5479
I've applied the linked PRs locally and continue to make progress integrating this new version into Druid.
On 2023/01/04 06:15:00 Gian Merlino wrote:
> Thank you! I've built a copy of main and have started working on integrating it into Druid. I'll let the group know if anything interesting comes up. There is also a good chunk of work to do here, as it's been a while since we updated our version of Calcite.
>
> On 2022/12/11 18:43:08 Julian Hyde wrote:
> > The bug is now fixed. (I started from your PR, but I had to do quite a lot of extra work. Despite what I said earlier, there were several places where Calcite was using APIs added between Guava 16 and 19.)
> >
> > > On Dec 9, 2022, at 1:30 PM, Gian Merlino <gi...@apache.org> wrote:
> > >
> > > Hmm. In that case I'm not sure why we came to the belief that newer Calcites could not be used with Guava 16.0.1. Maybe it was from reading the Calcite release notes rather than actually testing it. It would be great if this just works.
> > >
> > > I raised this PR: https://github.com/apache/calcite/pull/2999 and this JIRA: https://issues.apache.org/jira/browse/CALCITE-5428.
> > >
> > >> On 2022/12/09 20:54:46 Julian Hyde wrote:
> > >> I ran with '-Dguava.version=16.0.1' and it seems to just work.
> > >>
> > >> If you want this to be the new minimum, please create a Jira case and
> > >> PR. Changes will be needed to main.yml (GitHub CI), comments in
> > >> gradle.properties, and history.md (release notes for the upcoming
> > >> release).
> > >>
> > >> Julian
> > >>
> > >>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino <gi...@apache.org> wrote:
> > >>>
> > >>> If Calcite ends up dropping the minimum to 16, that would certainly be helpful, and appreciated, since it would mean Druid can upgrade Calcite without needing to also tackle the Hadoop 2 problem. Please let us know if it seems like that would be in the cards!
> > >>>
> > >>> In case Calcite sticks on Guava 19+, I asked on the Druid issue (https://github.com/apache/druid/issues/13532) to see if anyone's tried Hadoop 2 with Guava 19. I'm not sure if we've tried that specific combination; if not, maybe that would work for us. If that doesn't work out, I think our best remaining idea would be to build our own version of Calcite that uses a relocated Guava.
> > >>>
> > >>> On 2022/12/08 20:49:17 Julian Hyde wrote:
> > >>>> If it helps, it wouldn’t be impossible for Calcite to move its minimum Guava version down to 16. (If we rely on APIs introduced after 16 we can create a facade for those APIs.)
> > >>>>
> > >>>> I am frustrated that the Hadoop folks get locked into a particular version of Guava (and JDK… don’t get me started on that one) and in Calcite I would like to do the opposite.
> > >>>>
> > >>>> Julian
> > >>>>
> > >>>>
> > >>>>> On Dec 8, 2022, at 9:24 AM, Gian Merlino <gi...@apache.org> wrote:
> > >>>>>
> > >>>>> It pains me to admit that the reason Druid still uses Guava 16 is that we still support Hadoop 2, which relies on APIs that have been removed in newer versions of Guava. From time to time it is proposed that the project drops support for Hadoop 2 in order to be free to move up Guava versions, and also achieve various other benefits. There are generally concerns from people that are stuck with Hadoop 2 for various reasons. The most recent thread is here: https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I hope at some point we decide to bite the bullet and do it, but the community has not yet come to that decision.
> > >>>>>
> > >>>>> In the meantime, we're stuck on an older Calcite version, and really want to update it, so we're looking into options. There's a GitHub issue here with our thoughts: https://github.com/apache/druid/issues/13532
> > >>>>>
> > >>>>> On 2022/03/30 17:55:44 Julian Hyde wrote:
> > >>>>>> Wow, I had no idea that anyone was on a version of Guava that old.
> > >>>>>>
> > >>>>>> I checked our history of Guava versions:
> > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1590 <https://issues.apache.org/jira/browse/CALCITE-1590> (Calcite 1.12) we changed the default from 19 to 20, but still supported Guava 14.0.1 and higher.
> > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1715 <https://issues.apache.org/jira/browse/CALCITE-1715> (also Calcite 1.12) we changed the default version from 20 to 19
> > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-2191 <https://issues.apache.org/jira/browse/CALCITE-2191> (Calcite 1.16) we moved the minimum from 14.0.1 to 19 and cited a Druid email thread [1] about upgrading Druid’s version of Guava
> > >>>>>>
> > >>>>>> Foolish of me to assume that 4 years later Druid would have done something.
> > >>>>>>
> > >>>>>> Julian
> > >>>>>>
> > >>>>>> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ
> > >>>>>>
> > >>>>>>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal <ab...@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>> Thank you, Alessandro and Stamatis, for the info. Currently, the guava
> > >>>>>>> version being used in druid is 16. So simple exclusion is unlikely to work.
> > >>>>>>> I will still give it a try. If it doesn't, we will shade the jars on our
> > >>>>>>> end.
> > >>>>>>>
> > >>>>>>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando <
> > >>>>>>> alessandro.solimando@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hello Abhishek,
> > >>>>>>>> maybe I am re-stating what Stamatis suggested without realising, but since
> > >>>>>>>> Calcite works well with a broad range of guava versions, can't you simply
> > >>>>>>>> exclude guava when you include it in druid?
> > >>>>>>>>
> > >>>>>>>> At that point if the guava version of druid is in the list of those
> > >>>>>>>> supported by Calcite (>= 19 IIRC) all should work fine.
> > >>>>>>>>
> > >>>>>>>> I feel like Stamatis concerning shading.
> > >>>>>>>>
> > >>>>>>>> Best regards,
> > >>>>>>>> Alessandro
> > >>>>>>>>
> > >>>>>>>> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis <za...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Abhishek,
> > >>>>>>>>>
> > >>>>>>>>> Calcite supports multiple versions of Guava so in principle it should
> > >>>>>>>> work
> > >>>>>>>>> without problems with older versions.
> > >>>>>>>>> Instead of shading you could possibly select explicitly the version
> > >>>>>>>>> that you want to use via maven/gradle.
> > >>>>>>>>>
> > >>>>>>>>> I share your pain but shading is quite complex to get right so I am not
> > >>>>>>>> in
> > >>>>>>>>> favor of putting this burden on Calcite.
> > >>>>>>>>>
> > >>>>>>>>> I've seen it used in many projects and it rarely works as expected. Most
> > >>>>>>>> of
> > >>>>>>>>> the time it creates additional problems that need to be resolved.
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Stamatis
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> >
>