You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2018/10/09 08:04:12 UTC
[DISCUSS] Gradle for the build ?
Hi guys,
I know that's a hot topic, but I have to bring this discussion on the table.
Some months ago, we discussed about migrating our build from Maven to
Gradle. One of the key expected improvement was the time to build.
We proposed to do a PoC to evaluate the impacts and improvements, but
this PoC was actually directly a migrate on master.
Now, I would like to bring facts here:
1. Build time
On my machine, the build time is roughly 1h15. It's pretty long, and
regarding what the build is doing, I don't see huge improvement provided
by Gradle.
2. Build reliability
Even worse, most of the time, we need to use --no-parallel and
--no-daemon to have a reliable build (it's basically recommended for
release). It has an impact on build time, and we loose part of Gradle
benefits.
3. Release and repositories
Even if couple of releases has been performed with Gradle, it's not
obvious to see improvements around artifacts handling. I got my
repository polluted twice (that's part of the trick Gradle is doing to
speed up the build dealing around the repository).
4. IDE integration
We already had some comments on the mailing lists about the IDE
integration. Clearly, the situation is not good on that front too. The
integration on IDE (especially IntelliJ) is not good enough right now.
We are working hard to grow up the community, and from a contributor
perspective, our build system is not good today IMHO.
As a contributor, I resumed my work on some PRs, and I'm spending so
much time of the build, largely more than working on the PRs code itself.
So, obviously, the situation is not perfect, at least from a contributor
perspective.
The purpose of this thread is not again to have a bunch of replied
ending nowhere. I would like to be more "pushy" and let's try to be
concrete. So basically, we only have two options:
1. Improve the build, working hard on Gradle front. Not sure if it makes
such sense from a contributor perspective, as Maven is really well known
from most of contributors (and easier to start with IMHO).
2. Back on Maven. That's clearly my preferred approach. IDE integration
is better, Maven is well known from the contributors as already said.
The effort is not so huge. We tried to use Gradle, we don't have the
expected results now, that's not a problem, it's part of a project lifetime.
Thoughts ?
Regards
JB
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [DISCUSS] Gradle for the build ?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
+1000000000000000000 on 2, as mentionned multiple times on the list before
master became gradle driven, i saw gradle driven projects only working when
driven and enforced by community+a few tech lead. This is not how Apache
works so keeping a broken project setup sounds like the worse possible
state for a community driven project. Let's fix it!
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mar. 9 oct. 2018 à 10:04, Jean-Baptiste Onofré <jb...@nanthrax.net> a
écrit :
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion on the
> table.
>
> Some months ago, we discussed about migrating our build from Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement provided
> by Gradle.
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of Gradle
> benefits.
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is doing to
> speed up the build dealing around the repository).
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front too. The
> integration on IDE (especially IntelliJ) is not good enough right now.
>
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs code itself.
>
> So, obviously, the situation is not perfect, at least from a contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
> such sense from a contributor perspective, as Maven is really well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
> is better, Maven is well known from the contributors as already said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a project
> lifetime.
>
> Thoughts ?
>
> Regards
> JB
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [DISCUSS] Gradle for the build ?
Posted by Robert Burke <ro...@frantil.com>.
Hi all!
Coming from the Go SDK side, I'm in favour of splitting the repo after the
portability APIs have stablized. The reason being that it would be all to
easy for changes not be propagated to the SDKs, which would cause painful
drift.
As for the Go Gradle experience, it agree it leaves much to be desired,
since it's not cleaning up after itself in it's attempt to ensure
reproduceable go builds. BEAM-5379 has been filled as a possible solution
to that, with the suggestion to move to the Go Modules supported by Go 1.11.
Resolving this is on the short list of contributions I want to make this
quarter.
From a pure Go SDK contributor standpoint, I generally use the standard go
tools (mostly the go tool) and its existing IDE integrations rather than
attempt to make Gradle work for me, outside of building the core
components. I can't speak for everyone of course but it's my view on it.
On Thu, Oct 11, 2018, 10:33 AM Udi Meiri <eh...@google.com> wrote:
> I agree with the points made that our Gradle configuration is too
> complicated (learning Groovy and BeamModulePlugin come to mind).
>
> But I mainly write for Python SDK, and I've done experiments in the past
> with Bazel builds, which was a much better fit for Python as an incremental
> and parallel build and test system.
>
> On Thu, Oct 11, 2018 at 4:56 AM Alexey Romanenko <ar...@gmail.com>
> wrote:
>
>> I agree with all previous points related to Gradle-related problems.
>> Among all others, I’d highlight IDE integration issue since it can have
>> real negative impact on attracting new contributors and, finally, community
>> grow. The more obstacles there are to start code contribution, the more
>> chances that it will distract newcomers.
>>
>> In this way I see two points to do:
>> 1) Continue working on IDE integration improvement. I hope there are no
>> principal barriers for that and it’s only a question of applied forces,
>> otherwise, it can be a very serious issue.
>> 2) Improve our “Get started” documentation for contributors to make the
>> process of first-time contribution as smooth and simple as possible.
>>
>>
>> On 11 Oct 2018, at 00:22, Tim Robertson <ti...@gmail.com>
>> wrote:
>>
>> Thank you JB for starting this discussion.
>>
>> Others comment on many of these points far better than I can, but my
>> experience is similar to JB.
>>
>> 1. IDEA integration (and laptop slowing like crazy) being the biggest
>> contributor to my feeling of being unproductive
>> 2. Not knowing the correct way to modify the build scripts which I put
>> down to my own limitations
>>
>> It seems we also need to help build Gradle expertise in our community, so
>>> that those that are motivated are empowered to contribute.
>>
>>
>> Nicely phrased. +1
>>
>>
>>
>> On Wed, Oct 10, 2018 at 7:15 PM Scott Wegner <sc...@apache.org> wrote:
>>
>>> > Perhaps we should go through and prioritize (and add missing items to)
>>> BEAM-4045
>>>
>>> +1. It's hard to know where to start when there's such a laundry list of
>>> tasks. If you're having build issues, will you make sure it is represented
>>> in BEAM-4045, and "Vote" for the issues that you believe are the highest
>>> priority?
>>>
>>> I agree that the Gradle build is far from perfect (my top gripes are IDE
>>> integration and parallel/incremental build support). I believe that we're
>>> capable of making our build great, and continuing our investment in Gradle
>>> would be a shorter path than changing course again. Remember that our Maven
>>> build also had it's share of issues, which is why we as a community voted
>>> to replace it [1][2].
>>>
>>> It seems we also need to help build Gradle expertise in our community,
>>> so that those that are motivated are empowered to contribute. Does anybody
>>> have a good "Getting Started with Gradle" guide they recommend? Perhaps we
>>> could also link to it from the website/wiki.
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
>>> [2]
>>> https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E
>>>
>>> On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> Some rough stats (because I was curious): The gradle files have been
>>>> edited by ~79 unique contributors over 696 distinct commits, whereas the
>>>> maven ones were edited (over a longer time period) by ~130 unique
>>>> contributors over 1389 commits [1]. This doesn't capture how much effort
>>>> was put into these edits, but neither is restricted to a small set of
>>>> experts.
>>>>
>>>> Regarding "friendly for other languages" I don't think either is
>>>> necessarily easy to learn, but my impression is that the maven learning
>>>> curve shallower for those already firmly embedded in the Java ecosystem
>>>> (perhaps due to leveraging existing familiarity, and perhaps some due to
>>>> the implicit java-centric conventions that maven assumed about your
>>>> project), whereas with gradle at least I could keep pulling on the string
>>>> to unwind things to the bottom. The "I just want to build/test X without
>>>> editing/viewing the build files" seemed more natural with Gradle (e.g. I
>>>> can easily list all tasks).
>>>>
>>>> That being said, I don't think everyone needs to understand the full
>>>> build system. It's important that there be a critical mass that do (we have
>>>> that for both, and if we can simplify to improve this that'd be great),
>>>> it's easy enough to do basic changes (e.g. add a dependency, again I don't
>>>> think the barrier is sufficiently different for either), and works well out
>>>> of the box for someone who just wants to look up a command on the website
>>>> and edit code (the CLI is an improvement with Gradle, but it's clear that
>>>> (java) IDE support is a significant regression).
>>>>
>>>> Personally, I don't know much about IDE configuration (admittedly the
>>>> larger issue), but one action item I can take on is trying to eliminate the
>>>> need to do a "git clean" after building certain targets (assuming I can
>>>> reproduce this).
>>>>
>>>> Perhaps we should go through and prioritize (and add missing items to)
>>>> BEAM-4045
>>>> https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC
>>>> ? There's always a long tail with this kind of thing, and looking at the
>>>> whole list can be daunting, but putting it in the correct order and
>>>> knocking off the top N items could possibly go a long way.
>>>>
>>>> - Robert
>>>>
>>>> [1] The commands I ran were (with and without the uniq)
>>>>
>>>> $ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o
>>>> '[^< ]*@' | sort | uniq | wc
>>>> $ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^<
>>>> ]*@' | sort | uniq | wc
>>>>
>>>> On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <ec...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>> I must admit that I agree on the status especially regarding 2 points:
>>>>> 1. new contributors obstacles: gradle learning curve might be too long
>>>>> for spare-time contributors, also complex scripted build takes time to
>>>>> understand comparing to self-descriptive one.
>>>>> 2. IDE integration kind of slows down development.
>>>>>
>>>>> Now, regarding how we improve the situation, I think we need to
>>>>> discuss and identify tasks and tackle them all together even if they are
>>>>> not sexy tasks as Ismaël mentioned.
>>>>>
>>>>> Etienne
>>>>>
>>>>> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>>>>>
>>>>> Hi guys,
>>>>>
>>>>>
>>>>> I know that's a hot topic, but I have to bring this discussion on the table.
>>>>>
>>>>>
>>>>> Some months ago, we discussed about migrating our build from Maven to
>>>>>
>>>>> Gradle. One of the key expected improvement was the time to build.
>>>>>
>>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>>>
>>>>> this PoC was actually directly a migrate on master.
>>>>>
>>>>>
>>>>> Now, I would like to bring facts here:
>>>>>
>>>>>
>>>>> 1. Build time
>>>>>
>>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>>>
>>>>> regarding what the build is doing, I don't see huge improvement provided
>>>>>
>>>>> by Gradle.
>>>>>
>>>>> 2. Build reliability
>>>>>
>>>>> Even worse, most of the time, we need to use --no-parallel and
>>>>>
>>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>>>
>>>>> release). It has an impact on build time, and we loose part of Gradle
>>>>>
>>>>> benefits.
>>>>>
>>>>> 3. Release and repositories
>>>>>
>>>>> Even if couple of releases has been performed with Gradle, it's not
>>>>>
>>>>> obvious to see improvements around artifacts handling. I got my
>>>>>
>>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>>>
>>>>> speed up the build dealing around the repository).
>>>>>
>>>>> 4. IDE integration
>>>>>
>>>>> We already had some comments on the mailing lists about the IDE
>>>>>
>>>>> integration. Clearly, the situation is not good on that front too. The
>>>>>
>>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>>>
>>>>>
>>>>> We are working hard to grow up the community, and from a contributor
>>>>>
>>>>> perspective, our build system is not good today IMHO.
>>>>>
>>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>>>
>>>>> much time of the build, largely more than working on the PRs code itself.
>>>>>
>>>>>
>>>>> So, obviously, the situation is not perfect, at least from a contributor
>>>>>
>>>>> perspective.
>>>>>
>>>>>
>>>>> The purpose of this thread is not again to have a bunch of replied
>>>>>
>>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>>>
>>>>> concrete. So basically, we only have two options:
>>>>>
>>>>>
>>>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>>>>
>>>>> such sense from a contributor perspective, as Maven is really well known
>>>>>
>>>>> from most of contributors (and easier to start with IMHO).
>>>>>
>>>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>>>
>>>>> is better, Maven is well known from the contributors as already said.
>>>>>
>>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>>>
>>>>> expected results now, that's not a problem, it's part of a project lifetime.
>>>>>
>>>>>
>>>>> Thoughts ?
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>> JB
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>
>>
Re: [DISCUSS] Gradle for the build ?
Posted by Udi Meiri <eh...@google.com>.
I agree with the points made that our Gradle configuration is too
complicated (learning Groovy and BeamModulePlugin come to mind).
But I mainly write for Python SDK, and I've done experiments in the past
with Bazel builds, which was a much better fit for Python as an incremental
and parallel build and test system.
On Thu, Oct 11, 2018 at 4:56 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> I agree with all previous points related to Gradle-related problems. Among
> all others, I’d highlight IDE integration issue since it can have real
> negative impact on attracting new contributors and, finally, community
> grow. The more obstacles there are to start code contribution, the more
> chances that it will distract newcomers.
>
> In this way I see two points to do:
> 1) Continue working on IDE integration improvement. I hope there are no
> principal barriers for that and it’s only a question of applied forces,
> otherwise, it can be a very serious issue.
> 2) Improve our “Get started” documentation for contributors to make the
> process of first-time contribution as smooth and simple as possible.
>
>
> On 11 Oct 2018, at 00:22, Tim Robertson <ti...@gmail.com> wrote:
>
> Thank you JB for starting this discussion.
>
> Others comment on many of these points far better than I can, but my
> experience is similar to JB.
>
> 1. IDEA integration (and laptop slowing like crazy) being the biggest
> contributor to my feeling of being unproductive
> 2. Not knowing the correct way to modify the build scripts which I put
> down to my own limitations
>
> It seems we also need to help build Gradle expertise in our community, so
>> that those that are motivated are empowered to contribute.
>
>
> Nicely phrased. +1
>
>
>
> On Wed, Oct 10, 2018 at 7:15 PM Scott Wegner <sc...@apache.org> wrote:
>
>> > Perhaps we should go through and prioritize (and add missing items to)
>> BEAM-4045
>>
>> +1. It's hard to know where to start when there's such a laundry list of
>> tasks. If you're having build issues, will you make sure it is represented
>> in BEAM-4045, and "Vote" for the issues that you believe are the highest
>> priority?
>>
>> I agree that the Gradle build is far from perfect (my top gripes are IDE
>> integration and parallel/incremental build support). I believe that we're
>> capable of making our build great, and continuing our investment in Gradle
>> would be a shorter path than changing course again. Remember that our Maven
>> build also had it's share of issues, which is why we as a community voted
>> to replace it [1][2].
>>
>> It seems we also need to help build Gradle expertise in our community, so
>> that those that are motivated are empowered to contribute. Does anybody
>> have a good "Getting Started with Gradle" guide they recommend? Perhaps we
>> could also link to it from the website/wiki.
>>
>> [1]
>> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
>> [2]
>> https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> Some rough stats (because I was curious): The gradle files have been
>>> edited by ~79 unique contributors over 696 distinct commits, whereas the
>>> maven ones were edited (over a longer time period) by ~130 unique
>>> contributors over 1389 commits [1]. This doesn't capture how much effort
>>> was put into these edits, but neither is restricted to a small set of
>>> experts.
>>>
>>> Regarding "friendly for other languages" I don't think either is
>>> necessarily easy to learn, but my impression is that the maven learning
>>> curve shallower for those already firmly embedded in the Java ecosystem
>>> (perhaps due to leveraging existing familiarity, and perhaps some due to
>>> the implicit java-centric conventions that maven assumed about your
>>> project), whereas with gradle at least I could keep pulling on the string
>>> to unwind things to the bottom. The "I just want to build/test X without
>>> editing/viewing the build files" seemed more natural with Gradle (e.g. I
>>> can easily list all tasks).
>>>
>>> That being said, I don't think everyone needs to understand the full
>>> build system. It's important that there be a critical mass that do (we have
>>> that for both, and if we can simplify to improve this that'd be great),
>>> it's easy enough to do basic changes (e.g. add a dependency, again I don't
>>> think the barrier is sufficiently different for either), and works well out
>>> of the box for someone who just wants to look up a command on the website
>>> and edit code (the CLI is an improvement with Gradle, but it's clear that
>>> (java) IDE support is a significant regression).
>>>
>>> Personally, I don't know much about IDE configuration (admittedly the
>>> larger issue), but one action item I can take on is trying to eliminate the
>>> need to do a "git clean" after building certain targets (assuming I can
>>> reproduce this).
>>>
>>> Perhaps we should go through and prioritize (and add missing items to)
>>> BEAM-4045
>>> https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC
>>> ? There's always a long tail with this kind of thing, and looking at the
>>> whole list can be daunting, but putting it in the correct order and
>>> knocking off the top N items could possibly go a long way.
>>>
>>> - Robert
>>>
>>> [1] The commands I ran were (with and without the uniq)
>>>
>>> $ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o
>>> '[^< ]*@' | sort | uniq | wc
>>> $ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^<
>>> ]*@' | sort | uniq | wc
>>>
>>> On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <ec...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>> I must admit that I agree on the status especially regarding 2 points:
>>>> 1. new contributors obstacles: gradle learning curve might be too long
>>>> for spare-time contributors, also complex scripted build takes time to
>>>> understand comparing to self-descriptive one.
>>>> 2. IDE integration kind of slows down development.
>>>>
>>>> Now, regarding how we improve the situation, I think we need to discuss
>>>> and identify tasks and tackle them all together even if they are not sexy
>>>> tasks as Ismaël mentioned.
>>>>
>>>> Etienne
>>>>
>>>> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>>>>
>>>> Hi guys,
>>>>
>>>>
>>>> I know that's a hot topic, but I have to bring this discussion on the table.
>>>>
>>>>
>>>> Some months ago, we discussed about migrating our build from Maven to
>>>>
>>>> Gradle. One of the key expected improvement was the time to build.
>>>>
>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>>
>>>> this PoC was actually directly a migrate on master.
>>>>
>>>>
>>>> Now, I would like to bring facts here:
>>>>
>>>>
>>>> 1. Build time
>>>>
>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>>
>>>> regarding what the build is doing, I don't see huge improvement provided
>>>>
>>>> by Gradle.
>>>>
>>>> 2. Build reliability
>>>>
>>>> Even worse, most of the time, we need to use --no-parallel and
>>>>
>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>>
>>>> release). It has an impact on build time, and we loose part of Gradle
>>>>
>>>> benefits.
>>>>
>>>> 3. Release and repositories
>>>>
>>>> Even if couple of releases has been performed with Gradle, it's not
>>>>
>>>> obvious to see improvements around artifacts handling. I got my
>>>>
>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>>
>>>> speed up the build dealing around the repository).
>>>>
>>>> 4. IDE integration
>>>>
>>>> We already had some comments on the mailing lists about the IDE
>>>>
>>>> integration. Clearly, the situation is not good on that front too. The
>>>>
>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>>
>>>>
>>>> We are working hard to grow up the community, and from a contributor
>>>>
>>>> perspective, our build system is not good today IMHO.
>>>>
>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>>
>>>> much time of the build, largely more than working on the PRs code itself.
>>>>
>>>>
>>>> So, obviously, the situation is not perfect, at least from a contributor
>>>>
>>>> perspective.
>>>>
>>>>
>>>> The purpose of this thread is not again to have a bunch of replied
>>>>
>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>>
>>>> concrete. So basically, we only have two options:
>>>>
>>>>
>>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>>>
>>>> such sense from a contributor perspective, as Maven is really well known
>>>>
>>>> from most of contributors (and easier to start with IMHO).
>>>>
>>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>>
>>>> is better, Maven is well known from the contributors as already said.
>>>>
>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>>
>>>> expected results now, that's not a problem, it's part of a project lifetime.
>>>>
>>>>
>>>> Thoughts ?
>>>>
>>>>
>>>> Regards
>>>>
>>>> JB
>>>>
>>>>
>>>>
>>>>
>>
>> --
>>
>>
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>
>
Re: [DISCUSS] Gradle for the build ?
Posted by Alexey Romanenko <ar...@gmail.com>.
I agree with all previous points related to Gradle-related problems. Among all others, I’d highlight IDE integration issue since it can have real negative impact on attracting new contributors and, finally, community grow. The more obstacles there are to start code contribution, the more chances that it will distract newcomers.
In this way I see two points to do:
1) Continue working on IDE integration improvement. I hope there are no principal barriers for that and it’s only a question of applied forces, otherwise, it can be a very serious issue.
2) Improve our “Get started” documentation for contributors to make the process of first-time contribution as smooth and simple as possible.
> On 11 Oct 2018, at 00:22, Tim Robertson <ti...@gmail.com> wrote:
>
> Thank you JB for starting this discussion.
>
> Others comment on many of these points far better than I can, but my experience is similar to JB.
>
> 1. IDEA integration (and laptop slowing like crazy) being the biggest contributor to my feeling of being unproductive
> 2. Not knowing the correct way to modify the build scripts which I put down to my own limitations
>
> It seems we also need to help build Gradle expertise in our community, so that those that are motivated are empowered to contribute.
>
> Nicely phrased. +1
>
>
>
> On Wed, Oct 10, 2018 at 7:15 PM Scott Wegner <scott@apache.org <ma...@apache.org>> wrote:
> > Perhaps we should go through and prioritize (and add missing items to) BEAM-4045
>
> +1. It's hard to know where to start when there's such a laundry list of tasks. If you're having build issues, will you make sure it is represented in BEAM-4045, and "Vote" for the issues that you believe are the highest priority?
>
> I agree that the Gradle build is far from perfect (my top gripes are IDE integration and parallel/incremental build support). I believe that we're capable of making our build great, and continuing our investment in Gradle would be a shorter path than changing course again. Remember that our Maven build also had it's share of issues, which is why we as a community voted to replace it [1][2].
>
> It seems we also need to help build Gradle expertise in our community, so that those that are motivated are empowered to contribute. Does anybody have a good "Getting Started with Gradle" guide they recommend? Perhaps we could also link to it from the website/wiki.
>
> [1] https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E <https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E>
> [2] https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E <https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E>
>
> On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <robertwb@google.com <ma...@google.com>> wrote:
> Some rough stats (because I was curious): The gradle files have been edited by ~79 unique contributors over 696 distinct commits, whereas the maven ones were edited (over a longer time period) by ~130 unique contributors over 1389 commits [1]. This doesn't capture how much effort was put into these edits, but neither is restricted to a small set of experts.
>
> Regarding "friendly for other languages" I don't think either is necessarily easy to learn, but my impression is that the maven learning curve shallower for those already firmly embedded in the Java ecosystem (perhaps due to leveraging existing familiarity, and perhaps some due to the implicit java-centric conventions that maven assumed about your project), whereas with gradle at least I could keep pulling on the string to unwind things to the bottom. The "I just want to build/test X without editing/viewing the build files" seemed more natural with Gradle (e.g. I can easily list all tasks).
>
> That being said, I don't think everyone needs to understand the full build system. It's important that there be a critical mass that do (we have that for both, and if we can simplify to improve this that'd be great), it's easy enough to do basic changes (e.g. add a dependency, again I don't think the barrier is sufficiently different for either), and works well out of the box for someone who just wants to look up a command on the website and edit code (the CLI is an improvement with Gradle, but it's clear that (java) IDE support is a significant regression).
>
> Personally, I don't know much about IDE configuration (admittedly the larger issue), but one action item I can take on is trying to eliminate the need to do a "git clean" after building certain targets (assuming I can reproduce this).
>
> Perhaps we should go through and prioritize (and add missing items to) BEAM-4045 https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC <https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC> ? There's always a long tail with this kind of thing, and looking at the whole list can be daunting, but putting it in the correct order and knocking off the top N items could possibly go a long way.
>
> - Robert
>
> [1] The commands I ran were (with and without the uniq)
>
> $ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o '[^< ]*@' | sort | uniq | wc
> $ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^< ]*@' | sort | uniq | wc
>
> On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <echauchot@apache.org <ma...@apache.org>> wrote:
> Hi all,
> I must admit that I agree on the status especially regarding 2 points:
> 1. new contributors obstacles: gradle learning curve might be too long for spare-time contributors, also complex scripted build takes time to understand comparing to self-descriptive one.
> 2. IDE integration kind of slows down development.
>
> Now, regarding how we improve the situation, I think we need to discuss and identify tasks and tackle them all together even if they are not sexy tasks as Ismaël mentioned.
>
> Etienne
>
> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>> Hi guys,
>>
>> I know that's a hot topic, but I have to bring this discussion on the table.
>>
>> Some months ago, we discussed about migrating our build from Maven to
>> Gradle. One of the key expected improvement was the time to build.
>> We proposed to do a PoC to evaluate the impacts and improvements, but
>> this PoC was actually directly a migrate on master.
>>
>> Now, I would like to bring facts here:
>>
>> 1. Build time
>> On my machine, the build time is roughly 1h15. It's pretty long, and
>> regarding what the build is doing, I don't see huge improvement provided
>> by Gradle.
>> 2. Build reliability
>> Even worse, most of the time, we need to use --no-parallel and
>> --no-daemon to have a reliable build (it's basically recommended for
>> release). It has an impact on build time, and we loose part of Gradle
>> benefits.
>> 3. Release and repositories
>> Even if couple of releases has been performed with Gradle, it's not
>> obvious to see improvements around artifacts handling. I got my
>> repository polluted twice (that's part of the trick Gradle is doing to
>> speed up the build dealing around the repository).
>> 4. IDE integration
>> We already had some comments on the mailing lists about the IDE
>> integration. Clearly, the situation is not good on that front too. The
>> integration on IDE (especially IntelliJ) is not good enough right now.
>>
>> We are working hard to grow up the community, and from a contributor
>> perspective, our build system is not good today IMHO.
>> As a contributor, I resumed my work on some PRs, and I'm spending so
>> much time of the build, largely more than working on the PRs code itself.
>>
>> So, obviously, the situation is not perfect, at least from a contributor
>> perspective.
>>
>> The purpose of this thread is not again to have a bunch of replied
>> ending nowhere. I would like to be more "pushy" and let's try to be
>> concrete. So basically, we only have two options:
>>
>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>> such sense from a contributor perspective, as Maven is really well known
>> from most of contributors (and easier to start with IMHO).
>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>> is better, Maven is well known from the contributors as already said.
>> The effort is not so huge. We tried to use Gradle, we don't have the
>> expected results now, that's not a problem, it's part of a project lifetime.
>>
>> Thoughts ?
>>
>> Regards
>> JB
>>
>>
>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback <https://tinyurl.com/swegner-feedback>
Re: [DISCUSS] Gradle for the build ?
Posted by Tim Robertson <ti...@gmail.com>.
Thank you JB for starting this discussion.
Others comment on many of these points far better than I can, but my
experience is similar to JB.
1. IDEA integration (and laptop slowing like crazy) being the biggest
contributor to my feeling of being unproductive
2. Not knowing the correct way to modify the build scripts which I put down
to my own limitations
It seems we also need to help build Gradle expertise in our community, so
> that those that are motivated are empowered to contribute.
Nicely phrased. +1
On Wed, Oct 10, 2018 at 7:15 PM Scott Wegner <sc...@apache.org> wrote:
> > Perhaps we should go through and prioritize (and add missing items to)
> BEAM-4045
>
> +1. It's hard to know where to start when there's such a laundry list of
> tasks. If you're having build issues, will you make sure it is represented
> in BEAM-4045, and "Vote" for the issues that you believe are the highest
> priority?
>
> I agree that the Gradle build is far from perfect (my top gripes are IDE
> integration and parallel/incremental build support). I believe that we're
> capable of making our build great, and continuing our investment in Gradle
> would be a shorter path than changing course again. Remember that our Maven
> build also had it's share of issues, which is why we as a community voted
> to replace it [1][2].
>
> It seems we also need to help build Gradle expertise in our community, so
> that those that are motivated are empowered to contribute. Does anybody
> have a good "Getting Started with Gradle" guide they recommend? Perhaps we
> could also link to it from the website/wiki.
>
> [1]
> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E
>
> On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Some rough stats (because I was curious): The gradle files have been
>> edited by ~79 unique contributors over 696 distinct commits, whereas the
>> maven ones were edited (over a longer time period) by ~130 unique
>> contributors over 1389 commits [1]. This doesn't capture how much effort
>> was put into these edits, but neither is restricted to a small set of
>> experts.
>>
>> Regarding "friendly for other languages" I don't think either is
>> necessarily easy to learn, but my impression is that the maven learning
>> curve shallower for those already firmly embedded in the Java ecosystem
>> (perhaps due to leveraging existing familiarity, and perhaps some due to
>> the implicit java-centric conventions that maven assumed about your
>> project), whereas with gradle at least I could keep pulling on the string
>> to unwind things to the bottom. The "I just want to build/test X without
>> editing/viewing the build files" seemed more natural with Gradle (e.g. I
>> can easily list all tasks).
>>
>> That being said, I don't think everyone needs to understand the full
>> build system. It's important that there be a critical mass that do (we have
>> that for both, and if we can simplify to improve this that'd be great),
>> it's easy enough to do basic changes (e.g. add a dependency, again I don't
>> think the barrier is sufficiently different for either), and works well out
>> of the box for someone who just wants to look up a command on the website
>> and edit code (the CLI is an improvement with Gradle, but it's clear that
>> (java) IDE support is a significant regression).
>>
>> Personally, I don't know much about IDE configuration (admittedly the
>> larger issue), but one action item I can take on is trying to eliminate the
>> need to do a "git clean" after building certain targets (assuming I can
>> reproduce this).
>>
>> Perhaps we should go through and prioritize (and add missing items to)
>> BEAM-4045
>> https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC
>> ? There's always a long tail with this kind of thing, and looking at the
>> whole list can be daunting, but putting it in the correct order and
>> knocking off the top N items could possibly go a long way.
>>
>> - Robert
>>
>> [1] The commands I ran were (with and without the uniq)
>>
>> $ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o
>> '[^< ]*@' | sort | uniq | wc
>> $ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^<
>> ]*@' | sort | uniq | wc
>>
>> On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <ec...@apache.org>
>> wrote:
>>
>>> Hi all,
>>> I must admit that I agree on the status especially regarding 2 points:
>>> 1. new contributors obstacles: gradle learning curve might be too long
>>> for spare-time contributors, also complex scripted build takes time to
>>> understand comparing to self-descriptive one.
>>> 2. IDE integration kind of slows down development.
>>>
>>> Now, regarding how we improve the situation, I think we need to discuss
>>> and identify tasks and tackle them all together even if they are not sexy
>>> tasks as Ismaël mentioned.
>>>
>>> Etienne
>>>
>>> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>>>
>>> Hi guys,
>>>
>>>
>>> I know that's a hot topic, but I have to bring this discussion on the table.
>>>
>>>
>>> Some months ago, we discussed about migrating our build from Maven to
>>>
>>> Gradle. One of the key expected improvement was the time to build.
>>>
>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>
>>> this PoC was actually directly a migrate on master.
>>>
>>>
>>> Now, I would like to bring facts here:
>>>
>>>
>>> 1. Build time
>>>
>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>
>>> regarding what the build is doing, I don't see huge improvement provided
>>>
>>> by Gradle.
>>>
>>> 2. Build reliability
>>>
>>> Even worse, most of the time, we need to use --no-parallel and
>>>
>>> --no-daemon to have a reliable build (it's basically recommended for
>>>
>>> release). It has an impact on build time, and we loose part of Gradle
>>>
>>> benefits.
>>>
>>> 3. Release and repositories
>>>
>>> Even if couple of releases has been performed with Gradle, it's not
>>>
>>> obvious to see improvements around artifacts handling. I got my
>>>
>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>
>>> speed up the build dealing around the repository).
>>>
>>> 4. IDE integration
>>>
>>> We already had some comments on the mailing lists about the IDE
>>>
>>> integration. Clearly, the situation is not good on that front too. The
>>>
>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>
>>>
>>> We are working hard to grow up the community, and from a contributor
>>>
>>> perspective, our build system is not good today IMHO.
>>>
>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>
>>> much time of the build, largely more than working on the PRs code itself.
>>>
>>>
>>> So, obviously, the situation is not perfect, at least from a contributor
>>>
>>> perspective.
>>>
>>>
>>> The purpose of this thread is not again to have a bunch of replied
>>>
>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>
>>> concrete. So basically, we only have two options:
>>>
>>>
>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>>
>>> such sense from a contributor perspective, as Maven is really well known
>>>
>>> from most of contributors (and easier to start with IMHO).
>>>
>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>
>>> is better, Maven is well known from the contributors as already said.
>>>
>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>
>>> expected results now, that's not a problem, it's part of a project lifetime.
>>>
>>>
>>> Thoughts ?
>>>
>>>
>>> Regards
>>>
>>> JB
>>>
>>>
>>>
>>>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>
Re: [DISCUSS] Gradle for the build ?
Posted by Scott Wegner <sc...@apache.org>.
> Perhaps we should go through and prioritize (and add missing items to)
BEAM-4045
+1. It's hard to know where to start when there's such a laundry list of
tasks. If you're having build issues, will you make sure it is represented
in BEAM-4045, and "Vote" for the issues that you believe are the highest
priority?
I agree that the Gradle build is far from perfect (my top gripes are IDE
integration and parallel/incremental build support). I believe that we're
capable of making our build great, and continuing our investment in Gradle
would be a shorter path than changing course again. Remember that our Maven
build also had it's share of issues, which is why we as a community voted
to replace it [1][2].
It seems we also need to help build Gradle expertise in our community, so
that those that are motivated are empowered to contribute. Does anybody
have a good "Getting Started with Gradle" guide they recommend? Perhaps we
could also link to it from the website/wiki.
[1]
https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
[2]
https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E
On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <ro...@google.com> wrote:
> Some rough stats (because I was curious): The gradle files have been
> edited by ~79 unique contributors over 696 distinct commits, whereas the
> maven ones were edited (over a longer time period) by ~130 unique
> contributors over 1389 commits [1]. This doesn't capture how much effort
> was put into these edits, but neither is restricted to a small set of
> experts.
>
> Regarding "friendly for other languages" I don't think either is
> necessarily easy to learn, but my impression is that the maven learning
> curve shallower for those already firmly embedded in the Java ecosystem
> (perhaps due to leveraging existing familiarity, and perhaps some due to
> the implicit java-centric conventions that maven assumed about your
> project), whereas with gradle at least I could keep pulling on the string
> to unwind things to the bottom. The "I just want to build/test X without
> editing/viewing the build files" seemed more natural with Gradle (e.g. I
> can easily list all tasks).
>
> That being said, I don't think everyone needs to understand the full build
> system. It's important that there be a critical mass that do (we have that
> for both, and if we can simplify to improve this that'd be great), it's
> easy enough to do basic changes (e.g. add a dependency, again I don't think
> the barrier is sufficiently different for either), and works well out of
> the box for someone who just wants to look up a command on the website and
> edit code (the CLI is an improvement with Gradle, but it's clear that
> (java) IDE support is a significant regression).
>
> Personally, I don't know much about IDE configuration (admittedly the
> larger issue), but one action item I can take on is trying to eliminate the
> need to do a "git clean" after building certain targets (assuming I can
> reproduce this).
>
> Perhaps we should go through and prioritize (and add missing items to)
> BEAM-4045
> https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC
> ? There's always a long tail with this kind of thing, and looking at the
> whole list can be daunting, but putting it in the correct order and
> knocking off the top N items could possibly go a long way.
>
> - Robert
>
> [1] The commands I ran were (with and without the uniq)
>
> $ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o
> '[^< ]*@' | sort | uniq | wc
> $ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^<
> ]*@' | sort | uniq | wc
>
> On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi all,
>> I must admit that I agree on the status especially regarding 2 points:
>> 1. new contributors obstacles: gradle learning curve might be too long
>> for spare-time contributors, also complex scripted build takes time to
>> understand comparing to self-descriptive one.
>> 2. IDE integration kind of slows down development.
>>
>> Now, regarding how we improve the situation, I think we need to discuss
>> and identify tasks and tackle them all together even if they are not sexy
>> tasks as Ismaël mentioned.
>>
>> Etienne
>>
>> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>>
>> Hi guys,
>>
>>
>> I know that's a hot topic, but I have to bring this discussion on the table.
>>
>>
>> Some months ago, we discussed about migrating our build from Maven to
>>
>> Gradle. One of the key expected improvement was the time to build.
>>
>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>
>> this PoC was actually directly a migrate on master.
>>
>>
>> Now, I would like to bring facts here:
>>
>>
>> 1. Build time
>>
>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>
>> regarding what the build is doing, I don't see huge improvement provided
>>
>> by Gradle.
>>
>> 2. Build reliability
>>
>> Even worse, most of the time, we need to use --no-parallel and
>>
>> --no-daemon to have a reliable build (it's basically recommended for
>>
>> release). It has an impact on build time, and we loose part of Gradle
>>
>> benefits.
>>
>> 3. Release and repositories
>>
>> Even if couple of releases has been performed with Gradle, it's not
>>
>> obvious to see improvements around artifacts handling. I got my
>>
>> repository polluted twice (that's part of the trick Gradle is doing to
>>
>> speed up the build dealing around the repository).
>>
>> 4. IDE integration
>>
>> We already had some comments on the mailing lists about the IDE
>>
>> integration. Clearly, the situation is not good on that front too. The
>>
>> integration on IDE (especially IntelliJ) is not good enough right now.
>>
>>
>> We are working hard to grow up the community, and from a contributor
>>
>> perspective, our build system is not good today IMHO.
>>
>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>
>> much time of the build, largely more than working on the PRs code itself.
>>
>>
>> So, obviously, the situation is not perfect, at least from a contributor
>>
>> perspective.
>>
>>
>> The purpose of this thread is not again to have a bunch of replied
>>
>> ending nowhere. I would like to be more "pushy" and let's try to be
>>
>> concrete. So basically, we only have two options:
>>
>>
>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>
>> such sense from a contributor perspective, as Maven is really well known
>>
>> from most of contributors (and easier to start with IMHO).
>>
>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>
>> is better, Maven is well known from the contributors as already said.
>>
>> The effort is not so huge. We tried to use Gradle, we don't have the
>>
>> expected results now, that's not a problem, it's part of a project lifetime.
>>
>>
>> Thoughts ?
>>
>>
>> Regards
>>
>> JB
>>
>>
>>
>>
--
Got feedback? tinyurl.com/swegner-feedback
Re: [DISCUSS] Gradle for the build ?
Posted by Robert Bradshaw <ro...@google.com>.
Some rough stats (because I was curious): The gradle files have been edited
by ~79 unique contributors over 696 distinct commits, whereas the maven
ones were edited (over a longer time period) by ~130 unique
contributors over 1389 commits [1]. This doesn't capture how much effort
was put into these edits, but neither is restricted to a small set of
experts.
Regarding "friendly for other languages" I don't think either is
necessarily easy to learn, but my impression is that the maven learning
curve shallower for those already firmly embedded in the Java ecosystem
(perhaps due to leveraging existing familiarity, and perhaps some due to
the implicit java-centric conventions that maven assumed about your
project), whereas with gradle at least I could keep pulling on the string
to unwind things to the bottom. The "I just want to build/test X without
editing/viewing the build files" seemed more natural with Gradle (e.g. I
can easily list all tasks).
That being said, I don't think everyone needs to understand the full build
system. It's important that there be a critical mass that do (we have that
for both, and if we can simplify to improve this that'd be great), it's
easy enough to do basic changes (e.g. add a dependency, again I don't think
the barrier is sufficiently different for either), and works well out of
the box for someone who just wants to look up a command on the website and
edit code (the CLI is an improvement with Gradle, but it's clear that
(java) IDE support is a significant regression).
Personally, I don't know much about IDE configuration (admittedly the
larger issue), but one action item I can take on is trying to eliminate the
need to do a "git clean" after building certain targets (assuming I can
reproduce this).
Perhaps we should go through and prioritize (and add missing items to)
BEAM-4045
https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC
? There's always a long tail with this kind of thing, and looking at the
whole list can be daunting, but putting it in the correct order and
knocking off the top N items could possibly go a long way.
- Robert
[1] The commands I ran were (with and without the uniq)
$ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o '[^<
]*@' | sort | uniq | wc
$ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^< ]*@'
| sort | uniq | wc
On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi all,
> I must admit that I agree on the status especially regarding 2 points:
> 1. new contributors obstacles: gradle learning curve might be too long for
> spare-time contributors, also complex scripted build takes time to
> understand comparing to self-descriptive one.
> 2. IDE integration kind of slows down development.
>
> Now, regarding how we improve the situation, I think we need to discuss
> and identify tasks and tackle them all together even if they are not sexy
> tasks as Ismaël mentioned.
>
> Etienne
>
> Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
>
> Hi guys,
>
>
> I know that's a hot topic, but I have to bring this discussion on the table.
>
>
> Some months ago, we discussed about migrating our build from Maven to
>
> Gradle. One of the key expected improvement was the time to build.
>
> We proposed to do a PoC to evaluate the impacts and improvements, but
>
> this PoC was actually directly a migrate on master.
>
>
> Now, I would like to bring facts here:
>
>
> 1. Build time
>
> On my machine, the build time is roughly 1h15. It's pretty long, and
>
> regarding what the build is doing, I don't see huge improvement provided
>
> by Gradle.
>
> 2. Build reliability
>
> Even worse, most of the time, we need to use --no-parallel and
>
> --no-daemon to have a reliable build (it's basically recommended for
>
> release). It has an impact on build time, and we loose part of Gradle
>
> benefits.
>
> 3. Release and repositories
>
> Even if couple of releases has been performed with Gradle, it's not
>
> obvious to see improvements around artifacts handling. I got my
>
> repository polluted twice (that's part of the trick Gradle is doing to
>
> speed up the build dealing around the repository).
>
> 4. IDE integration
>
> We already had some comments on the mailing lists about the IDE
>
> integration. Clearly, the situation is not good on that front too. The
>
> integration on IDE (especially IntelliJ) is not good enough right now.
>
>
> We are working hard to grow up the community, and from a contributor
>
> perspective, our build system is not good today IMHO.
>
> As a contributor, I resumed my work on some PRs, and I'm spending so
>
> much time of the build, largely more than working on the PRs code itself.
>
>
> So, obviously, the situation is not perfect, at least from a contributor
>
> perspective.
>
>
> The purpose of this thread is not again to have a bunch of replied
>
> ending nowhere. I would like to be more "pushy" and let's try to be
>
> concrete. So basically, we only have two options:
>
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>
> such sense from a contributor perspective, as Maven is really well known
>
> from most of contributors (and easier to start with IMHO).
>
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>
> is better, Maven is well known from the contributors as already said.
>
> The effort is not so huge. We tried to use Gradle, we don't have the
>
> expected results now, that's not a problem, it's part of a project lifetime.
>
>
> Thoughts ?
>
>
> Regards
>
> JB
>
>
>
>
Re: [DISCUSS] Gradle for the build ?
Posted by Etienne Chauchot <ec...@apache.org>.
Hi all,
I must admit that I agree on the status especially regarding 2 points:
1. new contributors obstacles: gradle learning curve might be too long for spare-time contributors, also complex
scripted build takes time to understand comparing to self-descriptive one.
2. IDE integration kind of slows down development.
Now, regarding how we improve the situation, I think we need to discuss and identify tasks and tackle them all together
even if they are not sexy tasks as Ismaël mentioned.
Etienne
Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion on the table.
>
> Some months ago, we discussed about migrating our build from Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement provided
> by Gradle.
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of Gradle
> benefits.
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is doing to
> speed up the build dealing around the repository).
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front too. The
> integration on IDE (especially IntelliJ) is not good enough right now.
>
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs code itself.
>
> So, obviously, the situation is not perfect, at least from a contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
> such sense from a contributor perspective, as Maven is really well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
> is better, Maven is well known from the contributors as already said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a project lifetime.
>
> Thoughts ?
>
> Regards
> JB
>
>
Re: [DISCUSS] Gradle for the build ?
Posted by Maximilian Michels <mx...@apache.org>.
Thanks for bringing this up, JB! I think we are all very eager to do
something about the current situation.
Frankly, I agree with Reuven/Łukasz - going back at this point will come
at a huge cost. I think going Maven -> Gradle is much easier than going
the opposite way.
The situation is not that bad. The build system works ok but there are
some rough edges. The main problem for me is the IDE support which seems
like it could be fixed if somebody looked into this a bit more. Build
time was worse with Maven. Build reliability has been ok for me.
Maybe the team/person mainly involved in the Gradle refactoring could
comment on this as well?
-Max
On 09.10.18 11:50, Łukasz Gajowy wrote:
> Thanks for starting this thread.
>
> I would go for 1. To all the problems that you described I'd add
> improving documentation - there are still some places that have only
> maven instructions (eg. word count examples). There already are some
> docs that provide help [1], [2]. Should we incorporate them to
> Beam's confluence?
>
> I don't think 2 is the way to go. All the problems described above do
> not seem to be unfixable (are the issues with the current Gradle setup
> are impossible to fix?). We should focus on the fixes not on bringing
> back maven and adapting it to current requirements.
>
> FWIW, from my experience in working on contributions, I'm not affected
> by the first three problems that you described. I rarely need to build
> the whole project and I never had problems with daemon and
> parallel builds. It is also easier for me to work with Gradle thanks to
> its great DSL and composable tasks.
>
> Łukasz
>
> [1]
> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit?usp=sharing
> [2]
> https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
>
> wt., 9 paź 2018 o 10:25 Reuven Lax <relax@google.com
> <ma...@google.com>> napisał(a):
>
> I'm not going to comment too much on most of these points (as I
> think others can do so better). However I think that the effort
> required to migrate back to Maven will actually be quite
> significant. Much has been added to Beam (both to the codebase, and
> to our Jenkins tooling, etc.) since we moved to Gradle, and none of
> this has been added to Maven. I believe that going back and
> migrating all of this to Maven will be difficult at this point.
>
> I would vote for option 1. I believe that many of the current
> issues are easily fixable. For example, requiring no-parallel I
> believe is because some of our dependencies are incorrectly setup in
> gradle files, and nobody has taken the time to track this down and
> fix it (it was easier to just start setting that flag).
>
> Reuven
>
> On Tue, Oct 9, 2018 at 1:04 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion
> on the table.
>
> Some months ago, we discussed about migrating our build from
> Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and
> improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement
> provided
> by Gradle.
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of
> Gradle
> benefits.
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is
> doing to
> speed up the build dealing around the repository).
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front
> too. The
> integration on IDE (especially IntelliJ) is not good enough
> right now.
>
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs
> code itself.
>
> So, obviously, the situation is not perfect, at least from a
> contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if
> it makes
> such sense from a contributor perspective, as Maven is really
> well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE
> integration
> is better, Maven is well known from the contributors as already
> said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a
> project lifetime.
>
> Thoughts ?
>
> Regards
> JB
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [DISCUSS] Gradle for the build ?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Don't forget the point about community and contributor perspective. If it can be fixed just by documentation, that's great.
Regards
JB
Le 9 oct. 2018 à 12:50, à 12:50, "Łukasz Gajowy" <lg...@apache.org> a écrit:
>Thanks for starting this thread.
>
>I would go for 1. To all the problems that you described I'd add
>improving
>documentation - there are still some places that have only
>maven instructions (eg. word count examples). There already are some
>docs
>that provide help [1], [2]. Should we incorporate them to Beam's
>confluence?
>
>I don't think 2 is the way to go. All the problems described above do
>not
>seem to be unfixable (are the issues with the current Gradle setup are
>impossible to fix?). We should focus on the fixes not on bringing back
>maven and adapting it to current requirements.
>
>FWIW, from my experience in working on contributions, I'm not affected
>by
>the first three problems that you described. I rarely need to build the
>whole project and I never had problems with daemon and parallel builds.
>It
>is also easier for me to work with Gradle thanks to its great DSL and
>composable tasks.
>
>Łukasz
>
>[1]
>https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit?usp=sharing
>[2]
>https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
>
>wt., 9 paź 2018 o 10:25 Reuven Lax <re...@google.com> napisał(a):
>
>> I'm not going to comment too much on most of these points (as I think
>> others can do so better). However I think that the effort required to
>> migrate back to Maven will actually be quite significant. Much has
>been
>> added to Beam (both to the codebase, and to our Jenkins tooling,
>etc.)
>> since we moved to Gradle, and none of this has been added to Maven. I
>> believe that going back and migrating all of this to Maven will be
>> difficult at this point.
>>
>> I would vote for option 1. I believe that many of the current issues
>are
>> easily fixable. For example, requiring no-parallel I believe is
>because
>> some of our dependencies are incorrectly setup in gradle files, and
>nobody
>> has taken the time to track this down and fix it (it was easier to
>just
>> start setting that flag).
>>
>> Reuven
>>
>> On Tue, Oct 9, 2018 at 1:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi guys,
>>>
>>> I know that's a hot topic, but I have to bring this discussion on
>the
>>> table.
>>>
>>> Some months ago, we discussed about migrating our build from Maven
>to
>>> Gradle. One of the key expected improvement was the time to build.
>>> We proposed to do a PoC to evaluate the impacts and improvements,
>but
>>> this PoC was actually directly a migrate on master.
>>>
>>> Now, I would like to bring facts here:
>>>
>>> 1. Build time
>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>> regarding what the build is doing, I don't see huge improvement
>provided
>>> by Gradle.
>>> 2. Build reliability
>>> Even worse, most of the time, we need to use --no-parallel and
>>> --no-daemon to have a reliable build (it's basically recommended for
>>> release). It has an impact on build time, and we loose part of
>Gradle
>>> benefits.
>>> 3. Release and repositories
>>> Even if couple of releases has been performed with Gradle, it's not
>>> obvious to see improvements around artifacts handling. I got my
>>> repository polluted twice (that's part of the trick Gradle is doing
>to
>>> speed up the build dealing around the repository).
>>> 4. IDE integration
>>> We already had some comments on the mailing lists about the IDE
>>> integration. Clearly, the situation is not good on that front too.
>The
>>> integration on IDE (especially IntelliJ) is not good enough right
>now.
>>>
>>> We are working hard to grow up the community, and from a contributor
>>> perspective, our build system is not good today IMHO.
>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>> much time of the build, largely more than working on the PRs code
>itself.
>>>
>>> So, obviously, the situation is not perfect, at least from a
>contributor
>>> perspective.
>>>
>>> The purpose of this thread is not again to have a bunch of replied
>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>> concrete. So basically, we only have two options:
>>>
>>> 1. Improve the build, working hard on Gradle front. Not sure if it
>makes
>>> such sense from a contributor perspective, as Maven is really well
>known
>>> from most of contributors (and easier to start with IMHO).
>>> 2. Back on Maven. That's clearly my preferred approach. IDE
>integration
>>> is better, Maven is well known from the contributors as already
>said.
>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>> expected results now, that's not a problem, it's part of a project
>>> lifetime.
>>>
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
Re: [DISCUSS] Gradle for the build ?
Posted by Łukasz Gajowy <lg...@apache.org>.
Thanks for starting this thread.
I would go for 1. To all the problems that you described I'd add improving
documentation - there are still some places that have only
maven instructions (eg. word count examples). There already are some docs
that provide help [1], [2]. Should we incorporate them to Beam's confluence?
I don't think 2 is the way to go. All the problems described above do not
seem to be unfixable (are the issues with the current Gradle setup are
impossible to fix?). We should focus on the fixes not on bringing back
maven and adapting it to current requirements.
FWIW, from my experience in working on contributions, I'm not affected by
the first three problems that you described. I rarely need to build the
whole project and I never had problems with daemon and parallel builds. It
is also easier for me to work with Gradle thanks to its great DSL and
composable tasks.
Łukasz
[1]
https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit?usp=sharing
[2]
https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
wt., 9 paź 2018 o 10:25 Reuven Lax <re...@google.com> napisał(a):
> I'm not going to comment too much on most of these points (as I think
> others can do so better). However I think that the effort required to
> migrate back to Maven will actually be quite significant. Much has been
> added to Beam (both to the codebase, and to our Jenkins tooling, etc.)
> since we moved to Gradle, and none of this has been added to Maven. I
> believe that going back and migrating all of this to Maven will be
> difficult at this point.
>
> I would vote for option 1. I believe that many of the current issues are
> easily fixable. For example, requiring no-parallel I believe is because
> some of our dependencies are incorrectly setup in gradle files, and nobody
> has taken the time to track this down and fix it (it was easier to just
> start setting that flag).
>
> Reuven
>
> On Tue, Oct 9, 2018 at 1:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi guys,
>>
>> I know that's a hot topic, but I have to bring this discussion on the
>> table.
>>
>> Some months ago, we discussed about migrating our build from Maven to
>> Gradle. One of the key expected improvement was the time to build.
>> We proposed to do a PoC to evaluate the impacts and improvements, but
>> this PoC was actually directly a migrate on master.
>>
>> Now, I would like to bring facts here:
>>
>> 1. Build time
>> On my machine, the build time is roughly 1h15. It's pretty long, and
>> regarding what the build is doing, I don't see huge improvement provided
>> by Gradle.
>> 2. Build reliability
>> Even worse, most of the time, we need to use --no-parallel and
>> --no-daemon to have a reliable build (it's basically recommended for
>> release). It has an impact on build time, and we loose part of Gradle
>> benefits.
>> 3. Release and repositories
>> Even if couple of releases has been performed with Gradle, it's not
>> obvious to see improvements around artifacts handling. I got my
>> repository polluted twice (that's part of the trick Gradle is doing to
>> speed up the build dealing around the repository).
>> 4. IDE integration
>> We already had some comments on the mailing lists about the IDE
>> integration. Clearly, the situation is not good on that front too. The
>> integration on IDE (especially IntelliJ) is not good enough right now.
>>
>> We are working hard to grow up the community, and from a contributor
>> perspective, our build system is not good today IMHO.
>> As a contributor, I resumed my work on some PRs, and I'm spending so
>> much time of the build, largely more than working on the PRs code itself.
>>
>> So, obviously, the situation is not perfect, at least from a contributor
>> perspective.
>>
>> The purpose of this thread is not again to have a bunch of replied
>> ending nowhere. I would like to be more "pushy" and let's try to be
>> concrete. So basically, we only have two options:
>>
>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>> such sense from a contributor perspective, as Maven is really well known
>> from most of contributors (and easier to start with IMHO).
>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>> is better, Maven is well known from the contributors as already said.
>> The effort is not so huge. We tried to use Gradle, we don't have the
>> expected results now, that's not a problem, it's part of a project
>> lifetime.
>>
>> Thoughts ?
>>
>> Regards
>> JB
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: [DISCUSS] Gradle for the build ?
Posted by Reuven Lax <re...@google.com>.
I'm not going to comment too much on most of these points (as I think
others can do so better). However I think that the effort required to
migrate back to Maven will actually be quite significant. Much has been
added to Beam (both to the codebase, and to our Jenkins tooling, etc.)
since we moved to Gradle, and none of this has been added to Maven. I
believe that going back and migrating all of this to Maven will be
difficult at this point.
I would vote for option 1. I believe that many of the current issues are
easily fixable. For example, requiring no-parallel I believe is because
some of our dependencies are incorrectly setup in gradle files, and nobody
has taken the time to track this down and fix it (it was easier to just
start setting that flag).
Reuven
On Tue, Oct 9, 2018 at 1:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion on the
> table.
>
> Some months ago, we discussed about migrating our build from Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement provided
> by Gradle.
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of Gradle
> benefits.
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is doing to
> speed up the build dealing around the repository).
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front too. The
> integration on IDE (especially IntelliJ) is not good enough right now.
>
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs code itself.
>
> So, obviously, the situation is not perfect, at least from a contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
> such sense from a contributor perspective, as Maven is really well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
> is better, Maven is well known from the contributors as already said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a project
> lifetime.
>
> Thoughts ?
>
> Regards
> JB
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [DISCUSS] Gradle for the build ?
Posted by Ismaël Mejía <ie...@gmail.com>.
JB mentioned some factual points, I think most of the community
embraced the move to gradle with the best hopes about the promised
improvements, but with the perspective of time, it is clear that we
have not delivered on many of them. I can easily understand the
growing bitterness about the current status. There are two aspects
that really bother me:
1. Integration with IDEs is BAD and this not only has seriously
affected everybody’s daily work, but I think it does not help to
encourage new contributors.
2. Gradle has a ridiculously huge learning curve, first you need to
understand groovy and its intricacies, then gradle and its magic
around caches, daemons, etc and finally grasp a good chunk of our hand
tailored BeamModulePlugin configuration. I cannot see how the argument
that this is friendly for other language contributors holds.
A consequence of (2) is that so far I have barely seen any
contribution from any person apart of the ones who worked in the move
to gradle on improvements on the build. So we also lost contributions
in this area.
Now the question is how to solve this or improve the current
situation. I really have no clue of how we can improve the IDE
situation, (for me the most critical point), or how we can improve all
the pending points of BEAM-4045, but I can understand the apathy in
the subject, improving the build system is not sexy, and previous
contributors don’t feel encouraged to fix what wasn’t broken in the
past. In any case it is good that we are having an open discussion on
this.
On Wed, Oct 10, 2018 at 6:35 AM Kenneth Knowles <ke...@apache.org> wrote:
>
> Here are some things I hear a lot:
>
> Beam technical decision making is too Java-centric
> Gradle is a lot less Java-centric than Maven
> Gradle is still a Java-centric tool, but at least it isn't so slow/wasteful
>
> I can understand long-time Java devs wanting to have their familiar and dominant toolchain, with good tooling and integrations (ignoring any objective merits of Gradle for Java dev). But for non-Java devs you are meeting in the middle. Actually, you are not even in the middle - you are still in Java land. For example, setup.py can do arbitrary things, so we could use it to do the Java build! What do you think of trying this out? (j/k).
>
> What I would like is to better support language-native development workflows, and make sure Gradle is lightweight glue that is easy to use. Make the configurations as obvious to read as we can, with as little hackery as possible.
>
> ----------
>
> We can make Gradle a lot better (for the above, and for Java too). From my work on the core of our Gradle support code, I can name a few issues that I'm sure remain even after many months away from the code:
>
> 1. We were learning as we built it out. We don't always do things in the natural way.
> 2. We made our own abstractions to streamline converting lots of modules quickly. Clarity and efficiency of the build were not the primary concerns. (to be clear: it was an amazing undertaking to get this done and the decision was right for the moment)
> 2a. We tried to centralize a lot of policy, leading to the centralized bits containing the union of all complexity of all modules (or maybe it is multiplicative).
> 3. We tried very hard to match the mvn build exactly, rather than doing the best thing in Gradle.
> 4. We've built a lot of imperative code for "telling Gradle what to do" and that adds a lot of complexity compared with using Groovy as primarily a configuration language.
> 5. We have turned on things like "always rebuild" when we don't know the dependencies, rather than putting in the work to get the dependencies right.
>
> A lot of the above also makes it hard for IDEs to grok the config, since we deviate from the "golden path" a lot.
>
> It would be awesome if module owners took on the task of making their modules have an awesome incremental Gradle build.
>
> Kenn
>
> On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rm...@gmail.com> wrote:
>>
>> For me the vendoring issue is ok cause it should belong to another shade loduke released with beam when needed. It is not an uncommon practise.
>>
>> Now the lack of IDE integration for tests/debug (using gradle runner is a workaround and still hurts by its slowness compared to native run) is a clear showstopper for me.
>>
>> Also, from a community perspective, gradle adoption is far to be mainstream (even spark is built with maven) so does not serve beam at the end.
>>
>> Maven build didnt have any issue except the duration AFAIK, gradle has 2 blockers + several small drawbacks (custom build and no standard, no tooling without script execution, bad integration in enterprise chaines like security auditing etc). Overall gradle build is close to maven one - last time i tested it was within 15% so not worth it when you see the time you loose when developping anything. It is key to keep in mind jenkiks is cheaper than human time.
>>
>> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a écrit :
>>>
>>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> I know that's a hot topic, but I have to bring this discussion on the table.
>>>
>>>
>>> Thank you for bringing this up and revisiting it now that we have some experience.
>>>
>>>>
>>>> Some months ago, we discussed about migrating our build from Maven to
>>>> Gradle. One of the key expected improvement was the time to build.
>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>> this PoC was actually directly a migrate on master.
>>>>
>>>> Now, I would like to bring facts here:
>>>>
>>>> 1. Build time
>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>> regarding what the build is doing, I don't see huge improvement provided
>>>> by Gradle.
>>>
>>>
>>> I rarely, if ever, build from scratch so perhaps I have not been impacted by this nearly as much. (In particular, build and test times seem to have gone way down for me, probably due to better incremental support, but that's just anecdotal.)
>>>
>>> Is this worse than it was on maven, or just not as much better as was hoped?
>>>
>>>>
>>>> 2. Build reliability
>>>> Even worse, most of the time, we need to use --no-parallel and
>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>> release). It has an impact on build time, and we loose part of Gradle
>>>> benefits.
>>>
>>>
>>> I think this is a matter of incorrect dependency declarations (and is not unique to gradle). I'd have loved to been able to go with a build system that simply didn't let you have incorrect dependency declarations, but that wasn't an option for other reasons.
>>>
>>> I wonder if there's some automatic tooling we could leverage to fix (and keep fixed) this. Regardless, this is unfinished work that remains to be done so we can realize the full benefits.
>>>
>>>>
>>>> 3. Release and repositories
>>>> Even if couple of releases has been performed with Gradle, it's not
>>>> obvious to see improvements around artifacts handling. I got my
>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>> speed up the build dealing around the repository).
>>>
>>>
>>> Could you clarify what improvements we were expecting here? I thought the goal was that we could publish the same artifacts, with no regression.
>>>
>>>>
>>>> 4. IDE integration
>>>> We already had some comments on the mailing lists about the IDE
>>>> integration. Clearly, the situation is not good on that front too. The
>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>
>>>
>>> This is important. To be honest, I had also issues back in the day getting the maven setup working well out of the box in IntelliJ and Eclipse (mostly with respect to things like shadowing and protobufs), so we shouldn't fall prey to the golden age fallacy.
>>>
>>> It seems the recent move to vendoring has caused more issues here; I'm not sure that would be fixed just moving back to maven (or how to resolve it going forward).
>>>
>>> On the other hand, just last week I set up a new computer according to https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and that seems to be working fine.
>>>
>>>>
>>>> We are working hard to grow up the community, and from a contributor
>>>> perspective, our build system is not good today IMHO.
>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>> much time of the build, largely more than working on the PRs code itself.
>>>>
>>>> So, obviously, the situation is not perfect, at least from a contributor
>>>> perspective.
>>>>
>>>> The purpose of this thread is not again to have a bunch of replied
>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>> concrete. So basically, we only have two options:
>>>>
>>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>>> such sense from a contributor perspective, as Maven is really well known
>>>> from most of contributors (and easier to start with IMHO).
>>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>> is better, Maven is well known from the contributors as already said.
>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>> expected results now, that's not a problem, it's part of a project lifetime.
>>>>
>>>> Thoughts ?
>>>
>>>
>>> I'd like to add some perspective as a primarily non-Java contributor (having contributed "only" a couple thousand lines to the java side over the last couple of years) that maven feels much more java-centric (as is our repo, but that's a separate issue). Also, as someone not coming from with years of maven (or gradle for that matter) experience before working on Beam, I have found gradle much more intuitive and easier to learn. Especially when it comes to developing changes that span multiple modules (which for some reason I guess I tend to do a lot of, but with so many of our core sdk tests being validates runner that's likely to hit people just starting out as well). Now of course I don't want to discount the Java community, indeed it's arguably the largest and most important one at this point, but I also think that Beam's ability to not be limited to that one language (and ecosystem) is one of its huge selling points and differentiation (see the whole portability effort, which is for both SDKs and Runners).
>>>
>>> Even from a Java perspective, neither is so obscure as to be a significant barrier, and both are customizable enough that the average new developer is probably going to be looking at the documentation to see what commands to run to build/test/validate this project. So this is probably something documentation can address.
>>>
>>> But if we're going to reap the benefits and minimize the downsides of this migration, we have to finish the job.
>>>
>>> - Robert
>>>
>>>
Re: [DISCUSS] Gradle for the build ?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Reuven: any idea what is missing? I don't expect it to be ready very
quickly but having 2 repos does not hurt that much if both are working
better than a single one so can be worth a try maybe?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mer. 10 oct. 2018 à 08:31, Reuven Lax <re...@google.com> a écrit :
> Unrelated to the Maven/Gradle discussion, but I do somewhat agree with
> Romain that Beam could be split into separate repos, however I don't think
> it's quite ready yet. AFAIK the portability interfaces are still being
> modified, and until these interfaces are fixed it will be hard to split the
> project. Once those interfaces are fixed and mature, I completely agree we
> should then look into splitting the repos up.
>
> Reuven
>
> On Tue, Oct 9, 2018 at 10:35 PM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>>
>>
>>
>> Le mer. 10 oct. 2018 à 06:35, Kenneth Knowles <ke...@apache.org> a écrit :
>>
>>> Here are some things I hear a lot:
>>>
>>> - Beam technical decision making is too Java-centric
>>> - Gradle is a lot less Java-centric than Maven
>>>
>>> Hmm isnt it the opposite being groovy based? Maven tends to have the
>> same set of plugins than gradle so it is a kind of status-quo if you check
>> technically and are factual.
>>
>>
>>>
>>> - Gradle is still a Java-centric tool, but at least it isn't so
>>> slow/wasteful
>>>
>>>
>> Not what I experienced and it seems JB had the same experience. Gradle is
>> comparable to maven in terms of full build and is way slower in Idea cause
>> of the lack of integration and support (by design).
>>
>>
>>> I can understand long-time Java devs wanting to have their familiar and
>>> dominant toolchain, with good tooling and integrations (ignoring any
>>> objective merits of Gradle for Java dev). But for non-Java devs you are
>>> meeting in the middle. Actually, you are not even in the middle - you are
>>> still in Java land. For example, setup.py can do arbitrary things, so we
>>> could use it to do the Java build! What do you think of trying this out?
>>> (j/k).
>>>
>>> What I would like is to better support language-native development
>>> workflows, and make sure Gradle is lightweight glue that is easy to use.
>>> Make the configurations as obvious to read as we can, with as little
>>> hackery as possible.
>>>
>>
>> Agree and think it can be time, now Beam python/go support become
>> something, to split in N repos. Release lifecycles are different, codebase
>> has no real link except the portability layer which can get its own repo so
>> probably worth a PoC:
>>
>> 1. beam-portability
>> 2. beam-java
>> 3. beam-go
>> 4. beam-python
>>
>> Side note: beam portability would be saner if added on top of others than
>> the opposite which is done today.
>>
>>
>>>
>>> ----------
>>>
>>> We can make Gradle a lot better (for the above, and for Java too). From
>>> my work on the core of our Gradle support code, I can name a few issues
>>> that I'm sure remain even after many months away from the code:
>>>
>>> 1. We were learning as we built it out. We don't always do things in the
>>> natural way.
>>> 2. We made our own abstractions to streamline converting lots of modules
>>> quickly. Clarity and efficiency of the build were not the primary concerns.
>>> (to be clear: it was an amazing undertaking to get this done and the
>>> decision was right for the moment)
>>> 2a. We tried to centralize a lot of policy, leading to the centralized
>>> bits containing the union of all complexity of all modules (or maybe it is
>>> multiplicative).
>>> 3. We tried very hard to match the mvn build exactly, rather than doing
>>> the best thing in Gradle.
>>> 4. We've built a lot of imperative code for "telling Gradle what to do"
>>> and that adds a lot of complexity compared with using Groovy as primarily a
>>> configuration language.
>>> 5. We have turned on things like "always rebuild" when we don't know the
>>> dependencies, rather than putting in the work to get the dependencies right.
>>>
>>> A lot of the above also makes it hard for IDEs to grok the config, since
>>> we deviate from the "golden path" a lot.
>>>
>>> It would be awesome if module owners took on the task of making their
>>> modules have an awesome incremental Gradle build.
>>>
>>
>> Well incremental build only matters when you run a full subpart of a job
>> and is something very fragile - check how in months it didnt happen. In
>> practise what's a dev workflow:
>>
>> 1. loop { dev in the IDE, run test }
>> 2. run the full module or build to validate nothing has been broken
>> 3. PR (+ back to 1 if comments)
>>
>> This means that incremental support is only relevant for jenkins where
>> the perf diff is not significative and where you can't use incremental
>> build cause you want to have a fully reliable build.
>> So at the end the incremental build support is not that significative for
>> end users and contibutors. The cost of not having the IDE support, however,
>> is just a blocker.
>>
>> So my 2cts would be to stop trying to be good theorically at the cost of
>> loosing the users and try to embrace a community driven choices approach.
>>
>>
>>>
>>> Kenn
>>>
>>> On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rm...@gmail.com>
>>> wrote:
>>>
>>>> For me the vendoring issue is ok cause it should belong to another
>>>> shade loduke released with beam when needed. It is not an uncommon practise.
>>>>
>>>> Now the lack of IDE integration for tests/debug (using gradle runner is
>>>> a workaround and still hurts by its slowness compared to native run) is a
>>>> clear showstopper for me.
>>>>
>>>> Also, from a community perspective, gradle adoption is far to be
>>>> mainstream (even spark is built with maven) so does not serve beam at the
>>>> end.
>>>>
>>>> Maven build didnt have any issue except the duration AFAIK, gradle has
>>>> 2 blockers + several small drawbacks (custom build and no standard, no
>>>> tooling without script execution, bad integration in enterprise chaines
>>>> like security auditing etc). Overall gradle build is close to maven one -
>>>> last time i tested it was within 15% so not worth it when you see the time
>>>> you loose when developping anything. It is key to keep in mind jenkiks is
>>>> cheaper than human time.
>>>>
>>>> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a
>>>> écrit :
>>>>
>>>>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I know that's a hot topic, but I have to bring this discussion on the
>>>>>> table.
>>>>>>
>>>>>
>>>>> Thank you for bringing this up and revisiting it now that we have some
>>>>> experience.
>>>>>
>>>>>
>>>>>> Some months ago, we discussed about migrating our build from Maven to
>>>>>> Gradle. One of the key expected improvement was the time to build.
>>>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>>>> this PoC was actually directly a migrate on master.
>>>>>>
>>>>>> Now, I would like to bring facts here:
>>>>>>
>>>>>> 1. Build time
>>>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>>>> regarding what the build is doing, I don't see huge improvement
>>>>>> provided
>>>>>> by Gradle.
>>>>>>
>>>>>
>>>>> I rarely, if ever, build from scratch so perhaps I have not been
>>>>> impacted by this nearly as much. (In particular, build and test times seem
>>>>> to have gone way down for me, probably due to better incremental support,
>>>>> but that's just anecdotal.)
>>>>>
>>>>> Is this worse than it was on maven, or just not as much better as was
>>>>> hoped?
>>>>>
>>>>>
>>>>>> 2. Build reliability
>>>>>> Even worse, most of the time, we need to use --no-parallel and
>>>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>>>> release). It has an impact on build time, and we loose part of Gradle
>>>>>> benefits.
>>>>>>
>>>>>
>>>>> I think this is a matter of incorrect dependency declarations (and is
>>>>> not unique to gradle). I'd have loved to been able to go with a build
>>>>> system that simply didn't let you have incorrect dependency declarations,
>>>>> but that wasn't an option for other reasons.
>>>>>
>>>>> I wonder if there's some automatic tooling we could leverage to fix
>>>>> (and keep fixed) this. Regardless, this is unfinished work that remains to
>>>>> be done so we can realize the full benefits.
>>>>>
>>>>>
>>>>>> 3. Release and repositories
>>>>>> Even if couple of releases has been performed with Gradle, it's not
>>>>>> obvious to see improvements around artifacts handling. I got my
>>>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>>>> speed up the build dealing around the repository).
>>>>>>
>>>>>
>>>>> Could you clarify what improvements we were expecting here? I thought
>>>>> the goal was that we could publish the same artifacts, with no regression.
>>>>>
>>>>>
>>>>>> 4. IDE integration
>>>>>> We already had some comments on the mailing lists about the IDE
>>>>>> integration. Clearly, the situation is not good on that front too. The
>>>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>>>>
>>>>>
>>>>> This is important. To be honest, I had also issues back in the day
>>>>> getting the maven setup working well out of the box in IntelliJ and Eclipse
>>>>> (mostly with respect to things like shadowing and protobufs), so we
>>>>> shouldn't fall prey to the golden age fallacy.
>>>>>
>>>>> It seems the recent move to vendoring has caused more issues here; I'm
>>>>> not sure that would be fixed just moving back to maven (or how to resolve
>>>>> it going forward).
>>>>>
>>>>> On the other hand, just last week I set up a new computer according to
>>>>> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and
>>>>> that seems to be working fine.
>>>>>
>>>>>
>>>>>> We are working hard to grow up the community, and from a contributor
>>>>>> perspective, our build system is not good today IMHO.
>>>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>>>> much time of the build, largely more than working on the PRs code
>>>>>> itself.
>>>>>>
>>>>>> So, obviously, the situation is not perfect, at least from a
>>>>>> contributor
>>>>>> perspective.
>>>>>>
>>>>>> The purpose of this thread is not again to have a bunch of replied
>>>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>>>> concrete. So basically, we only have two options:
>>>>>>
>>>>>> 1. Improve the build, working hard on Gradle front. Not sure if it
>>>>>> makes
>>>>>> such sense from a contributor perspective, as Maven is really well
>>>>>> known
>>>>>> from most of contributors (and easier to start with IMHO).
>>>>>> 2. Back on Maven. That's clearly my preferred approach. IDE
>>>>>> integration
>>>>>> is better, Maven is well known from the contributors as already said.
>>>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>>>> expected results now, that's not a problem, it's part of a project
>>>>>> lifetime.
>>>>>>
>>>>>> Thoughts ?
>>>>>>
>>>>>
>>>>> I'd like to add some perspective as a primarily non-Java contributor
>>>>> (having contributed "only" a couple thousand lines to the java side over
>>>>> the last couple of years) that maven feels much more java-centric (as is
>>>>> our repo, but that's a separate issue). Also, as someone not coming from
>>>>> with years of maven (or gradle for that matter) experience before working
>>>>> on Beam, I have found gradle much more intuitive and easier to learn.
>>>>> Especially when it comes to developing changes that span multiple modules
>>>>> (which for some reason I guess I tend to do a lot of, but with so many of
>>>>> our core sdk tests being validates runner that's likely to hit people just
>>>>> starting out as well). Now of course I don't want to discount the Java
>>>>> community, indeed it's arguably the largest and most important one at this
>>>>> point, but I also think that Beam's ability to not be limited to that one
>>>>> language (and ecosystem) is one of its huge selling points and
>>>>> differentiation (see the whole portability effort, which is for both SDKs
>>>>> and Runners).
>>>>>
>>>>> Even from a Java perspective, neither is so obscure as to be a
>>>>> significant barrier, and both are customizable enough that the average new
>>>>> developer is probably going to be looking at the documentation to see what
>>>>> commands to run to build/test/validate this project. So this is probably
>>>>> something documentation can address.
>>>>>
>>>>> But if we're going to reap the benefits and minimize the downsides of
>>>>> this migration, we have to finish the job.
>>>>>
>>>>> - Robert
>>>>>
>>>>>
>>>>>
Re: [DISCUSS] Gradle for the build ?
Posted by Reuven Lax <re...@google.com>.
Unrelated to the Maven/Gradle discussion, but I do somewhat agree with
Romain that Beam could be split into separate repos, however I don't think
it's quite ready yet. AFAIK the portability interfaces are still being
modified, and until these interfaces are fixed it will be hard to split the
project. Once those interfaces are fixed and mature, I completely agree we
should then look into splitting the repos up.
Reuven
On Tue, Oct 9, 2018 at 10:35 PM Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
>
> Le mer. 10 oct. 2018 à 06:35, Kenneth Knowles <ke...@apache.org> a écrit :
>
>> Here are some things I hear a lot:
>>
>> - Beam technical decision making is too Java-centric
>> - Gradle is a lot less Java-centric than Maven
>>
>> Hmm isnt it the opposite being groovy based? Maven tends to have the same
> set of plugins than gradle so it is a kind of status-quo if you check
> technically and are factual.
>
>
>>
>> - Gradle is still a Java-centric tool, but at least it isn't so
>> slow/wasteful
>>
>>
> Not what I experienced and it seems JB had the same experience. Gradle is
> comparable to maven in terms of full build and is way slower in Idea cause
> of the lack of integration and support (by design).
>
>
>> I can understand long-time Java devs wanting to have their familiar and
>> dominant toolchain, with good tooling and integrations (ignoring any
>> objective merits of Gradle for Java dev). But for non-Java devs you are
>> meeting in the middle. Actually, you are not even in the middle - you are
>> still in Java land. For example, setup.py can do arbitrary things, so we
>> could use it to do the Java build! What do you think of trying this out?
>> (j/k).
>>
>> What I would like is to better support language-native development
>> workflows, and make sure Gradle is lightweight glue that is easy to use.
>> Make the configurations as obvious to read as we can, with as little
>> hackery as possible.
>>
>
> Agree and think it can be time, now Beam python/go support become
> something, to split in N repos. Release lifecycles are different, codebase
> has no real link except the portability layer which can get its own repo so
> probably worth a PoC:
>
> 1. beam-portability
> 2. beam-java
> 3. beam-go
> 4. beam-python
>
> Side note: beam portability would be saner if added on top of others than
> the opposite which is done today.
>
>
>>
>> ----------
>>
>> We can make Gradle a lot better (for the above, and for Java too). From
>> my work on the core of our Gradle support code, I can name a few issues
>> that I'm sure remain even after many months away from the code:
>>
>> 1. We were learning as we built it out. We don't always do things in the
>> natural way.
>> 2. We made our own abstractions to streamline converting lots of modules
>> quickly. Clarity and efficiency of the build were not the primary concerns.
>> (to be clear: it was an amazing undertaking to get this done and the
>> decision was right for the moment)
>> 2a. We tried to centralize a lot of policy, leading to the centralized
>> bits containing the union of all complexity of all modules (or maybe it is
>> multiplicative).
>> 3. We tried very hard to match the mvn build exactly, rather than doing
>> the best thing in Gradle.
>> 4. We've built a lot of imperative code for "telling Gradle what to do"
>> and that adds a lot of complexity compared with using Groovy as primarily a
>> configuration language.
>> 5. We have turned on things like "always rebuild" when we don't know the
>> dependencies, rather than putting in the work to get the dependencies right.
>>
>> A lot of the above also makes it hard for IDEs to grok the config, since
>> we deviate from the "golden path" a lot.
>>
>> It would be awesome if module owners took on the task of making their
>> modules have an awesome incremental Gradle build.
>>
>
> Well incremental build only matters when you run a full subpart of a job
> and is something very fragile - check how in months it didnt happen. In
> practise what's a dev workflow:
>
> 1. loop { dev in the IDE, run test }
> 2. run the full module or build to validate nothing has been broken
> 3. PR (+ back to 1 if comments)
>
> This means that incremental support is only relevant for jenkins where the
> perf diff is not significative and where you can't use incremental build
> cause you want to have a fully reliable build.
> So at the end the incremental build support is not that significative for
> end users and contibutors. The cost of not having the IDE support, however,
> is just a blocker.
>
> So my 2cts would be to stop trying to be good theorically at the cost of
> loosing the users and try to embrace a community driven choices approach.
>
>
>>
>> Kenn
>>
>> On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> For me the vendoring issue is ok cause it should belong to another shade
>>> loduke released with beam when needed. It is not an uncommon practise.
>>>
>>> Now the lack of IDE integration for tests/debug (using gradle runner is
>>> a workaround and still hurts by its slowness compared to native run) is a
>>> clear showstopper for me.
>>>
>>> Also, from a community perspective, gradle adoption is far to be
>>> mainstream (even spark is built with maven) so does not serve beam at the
>>> end.
>>>
>>> Maven build didnt have any issue except the duration AFAIK, gradle has 2
>>> blockers + several small drawbacks (custom build and no standard, no
>>> tooling without script execution, bad integration in enterprise chaines
>>> like security auditing etc). Overall gradle build is close to maven one -
>>> last time i tested it was within 15% so not worth it when you see the time
>>> you loose when developping anything. It is key to keep in mind jenkiks is
>>> cheaper than human time.
>>>
>>> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a
>>> écrit :
>>>
>>>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I know that's a hot topic, but I have to bring this discussion on the
>>>>> table.
>>>>>
>>>>
>>>> Thank you for bringing this up and revisiting it now that we have some
>>>> experience.
>>>>
>>>>
>>>>> Some months ago, we discussed about migrating our build from Maven to
>>>>> Gradle. One of the key expected improvement was the time to build.
>>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>>> this PoC was actually directly a migrate on master.
>>>>>
>>>>> Now, I would like to bring facts here:
>>>>>
>>>>> 1. Build time
>>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>>> regarding what the build is doing, I don't see huge improvement
>>>>> provided
>>>>> by Gradle.
>>>>>
>>>>
>>>> I rarely, if ever, build from scratch so perhaps I have not been
>>>> impacted by this nearly as much. (In particular, build and test times seem
>>>> to have gone way down for me, probably due to better incremental support,
>>>> but that's just anecdotal.)
>>>>
>>>> Is this worse than it was on maven, or just not as much better as was
>>>> hoped?
>>>>
>>>>
>>>>> 2. Build reliability
>>>>> Even worse, most of the time, we need to use --no-parallel and
>>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>>> release). It has an impact on build time, and we loose part of Gradle
>>>>> benefits.
>>>>>
>>>>
>>>> I think this is a matter of incorrect dependency declarations (and is
>>>> not unique to gradle). I'd have loved to been able to go with a build
>>>> system that simply didn't let you have incorrect dependency declarations,
>>>> but that wasn't an option for other reasons.
>>>>
>>>> I wonder if there's some automatic tooling we could leverage to fix
>>>> (and keep fixed) this. Regardless, this is unfinished work that remains to
>>>> be done so we can realize the full benefits.
>>>>
>>>>
>>>>> 3. Release and repositories
>>>>> Even if couple of releases has been performed with Gradle, it's not
>>>>> obvious to see improvements around artifacts handling. I got my
>>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>>> speed up the build dealing around the repository).
>>>>>
>>>>
>>>> Could you clarify what improvements we were expecting here? I thought
>>>> the goal was that we could publish the same artifacts, with no regression.
>>>>
>>>>
>>>>> 4. IDE integration
>>>>> We already had some comments on the mailing lists about the IDE
>>>>> integration. Clearly, the situation is not good on that front too. The
>>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>>>
>>>>
>>>> This is important. To be honest, I had also issues back in the day
>>>> getting the maven setup working well out of the box in IntelliJ and Eclipse
>>>> (mostly with respect to things like shadowing and protobufs), so we
>>>> shouldn't fall prey to the golden age fallacy.
>>>>
>>>> It seems the recent move to vendoring has caused more issues here; I'm
>>>> not sure that would be fixed just moving back to maven (or how to resolve
>>>> it going forward).
>>>>
>>>> On the other hand, just last week I set up a new computer according to
>>>> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and
>>>> that seems to be working fine.
>>>>
>>>>
>>>>> We are working hard to grow up the community, and from a contributor
>>>>> perspective, our build system is not good today IMHO.
>>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>>> much time of the build, largely more than working on the PRs code
>>>>> itself.
>>>>>
>>>>> So, obviously, the situation is not perfect, at least from a
>>>>> contributor
>>>>> perspective.
>>>>>
>>>>> The purpose of this thread is not again to have a bunch of replied
>>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>>> concrete. So basically, we only have two options:
>>>>>
>>>>> 1. Improve the build, working hard on Gradle front. Not sure if it
>>>>> makes
>>>>> such sense from a contributor perspective, as Maven is really well
>>>>> known
>>>>> from most of contributors (and easier to start with IMHO).
>>>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>>> is better, Maven is well known from the contributors as already said.
>>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>>> expected results now, that's not a problem, it's part of a project
>>>>> lifetime.
>>>>>
>>>>> Thoughts ?
>>>>>
>>>>
>>>> I'd like to add some perspective as a primarily non-Java contributor
>>>> (having contributed "only" a couple thousand lines to the java side over
>>>> the last couple of years) that maven feels much more java-centric (as is
>>>> our repo, but that's a separate issue). Also, as someone not coming from
>>>> with years of maven (or gradle for that matter) experience before working
>>>> on Beam, I have found gradle much more intuitive and easier to learn.
>>>> Especially when it comes to developing changes that span multiple modules
>>>> (which for some reason I guess I tend to do a lot of, but with so many of
>>>> our core sdk tests being validates runner that's likely to hit people just
>>>> starting out as well). Now of course I don't want to discount the Java
>>>> community, indeed it's arguably the largest and most important one at this
>>>> point, but I also think that Beam's ability to not be limited to that one
>>>> language (and ecosystem) is one of its huge selling points and
>>>> differentiation (see the whole portability effort, which is for both SDKs
>>>> and Runners).
>>>>
>>>> Even from a Java perspective, neither is so obscure as to be a
>>>> significant barrier, and both are customizable enough that the average new
>>>> developer is probably going to be looking at the documentation to see what
>>>> commands to run to build/test/validate this project. So this is probably
>>>> something documentation can address.
>>>>
>>>> But if we're going to reap the benefits and minimize the downsides of
>>>> this migration, we have to finish the job.
>>>>
>>>> - Robert
>>>>
>>>>
>>>>
Re: [DISCUSS] Gradle for the build ?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le mer. 10 oct. 2018 à 06:35, Kenneth Knowles <ke...@apache.org> a écrit :
> Here are some things I hear a lot:
>
> - Beam technical decision making is too Java-centric
> - Gradle is a lot less Java-centric than Maven
>
> Hmm isnt it the opposite being groovy based? Maven tends to have the same
set of plugins than gradle so it is a kind of status-quo if you check
technically and are factual.
>
> - Gradle is still a Java-centric tool, but at least it isn't so
> slow/wasteful
>
>
Not what I experienced and it seems JB had the same experience. Gradle is
comparable to maven in terms of full build and is way slower in Idea cause
of the lack of integration and support (by design).
> I can understand long-time Java devs wanting to have their familiar and
> dominant toolchain, with good tooling and integrations (ignoring any
> objective merits of Gradle for Java dev). But for non-Java devs you are
> meeting in the middle. Actually, you are not even in the middle - you are
> still in Java land. For example, setup.py can do arbitrary things, so we
> could use it to do the Java build! What do you think of trying this out?
> (j/k).
>
> What I would like is to better support language-native development
> workflows, and make sure Gradle is lightweight glue that is easy to use.
> Make the configurations as obvious to read as we can, with as little
> hackery as possible.
>
Agree and think it can be time, now Beam python/go support become
something, to split in N repos. Release lifecycles are different, codebase
has no real link except the portability layer which can get its own repo so
probably worth a PoC:
1. beam-portability
2. beam-java
3. beam-go
4. beam-python
Side note: beam portability would be saner if added on top of others than
the opposite which is done today.
>
> ----------
>
> We can make Gradle a lot better (for the above, and for Java too). From my
> work on the core of our Gradle support code, I can name a few issues that
> I'm sure remain even after many months away from the code:
>
> 1. We were learning as we built it out. We don't always do things in the
> natural way.
> 2. We made our own abstractions to streamline converting lots of modules
> quickly. Clarity and efficiency of the build were not the primary concerns.
> (to be clear: it was an amazing undertaking to get this done and the
> decision was right for the moment)
> 2a. We tried to centralize a lot of policy, leading to the centralized
> bits containing the union of all complexity of all modules (or maybe it is
> multiplicative).
> 3. We tried very hard to match the mvn build exactly, rather than doing
> the best thing in Gradle.
> 4. We've built a lot of imperative code for "telling Gradle what to do"
> and that adds a lot of complexity compared with using Groovy as primarily a
> configuration language.
> 5. We have turned on things like "always rebuild" when we don't know the
> dependencies, rather than putting in the work to get the dependencies right.
>
> A lot of the above also makes it hard for IDEs to grok the config, since
> we deviate from the "golden path" a lot.
>
> It would be awesome if module owners took on the task of making their
> modules have an awesome incremental Gradle build.
>
Well incremental build only matters when you run a full subpart of a job
and is something very fragile - check how in months it didnt happen. In
practise what's a dev workflow:
1. loop { dev in the IDE, run test }
2. run the full module or build to validate nothing has been broken
3. PR (+ back to 1 if comments)
This means that incremental support is only relevant for jenkins where the
perf diff is not significative and where you can't use incremental build
cause you want to have a fully reliable build.
So at the end the incremental build support is not that significative for
end users and contibutors. The cost of not having the IDE support, however,
is just a blocker.
So my 2cts would be to stop trying to be good theorically at the cost of
loosing the users and try to embrace a community driven choices approach.
>
> Kenn
>
> On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> For me the vendoring issue is ok cause it should belong to another shade
>> loduke released with beam when needed. It is not an uncommon practise.
>>
>> Now the lack of IDE integration for tests/debug (using gradle runner is a
>> workaround and still hurts by its slowness compared to native run) is a
>> clear showstopper for me.
>>
>> Also, from a community perspective, gradle adoption is far to be
>> mainstream (even spark is built with maven) so does not serve beam at the
>> end.
>>
>> Maven build didnt have any issue except the duration AFAIK, gradle has 2
>> blockers + several small drawbacks (custom build and no standard, no
>> tooling without script execution, bad integration in enterprise chaines
>> like security auditing etc). Overall gradle build is close to maven one -
>> last time i tested it was within 15% so not worth it when you see the time
>> you loose when developping anything. It is key to keep in mind jenkiks is
>> cheaper than human time.
>>
>> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a
>> écrit :
>>
>>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I know that's a hot topic, but I have to bring this discussion on the
>>>> table.
>>>>
>>>
>>> Thank you for bringing this up and revisiting it now that we have some
>>> experience.
>>>
>>>
>>>> Some months ago, we discussed about migrating our build from Maven to
>>>> Gradle. One of the key expected improvement was the time to build.
>>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>>> this PoC was actually directly a migrate on master.
>>>>
>>>> Now, I would like to bring facts here:
>>>>
>>>> 1. Build time
>>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>>> regarding what the build is doing, I don't see huge improvement provided
>>>> by Gradle.
>>>>
>>>
>>> I rarely, if ever, build from scratch so perhaps I have not been
>>> impacted by this nearly as much. (In particular, build and test times seem
>>> to have gone way down for me, probably due to better incremental support,
>>> but that's just anecdotal.)
>>>
>>> Is this worse than it was on maven, or just not as much better as was
>>> hoped?
>>>
>>>
>>>> 2. Build reliability
>>>> Even worse, most of the time, we need to use --no-parallel and
>>>> --no-daemon to have a reliable build (it's basically recommended for
>>>> release). It has an impact on build time, and we loose part of Gradle
>>>> benefits.
>>>>
>>>
>>> I think this is a matter of incorrect dependency declarations (and is
>>> not unique to gradle). I'd have loved to been able to go with a build
>>> system that simply didn't let you have incorrect dependency declarations,
>>> but that wasn't an option for other reasons.
>>>
>>> I wonder if there's some automatic tooling we could leverage to fix (and
>>> keep fixed) this. Regardless, this is unfinished work that remains to be
>>> done so we can realize the full benefits.
>>>
>>>
>>>> 3. Release and repositories
>>>> Even if couple of releases has been performed with Gradle, it's not
>>>> obvious to see improvements around artifacts handling. I got my
>>>> repository polluted twice (that's part of the trick Gradle is doing to
>>>> speed up the build dealing around the repository).
>>>>
>>>
>>> Could you clarify what improvements we were expecting here? I thought
>>> the goal was that we could publish the same artifacts, with no regression.
>>>
>>>
>>>> 4. IDE integration
>>>> We already had some comments on the mailing lists about the IDE
>>>> integration. Clearly, the situation is not good on that front too. The
>>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>>
>>>
>>> This is important. To be honest, I had also issues back in the day
>>> getting the maven setup working well out of the box in IntelliJ and Eclipse
>>> (mostly with respect to things like shadowing and protobufs), so we
>>> shouldn't fall prey to the golden age fallacy.
>>>
>>> It seems the recent move to vendoring has caused more issues here; I'm
>>> not sure that would be fixed just moving back to maven (or how to resolve
>>> it going forward).
>>>
>>> On the other hand, just last week I set up a new computer according to
>>> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and that
>>> seems to be working fine.
>>>
>>>
>>>> We are working hard to grow up the community, and from a contributor
>>>> perspective, our build system is not good today IMHO.
>>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>>> much time of the build, largely more than working on the PRs code
>>>> itself.
>>>>
>>>> So, obviously, the situation is not perfect, at least from a contributor
>>>> perspective.
>>>>
>>>> The purpose of this thread is not again to have a bunch of replied
>>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>>> concrete. So basically, we only have two options:
>>>>
>>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>>> such sense from a contributor perspective, as Maven is really well known
>>>> from most of contributors (and easier to start with IMHO).
>>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>>> is better, Maven is well known from the contributors as already said.
>>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>>> expected results now, that's not a problem, it's part of a project
>>>> lifetime.
>>>>
>>>> Thoughts ?
>>>>
>>>
>>> I'd like to add some perspective as a primarily non-Java contributor
>>> (having contributed "only" a couple thousand lines to the java side over
>>> the last couple of years) that maven feels much more java-centric (as is
>>> our repo, but that's a separate issue). Also, as someone not coming from
>>> with years of maven (or gradle for that matter) experience before working
>>> on Beam, I have found gradle much more intuitive and easier to learn.
>>> Especially when it comes to developing changes that span multiple modules
>>> (which for some reason I guess I tend to do a lot of, but with so many of
>>> our core sdk tests being validates runner that's likely to hit people just
>>> starting out as well). Now of course I don't want to discount the Java
>>> community, indeed it's arguably the largest and most important one at this
>>> point, but I also think that Beam's ability to not be limited to that one
>>> language (and ecosystem) is one of its huge selling points and
>>> differentiation (see the whole portability effort, which is for both SDKs
>>> and Runners).
>>>
>>> Even from a Java perspective, neither is so obscure as to be a
>>> significant barrier, and both are customizable enough that the average new
>>> developer is probably going to be looking at the documentation to see what
>>> commands to run to build/test/validate this project. So this is probably
>>> something documentation can address.
>>>
>>> But if we're going to reap the benefits and minimize the downsides of
>>> this migration, we have to finish the job.
>>>
>>> - Robert
>>>
>>>
>>>
Re: [DISCUSS] Gradle for the build ?
Posted by Kenneth Knowles <ke...@apache.org>.
Here are some things I hear a lot:
- Beam technical decision making is too Java-centric
- Gradle is a lot less Java-centric than Maven
- Gradle is still a Java-centric tool, but at least it isn't so
slow/wasteful
I can understand long-time Java devs wanting to have their familiar and
dominant toolchain, with good tooling and integrations (ignoring any
objective merits of Gradle for Java dev). But for non-Java devs you are
meeting in the middle. Actually, you are not even in the middle - you are
still in Java land. For example, setup.py can do arbitrary things, so we
could use it to do the Java build! What do you think of trying this out?
(j/k).
What I would like is to better support language-native development
workflows, and make sure Gradle is lightweight glue that is easy to use.
Make the configurations as obvious to read as we can, with as little
hackery as possible.
----------
We can make Gradle a lot better (for the above, and for Java too). From my
work on the core of our Gradle support code, I can name a few issues that
I'm sure remain even after many months away from the code:
1. We were learning as we built it out. We don't always do things in the
natural way.
2. We made our own abstractions to streamline converting lots of modules
quickly. Clarity and efficiency of the build were not the primary concerns.
(to be clear: it was an amazing undertaking to get this done and the
decision was right for the moment)
2a. We tried to centralize a lot of policy, leading to the centralized bits
containing the union of all complexity of all modules (or maybe it is
multiplicative).
3. We tried very hard to match the mvn build exactly, rather than doing the
best thing in Gradle.
4. We've built a lot of imperative code for "telling Gradle what to do" and
that adds a lot of complexity compared with using Groovy as primarily a
configuration language.
5. We have turned on things like "always rebuild" when we don't know the
dependencies, rather than putting in the work to get the dependencies right.
A lot of the above also makes it hard for IDEs to grok the config, since we
deviate from the "golden path" a lot.
It would be awesome if module owners took on the task of making their
modules have an awesome incremental Gradle build.
Kenn
On Tue, Oct 9, 2018 at 3:38 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> For me the vendoring issue is ok cause it should belong to another shade
> loduke released with beam when needed. It is not an uncommon practise.
>
> Now the lack of IDE integration for tests/debug (using gradle runner is a
> workaround and still hurts by its slowness compared to native run) is a
> clear showstopper for me.
>
> Also, from a community perspective, gradle adoption is far to be
> mainstream (even spark is built with maven) so does not serve beam at the
> end.
>
> Maven build didnt have any issue except the duration AFAIK, gradle has 2
> blockers + several small drawbacks (custom build and no standard, no
> tooling without script execution, bad integration in enterprise chaines
> like security auditing etc). Overall gradle build is close to maven one -
> last time i tested it was within 15% so not worth it when you see the time
> you loose when developping anything. It is key to keep in mind jenkiks is
> cheaper than human time.
>
> Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a écrit :
>
>> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi guys,
>>>
>>> I know that's a hot topic, but I have to bring this discussion on the
>>> table.
>>>
>>
>> Thank you for bringing this up and revisiting it now that we have some
>> experience.
>>
>>
>>> Some months ago, we discussed about migrating our build from Maven to
>>> Gradle. One of the key expected improvement was the time to build.
>>> We proposed to do a PoC to evaluate the impacts and improvements, but
>>> this PoC was actually directly a migrate on master.
>>>
>>> Now, I would like to bring facts here:
>>>
>>> 1. Build time
>>> On my machine, the build time is roughly 1h15. It's pretty long, and
>>> regarding what the build is doing, I don't see huge improvement provided
>>> by Gradle.
>>>
>>
>> I rarely, if ever, build from scratch so perhaps I have not been impacted
>> by this nearly as much. (In particular, build and test times seem to have
>> gone way down for me, probably due to better incremental support, but
>> that's just anecdotal.)
>>
>> Is this worse than it was on maven, or just not as much better as was
>> hoped?
>>
>>
>>> 2. Build reliability
>>> Even worse, most of the time, we need to use --no-parallel and
>>> --no-daemon to have a reliable build (it's basically recommended for
>>> release). It has an impact on build time, and we loose part of Gradle
>>> benefits.
>>>
>>
>> I think this is a matter of incorrect dependency declarations (and is not
>> unique to gradle). I'd have loved to been able to go with a build system
>> that simply didn't let you have incorrect dependency declarations, but that
>> wasn't an option for other reasons.
>>
>> I wonder if there's some automatic tooling we could leverage to fix (and
>> keep fixed) this. Regardless, this is unfinished work that remains to be
>> done so we can realize the full benefits.
>>
>>
>>> 3. Release and repositories
>>> Even if couple of releases has been performed with Gradle, it's not
>>> obvious to see improvements around artifacts handling. I got my
>>> repository polluted twice (that's part of the trick Gradle is doing to
>>> speed up the build dealing around the repository).
>>>
>>
>> Could you clarify what improvements we were expecting here? I thought the
>> goal was that we could publish the same artifacts, with no regression.
>>
>>
>>> 4. IDE integration
>>> We already had some comments on the mailing lists about the IDE
>>> integration. Clearly, the situation is not good on that front too. The
>>> integration on IDE (especially IntelliJ) is not good enough right now.
>>>
>>
>> This is important. To be honest, I had also issues back in the day
>> getting the maven setup working well out of the box in IntelliJ and Eclipse
>> (mostly with respect to things like shadowing and protobufs), so we
>> shouldn't fall prey to the golden age fallacy.
>>
>> It seems the recent move to vendoring has caused more issues here; I'm
>> not sure that would be fixed just moving back to maven (or how to resolve
>> it going forward).
>>
>> On the other hand, just last week I set up a new computer according to
>> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and that
>> seems to be working fine.
>>
>>
>>> We are working hard to grow up the community, and from a contributor
>>> perspective, our build system is not good today IMHO.
>>> As a contributor, I resumed my work on some PRs, and I'm spending so
>>> much time of the build, largely more than working on the PRs code itself.
>>>
>>> So, obviously, the situation is not perfect, at least from a contributor
>>> perspective.
>>>
>>> The purpose of this thread is not again to have a bunch of replied
>>> ending nowhere. I would like to be more "pushy" and let's try to be
>>> concrete. So basically, we only have two options:
>>>
>>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>>> such sense from a contributor perspective, as Maven is really well known
>>> from most of contributors (and easier to start with IMHO).
>>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>>> is better, Maven is well known from the contributors as already said.
>>> The effort is not so huge. We tried to use Gradle, we don't have the
>>> expected results now, that's not a problem, it's part of a project
>>> lifetime.
>>>
>>> Thoughts ?
>>>
>>
>> I'd like to add some perspective as a primarily non-Java contributor
>> (having contributed "only" a couple thousand lines to the java side over
>> the last couple of years) that maven feels much more java-centric (as is
>> our repo, but that's a separate issue). Also, as someone not coming from
>> with years of maven (or gradle for that matter) experience before working
>> on Beam, I have found gradle much more intuitive and easier to learn.
>> Especially when it comes to developing changes that span multiple modules
>> (which for some reason I guess I tend to do a lot of, but with so many of
>> our core sdk tests being validates runner that's likely to hit people just
>> starting out as well). Now of course I don't want to discount the Java
>> community, indeed it's arguably the largest and most important one at this
>> point, but I also think that Beam's ability to not be limited to that one
>> language (and ecosystem) is one of its huge selling points and
>> differentiation (see the whole portability effort, which is for both SDKs
>> and Runners).
>>
>> Even from a Java perspective, neither is so obscure as to be a
>> significant barrier, and both are customizable enough that the average new
>> developer is probably going to be looking at the documentation to see what
>> commands to run to build/test/validate this project. So this is probably
>> something documentation can address.
>>
>> But if we're going to reap the benefits and minimize the downsides of
>> this migration, we have to finish the job.
>>
>> - Robert
>>
>>
>>
Re: [DISCUSS] Gradle for the build ?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
For me the vendoring issue is ok cause it should belong to another shade
loduke released with beam when needed. It is not an uncommon practise.
Now the lack of IDE integration for tests/debug (using gradle runner is a
workaround and still hurts by its slowness compared to native run) is a
clear showstopper for me.
Also, from a community perspective, gradle adoption is far to be mainstream
(even spark is built with maven) so does not serve beam at the end.
Maven build didnt have any issue except the duration AFAIK, gradle has 2
blockers + several small drawbacks (custom build and no standard, no
tooling without script execution, bad integration in enterprise chaines
like security auditing etc). Overall gradle build is close to maven one -
last time i tested it was within 15% so not worth it when you see the time
you loose when developping anything. It is key to keep in mind jenkiks is
cheaper than human time.
Le mar. 9 oct. 2018 13:22, Robert Bradshaw <ro...@google.com> a écrit :
> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi guys,
>>
>> I know that's a hot topic, but I have to bring this discussion on the
>> table.
>>
>
> Thank you for bringing this up and revisiting it now that we have some
> experience.
>
>
>> Some months ago, we discussed about migrating our build from Maven to
>> Gradle. One of the key expected improvement was the time to build.
>> We proposed to do a PoC to evaluate the impacts and improvements, but
>> this PoC was actually directly a migrate on master.
>>
>> Now, I would like to bring facts here:
>>
>> 1. Build time
>> On my machine, the build time is roughly 1h15. It's pretty long, and
>> regarding what the build is doing, I don't see huge improvement provided
>> by Gradle.
>>
>
> I rarely, if ever, build from scratch so perhaps I have not been impacted
> by this nearly as much. (In particular, build and test times seem to have
> gone way down for me, probably due to better incremental support, but
> that's just anecdotal.)
>
> Is this worse than it was on maven, or just not as much better as was
> hoped?
>
>
>> 2. Build reliability
>> Even worse, most of the time, we need to use --no-parallel and
>> --no-daemon to have a reliable build (it's basically recommended for
>> release). It has an impact on build time, and we loose part of Gradle
>> benefits.
>>
>
> I think this is a matter of incorrect dependency declarations (and is not
> unique to gradle). I'd have loved to been able to go with a build system
> that simply didn't let you have incorrect dependency declarations, but that
> wasn't an option for other reasons.
>
> I wonder if there's some automatic tooling we could leverage to fix (and
> keep fixed) this. Regardless, this is unfinished work that remains to be
> done so we can realize the full benefits.
>
>
>> 3. Release and repositories
>> Even if couple of releases has been performed with Gradle, it's not
>> obvious to see improvements around artifacts handling. I got my
>> repository polluted twice (that's part of the trick Gradle is doing to
>> speed up the build dealing around the repository).
>>
>
> Could you clarify what improvements we were expecting here? I thought the
> goal was that we could publish the same artifacts, with no regression.
>
>
>> 4. IDE integration
>> We already had some comments on the mailing lists about the IDE
>> integration. Clearly, the situation is not good on that front too. The
>> integration on IDE (especially IntelliJ) is not good enough right now.
>>
>
> This is important. To be honest, I had also issues back in the day getting
> the maven setup working well out of the box in IntelliJ and Eclipse (mostly
> with respect to things like shadowing and protobufs), so we shouldn't fall
> prey to the golden age fallacy.
>
> It seems the recent move to vendoring has caused more issues here; I'm not
> sure that would be fixed just moving back to maven (or how to resolve it
> going forward).
>
> On the other hand, just last week I set up a new computer according to
> https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and that
> seems to be working fine.
>
>
>> We are working hard to grow up the community, and from a contributor
>> perspective, our build system is not good today IMHO.
>> As a contributor, I resumed my work on some PRs, and I'm spending so
>> much time of the build, largely more than working on the PRs code itself.
>>
>> So, obviously, the situation is not perfect, at least from a contributor
>> perspective.
>>
>> The purpose of this thread is not again to have a bunch of replied
>> ending nowhere. I would like to be more "pushy" and let's try to be
>> concrete. So basically, we only have two options:
>>
>> 1. Improve the build, working hard on Gradle front. Not sure if it makes
>> such sense from a contributor perspective, as Maven is really well known
>> from most of contributors (and easier to start with IMHO).
>> 2. Back on Maven. That's clearly my preferred approach. IDE integration
>> is better, Maven is well known from the contributors as already said.
>> The effort is not so huge. We tried to use Gradle, we don't have the
>> expected results now, that's not a problem, it's part of a project
>> lifetime.
>>
>> Thoughts ?
>>
>
> I'd like to add some perspective as a primarily non-Java contributor
> (having contributed "only" a couple thousand lines to the java side over
> the last couple of years) that maven feels much more java-centric (as is
> our repo, but that's a separate issue). Also, as someone not coming from
> with years of maven (or gradle for that matter) experience before working
> on Beam, I have found gradle much more intuitive and easier to learn.
> Especially when it comes to developing changes that span multiple modules
> (which for some reason I guess I tend to do a lot of, but with so many of
> our core sdk tests being validates runner that's likely to hit people just
> starting out as well). Now of course I don't want to discount the Java
> community, indeed it's arguably the largest and most important one at this
> point, but I also think that Beam's ability to not be limited to that one
> language (and ecosystem) is one of its huge selling points and
> differentiation (see the whole portability effort, which is for both SDKs
> and Runners).
>
> Even from a Java perspective, neither is so obscure as to be a significant
> barrier, and both are customizable enough that the average new developer is
> probably going to be looking at the documentation to see what commands to
> run to build/test/validate this project. So this is probably something
> documentation can address.
>
> But if we're going to reap the benefits and minimize the downsides of this
> migration, we have to finish the job.
>
> - Robert
>
>
>
Re: [DISCUSS] Gradle for the build ?
Posted by Robert Bradshaw <ro...@google.com>.
On Wed, Oct 10, 2018 at 8:03 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Robert,
>
> about your point about we never fully build the project, even if I
> agree, it's what we "sold" with Gradle.
> Because, with Maven you can also build a single module without problem.
>
Good incremental support for the edit/build/test cycle is critical, which
(for me at least) seems to have improved a lot.
One particularly painful scenario was "I want to test changes in module B
that depends on changes in module A."
So, if I don't get the argument about build time for single module: in
> that case, build speed was not a valid argument for Maven to Gradle topic.
>
> Again, I would like to emphasize about the contribution adoption.
>
> If you really think that documenting & improving the build system using
> Gradle, that's fine.
> About the effort to go back on Maven, I really think that it's do-able.
>
> Probably this thread will end nowhere and we will stay with Gradle,
> that's fair. I just hope that we won't have any "brake" on contribution.
>
> Again, don't get my wrong, I just wanted to bring the discussion on the
> table, no pressure, no harsh, just a fairly discussion and concern that
> I wanted to share ;)
>
I'm really glad you brought this up. Hopefully this will be a call to
action for those that sold Gradle to (finish) delivering on what was
promised. Or even if we went back to Maven, ended up in a state better than
what we could have had with Gradle (if we determined it wasn't fixable or
going to be fixed). It seems like most IDEs have support for gradle in
general, so I'd be inclined to agree with Kenn that it's an issue with how
we're using it rather than the tool itself. I'm less convinced it's up to
each module owner to clean this up locally (unless there's some good
pattern to follow). Recognizing we have a problem is the first step.
Re: [DISCUSS] Gradle for the build ?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Robert,
about your point about we never fully build the project, even if I
agree, it's what we "sold" with Gradle.
Because, with Maven you can also build a single module without problem.
So, if I don't get the argument about build time for single module: in
that case, build speed was not a valid argument for Maven to Gradle topic.
Again, I would like to emphasize about the contribution adoption.
If you really think that documenting & improving the build system using
Gradle, that's fine.
About the effort to go back on Maven, I really think that it's do-able.
Probably this thread will end nowhere and we will stay with Gradle,
that's fair. I just hope that we won't have any "brake" on contribution.
Again, don't get my wrong, I just wanted to bring the discussion on the
table, no pressure, no harsh, just a fairly discussion and concern that
I wanted to share ;)
Regards
JB
On 09/10/2018 12:22, Robert Bradshaw wrote:
> On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion on
> the table.
>
>
> Thank you for bringing this up and revisiting it now that we have some
> experience.
>
>
> Some months ago, we discussed about migrating our build from Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement provided
> by Gradle.
>
>
> I rarely, if ever, build from scratch so perhaps I have not been
> impacted by this nearly as much. (In particular, build and test times
> seem to have gone way down for me, probably due to better incremental
> support, but that's just anecdotal.)
>
> Is this worse than it was on maven, or just not as much better as was
> hoped?
>
>
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of Gradle
> benefits.
>
>
> I think this is a matter of incorrect dependency declarations (and is
> not unique to gradle). I'd have loved to been able to go with a build
> system that simply didn't let you have incorrect dependency
> declarations, but that wasn't an option for other reasons.
>
> I wonder if there's some automatic tooling we could leverage to fix (and
> keep fixed) this. Regardless, this is unfinished work that remains to be
> done so we can realize the full benefits.
>
>
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is doing to
> speed up the build dealing around the repository).
>
>
> Could you clarify what improvements we were expecting here? I thought
> the goal was that we could publish the same artifacts, with no regression.
>
>
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front too. The
> integration on IDE (especially IntelliJ) is not good enough right now.
>
>
> This is important. To be honest, I had also issues back in the day
> getting the maven setup working well out of the box in IntelliJ and
> Eclipse (mostly with respect to things like shadowing and protobufs), so
> we shouldn't fall prey to the golden age fallacy.
>
> It seems the recent move to vendoring has caused more issues here; I'm
> not sure that would be fixed just moving back to maven (or how to
> resolve it going forward).
>
> On the other hand, just last week I set up a new computer according
> to https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and
> that seems to be working fine.
>
>
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs code
> itself.
>
> So, obviously, the situation is not perfect, at least from a contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
> such sense from a contributor perspective, as Maven is really well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
> is better, Maven is well known from the contributors as already said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a project
> lifetime.
>
> Thoughts ?
>
>
> I'd like to add some perspective as a primarily non-Java contributor
> (having contributed "only" a couple thousand lines to the java side over
> the last couple of years) that maven feels much more java-centric (as is
> our repo, but that's a separate issue). Also, as someone not coming from
> with years of maven (or gradle for that matter) experience before
> working on Beam, I have found gradle much more intuitive and easier to
> learn. Especially when it comes to developing changes that span multiple
> modules (which for some reason I guess I tend to do a lot of, but with
> so many of our core sdk tests being validates runner that's likely to
> hit people just starting out as well). Now of course I don't want to
> discount the Java community, indeed it's arguably the largest and most
> important one at this point, but I also think that Beam's ability to not
> be limited to that one language (and ecosystem) is one of its huge
> selling points and differentiation (see the whole portability effort,
> which is for both SDKs and Runners).
>
> Even from a Java perspective, neither is so obscure as to be a
> significant barrier, and both are customizable enough that the average
> new developer is probably going to be looking at the documentation to
> see what commands to run to build/test/validate this project. So this is
> probably something documentation can address.
>
> But if we're going to reap the benefits and minimize the downsides of
> this migration, we have to finish the job.
>
> - Robert
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [DISCUSS] Gradle for the build ?
Posted by Robert Bradshaw <ro...@google.com>.
On Tue, Oct 9, 2018 at 10:04 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> I know that's a hot topic, but I have to bring this discussion on the
> table.
>
Thank you for bringing this up and revisiting it now that we have some
experience.
> Some months ago, we discussed about migrating our build from Maven to
> Gradle. One of the key expected improvement was the time to build.
> We proposed to do a PoC to evaluate the impacts and improvements, but
> this PoC was actually directly a migrate on master.
>
> Now, I would like to bring facts here:
>
> 1. Build time
> On my machine, the build time is roughly 1h15. It's pretty long, and
> regarding what the build is doing, I don't see huge improvement provided
> by Gradle.
>
I rarely, if ever, build from scratch so perhaps I have not been impacted
by this nearly as much. (In particular, build and test times seem to have
gone way down for me, probably due to better incremental support, but
that's just anecdotal.)
Is this worse than it was on maven, or just not as much better as was
hoped?
> 2. Build reliability
> Even worse, most of the time, we need to use --no-parallel and
> --no-daemon to have a reliable build (it's basically recommended for
> release). It has an impact on build time, and we loose part of Gradle
> benefits.
>
I think this is a matter of incorrect dependency declarations (and is not
unique to gradle). I'd have loved to been able to go with a build system
that simply didn't let you have incorrect dependency declarations, but that
wasn't an option for other reasons.
I wonder if there's some automatic tooling we could leverage to fix (and
keep fixed) this. Regardless, this is unfinished work that remains to be
done so we can realize the full benefits.
> 3. Release and repositories
> Even if couple of releases has been performed with Gradle, it's not
> obvious to see improvements around artifacts handling. I got my
> repository polluted twice (that's part of the trick Gradle is doing to
> speed up the build dealing around the repository).
>
Could you clarify what improvements we were expecting here? I thought the
goal was that we could publish the same artifacts, with no regression.
> 4. IDE integration
> We already had some comments on the mailing lists about the IDE
> integration. Clearly, the situation is not good on that front too. The
> integration on IDE (especially IntelliJ) is not good enough right now.
>
This is important. To be honest, I had also issues back in the day getting
the maven setup working well out of the box in IntelliJ and Eclipse (mostly
with respect to things like shadowing and protobufs), so we shouldn't fall
prey to the golden age fallacy.
It seems the recent move to vendoring has caused more issues here; I'm not
sure that would be fixed just moving back to maven (or how to resolve it
going forward).
On the other hand, just last week I set up a new computer according to
https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips and that
seems to be working fine.
> We are working hard to grow up the community, and from a contributor
> perspective, our build system is not good today IMHO.
> As a contributor, I resumed my work on some PRs, and I'm spending so
> much time of the build, largely more than working on the PRs code itself.
>
> So, obviously, the situation is not perfect, at least from a contributor
> perspective.
>
> The purpose of this thread is not again to have a bunch of replied
> ending nowhere. I would like to be more "pushy" and let's try to be
> concrete. So basically, we only have two options:
>
> 1. Improve the build, working hard on Gradle front. Not sure if it makes
> such sense from a contributor perspective, as Maven is really well known
> from most of contributors (and easier to start with IMHO).
> 2. Back on Maven. That's clearly my preferred approach. IDE integration
> is better, Maven is well known from the contributors as already said.
> The effort is not so huge. We tried to use Gradle, we don't have the
> expected results now, that's not a problem, it's part of a project
> lifetime.
>
> Thoughts ?
>
I'd like to add some perspective as a primarily non-Java contributor
(having contributed "only" a couple thousand lines to the java side over
the last couple of years) that maven feels much more java-centric (as is
our repo, but that's a separate issue). Also, as someone not coming from
with years of maven (or gradle for that matter) experience before working
on Beam, I have found gradle much more intuitive and easier to learn.
Especially when it comes to developing changes that span multiple modules
(which for some reason I guess I tend to do a lot of, but with so many of
our core sdk tests being validates runner that's likely to hit people just
starting out as well). Now of course I don't want to discount the Java
community, indeed it's arguably the largest and most important one at this
point, but I also think that Beam's ability to not be limited to that one
language (and ecosystem) is one of its huge selling points and
differentiation (see the whole portability effort, which is for both SDKs
and Runners).
Even from a Java perspective, neither is so obscure as to be a significant
barrier, and both are customizable enough that the average new developer is
probably going to be looking at the documentation to see what commands to
run to build/test/validate this project. So this is probably something
documentation can address.
But if we're going to reap the benefits and minimize the downsides of this
migration, we have to finish the job.
- Robert