You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by Nick Burch <ni...@apache.org> on 2022/10/05 19:05:17 UTC
Possibly speeding up tests with Gradle - anyone interested?
Hi All
At ApacheCon this week, a Bob and myself ended up chatting with the folks
from Gradle, who are keen to help ASF projects, and are discussing with
the Infra team.
The easier bit - they think they might be able to help speed up our maven
build, especially the running of tests. Anyone have some time to give that
a try? Will pass details along to anyone with the volunteer cycles
The interesting bit - we told them about the regression corpus, and they
got very excited as it sounds completely different to most of their normal
"my build is slow" type problems. The size of it, and the fact that it
isn't a simple pass/fail, seemed to catch their interest. Anyone (though
probably only Tim...) intersted in talking them through how it works, and
maybe getting one of their team access to the VM?
Cheers
Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Tim Allison <ta...@apache.org>.
I dropped the ball in following up with the gradle contact that Nick
met at ApacheCon. Konstantin, if you'd like to coordinate a meeting
with him (and any other devs!), please let me know. I just sent a
follow up message to him for potentially trying to set something up
next week.
Best,
Tim
On Wed, Nov 9, 2022 at 4:08 PM Konstantin Gribov <gr...@gmail.com> wrote:
>
> Any follow up on using build cache and/or experiment with migration to
> gradle? I have a little bit of expertise with gradle and wanted to join in
> with the experiment.
> Compilation avoidance, dependency resolution engine and ease of writing
> ad-hoc plugins are very strong points with gradle. I migrated a bunch of
> projects at work to gradle and improved things quite a lot.
>
> Configuration phase could be quite long, I'm impatiently waiting when
> configuration cache lands. It also avoids dependency resolution step on
> each gradle run.
>
> Nicholas, could you elaborate on what you mean with "New versions of gradle
> result in full redownload all the deps"? Never seen such behavior and it
> sounds like a bug or some elaborate repositories' misconfiguration. I only
> got extremely slow resolution with the spring dependency plugin and
> misconfigured reverse proxy before Nexus.
>
> As for build cache it brings some good on the ci but not much on local
> development in cases where we tried to use it. IIRC Build Cache server
> itself is free if we want to use it and burden Infra with running it. But
> likely it doesn't support clustering/HA which could be an issue for the ASF
> scale.
>
> We certainly could use build scans for free publishing them from CI builds,
> it would help with analyzing dependency graph and build performance.
>
> --
> Best regards,
> Konstantin Gribov
>
> пт, 14 окт. 2022 г., 17:13 Nicholas DiPiazza <ni...@gmail.com>:
>
> > Is there a branch started on the Gradle thing? I have some cycles and can
> > use them to upgrade to Gradle on Tika, if desired.
> >
> > On Thu, Oct 6, 2022 at 8:50 AM Nick Burch <ni...@apache.org> wrote:
> >
> > > On Thu, 6 Oct 2022, Tim Allison wrote:
> > > > Happy to chat. Please put them in touch.
> > >
> > > Excellent, thanks Tim!
> > >
> > > Other than your past talks, have we got any info (eg on the wiki?) about
> > > how to run the regression corpus?
> > >
> > > > I've been really impressed with what the POI team has done migrating
> > > > from ant to gradle. On Tika, I don't think we have any special needs
> > > > that would require deep gradle knowledge, but given the number of
> > > > modules now, it will be non-trivial. Also, I take Nick D's concerns
> > > > seriously.
> > >
> > > We don't have to swap from Maven - they have a plugin that integrates it
> > >
> > > Nick
> > >
> >
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Konstantin Gribov <gr...@gmail.com>.
Any follow up on using build cache and/or experiment with migration to
gradle? I have a little bit of expertise with gradle and wanted to join in
with the experiment.
Compilation avoidance, dependency resolution engine and ease of writing
ad-hoc plugins are very strong points with gradle. I migrated a bunch of
projects at work to gradle and improved things quite a lot.
Configuration phase could be quite long, I'm impatiently waiting when
configuration cache lands. It also avoids dependency resolution step on
each gradle run.
Nicholas, could you elaborate on what you mean with "New versions of gradle
result in full redownload all the deps"? Never seen such behavior and it
sounds like a bug or some elaborate repositories' misconfiguration. I only
got extremely slow resolution with the spring dependency plugin and
misconfigured reverse proxy before Nexus.
As for build cache it brings some good on the ci but not much on local
development in cases where we tried to use it. IIRC Build Cache server
itself is free if we want to use it and burden Infra with running it. But
likely it doesn't support clustering/HA which could be an issue for the ASF
scale.
We certainly could use build scans for free publishing them from CI builds,
it would help with analyzing dependency graph and build performance.
--
Best regards,
Konstantin Gribov
пт, 14 окт. 2022 г., 17:13 Nicholas DiPiazza <ni...@gmail.com>:
> Is there a branch started on the Gradle thing? I have some cycles and can
> use them to upgrade to Gradle on Tika, if desired.
>
> On Thu, Oct 6, 2022 at 8:50 AM Nick Burch <ni...@apache.org> wrote:
>
> > On Thu, 6 Oct 2022, Tim Allison wrote:
> > > Happy to chat. Please put them in touch.
> >
> > Excellent, thanks Tim!
> >
> > Other than your past talks, have we got any info (eg on the wiki?) about
> > how to run the regression corpus?
> >
> > > I've been really impressed with what the POI team has done migrating
> > > from ant to gradle. On Tika, I don't think we have any special needs
> > > that would require deep gradle knowledge, but given the number of
> > > modules now, it will be non-trivial. Also, I take Nick D's concerns
> > > seriously.
> >
> > We don't have to swap from Maven - they have a plugin that integrates it
> >
> > Nick
> >
>
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Nicholas DiPiazza <ni...@gmail.com>.
Is there a branch started on the Gradle thing? I have some cycles and can
use them to upgrade to Gradle on Tika, if desired.
On Thu, Oct 6, 2022 at 8:50 AM Nick Burch <ni...@apache.org> wrote:
> On Thu, 6 Oct 2022, Tim Allison wrote:
> > Happy to chat. Please put them in touch.
>
> Excellent, thanks Tim!
>
> Other than your past talks, have we got any info (eg on the wiki?) about
> how to run the regression corpus?
>
> > I've been really impressed with what the POI team has done migrating
> > from ant to gradle. On Tika, I don't think we have any special needs
> > that would require deep gradle knowledge, but given the number of
> > modules now, it will be non-trivial. Also, I take Nick D's concerns
> > seriously.
>
> We don't have to swap from Maven - they have a plugin that integrates it
>
> Nick
>
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Nick Burch <ni...@apache.org>.
On Thu, 6 Oct 2022, Tim Allison wrote:
> Happy to chat. Please put them in touch.
Excellent, thanks Tim!
Other than your past talks, have we got any info (eg on the wiki?) about
how to run the regression corpus?
> I've been really impressed with what the POI team has done migrating
> from ant to gradle. On Tika, I don't think we have any special needs
> that would require deep gradle knowledge, but given the number of
> modules now, it will be non-trivial. Also, I take Nick D's concerns
> seriously.
We don't have to swap from Maven - they have a plugin that integrates it
Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Tim Allison <ta...@apache.org>.
Happy to chat. Please put them in touch.
I've been really impressed with what the POI team has done migrating
from ant to gradle. On Tika, I don't think we have any special needs
that would require deep gradle knowledge, but given the number of
modules now, it will be non-trivial. Also, I take Nick D's concerns
seriously.
If there's something cheap and easy to do with our current build, great.
If we're looking at a full migration, have at it on a dev branch!
Perhaps we can put that in a Tika 3.x that requires Java 11...early
next year?
Cheers,
Tim
On Wed, Oct 5, 2022 at 3:05 PM Nick Burch <ni...@apache.org> wrote:
>
> Hi All
>
> At ApacheCon this week, a Bob and myself ended up chatting with the folks
> from Gradle, who are keen to help ASF projects, and are discussing with
> the Infra team.
>
> The easier bit - they think they might be able to help speed up our maven
> build, especially the running of tests. Anyone have some time to give that
> a try? Will pass details along to anyone with the volunteer cycles
>
> The interesting bit - we told them about the regression corpus, and they
> got very excited as it sounds completely different to most of their normal
> "my build is slow" type problems. The size of it, and the fact that it
> isn't a simple pass/fail, seemed to catch their interest. Anyone (though
> probably only Tim...) intersted in talking them through how it works, and
> maybe getting one of their team access to the VM?
>
> Cheers
> Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Nick Burch <ni...@apache.org>.
On Wed, 5 Oct 2022, Oleg Tikhonov wrote:
> Honestly I am trying to port our project to gradle. But it goes not well.
> It is good idea. Is some folk can help, we can do it together.
Apparently Gradle Enterprise works with both Gradle and Maven! So we don't
even have to change our build -
https://docs.gradle.com/enterprise/maven-extension/
Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Oleg Tikhonov <ol...@apache.org>.
Hi Nick,
Honestly I am trying to port our project to gradle. But it goes not well.
It is good idea. Is some folk can help, we can do it together.
+1
Cheers,
Oleg
On Wed, Oct 5, 2022, 22:05 Nick Burch <ni...@apache.org> wrote:
> Hi All
>
> At ApacheCon this week, a Bob and myself ended up chatting with the folks
> from Gradle, who are keen to help ASF projects, and are discussing with
> the Infra team.
>
> The easier bit - they think they might be able to help speed up our maven
> build, especially the running of tests. Anyone have some time to give that
> a try? Will pass details along to anyone with the volunteer cycles
>
> The interesting bit - we told them about the regression corpus, and they
> got very excited as it sounds completely different to most of their normal
> "my build is slow" type problems. The size of it, and the fact that it
> isn't a simple pass/fail, seemed to catch their interest. Anyone (though
> probably only Tim...) intersted in talking them through how it works, and
> maybe getting one of their team access to the VM?
>
> Cheers
> Nick
>
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Nick Burch <ni...@apache.org>.
On Wed, 5 Oct 2022, Nicholas DiPiazza wrote:
> Are they offering the Gradle Build Cache stuff free for apache projects?
There's an announcement at ApacheCon in about an hour... I think the Infra
team are still working out the details on how it'll all work.
However, there's an additional offer to Tika for us to get some help on
our tests, especially the regression run. (I think it's open to other ASF
projects with "interesting" tests but we're the first ones to ask!)
Nick
Re: Possibly speeding up tests with Gradle - anyone interested?
Posted by Nicholas DiPiazza <ni...@gmail.com>.
Yeah i'd be interested in the details.
Are they offering the Gradle Build Cache stuff free for apache projects?
In my experience, Gradle actually can slow things down quite a bit in
general day-to-day development. But you get tradeoff of having a much more
functional build process (tasks written in Kotlin).
Example on one of my builds, a simple
*./gradlew tasks*
Took 18 seconds most of that time sat around derping in the Configuring
step.
It can also leak Gradle daemons like a Siv.
Example from my same project:
> Starting a Gradle Daemon, 4 busy and 18 incompatible Daemons could not be
reused, use --status for details
They sold us Gradle Build Cache but we quickly abandoned it because it kept
causing errors and licensing problems.
New versions of gradle result in full redownload all the deps.
Very interested to hear how they aim to improve speed.
On Wed, Oct 5, 2022 at 2:05 PM Nick Burch <ni...@apache.org> wrote:
> Hi All
>
> At ApacheCon this week, a Bob and myself ended up chatting with the folks
> from Gradle, who are keen to help ASF projects, and are discussing with
> the Infra team.
>
> The easier bit - they think they might be able to help speed up our maven
> build, especially the running of tests. Anyone have some time to give that
> a try? Will pass details along to anyone with the volunteer cycles
>
> The interesting bit - we told them about the regression corpus, and they
> got very excited as it sounds completely different to most of their normal
> "my build is slow" type problems. The size of it, and the fact that it
> isn't a simple pass/fail, seemed to catch their interest. Anyone (though
> probably only Tim...) intersted in talking them through how it works, and
> maybe getting one of their team access to the VM?
>
> Cheers
> Nick
>