You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Chris Hostetter <ho...@fucit.org> on 2024/07/08 19:29:58 UTC
`./gradlew clean check` no longer works on main branch?
It's been a bit since I built anything on the main branch, but it seems
like smething is now broken in the gradle task
depenendencies/parallelization if you try to run...
./gradlew clean check
A truncated example of what the output looks like is below, the final
failure message being "Unable to make progress running work. There are
items queued for execution but none of them can be started"
This problem doesn't happen if you run...
./gradlew clean; ./gradlew check
...so it seems like it has something to do with the task graph. It also
doesn't seem to happen on branch_9x (allthough there seem to be other
compilations on branch_9x at the moment)
Anybody else seeing this?
-Hoss
http://www.lucidworks.com/
----
hossman@slate:~/lucene/lucene [j21] [*main] $ ./gradlew clean check
> Task :errorProneSkipped
WARNING: errorprone disabled (skipped on builds not running inside CI
environments, pass -Pvalidation.errorprone=true to enable)
> Task :randomizationInfo
Running tests with randomization seed: tests.seed=AE138305D034A14A
Unable to make progress running work. The following items are queued for
execution but none of them can be started:
- Build ':':
- Waiting for nodes:
- :lucene:analysis.tests:wipeTaskTemp (state=SHOULD_RUN,
dependencies=NOT_COMPLETE, group=finalizer
:lucene:analysis.tests:wipeTaskTemp ordinal: task group 1, delegate:
default group, no dependencies, finalizes=[:lucene:analysis.tests:test
(SHOULD_RUN)] )
- :lucene:analysis:kuromoji:check (state=SHOULD_RUN,
dependencies=NOT_COMPLETE, group=task group 1,
dependencies=[:lucene:analysis:kuromoji:ecjLint (SHOULD_RUN),
:lucene:analysis:kuromoji:forbiddenApis (SHOULD_RUN),
:lucene:analysis:kuromoji:licenses (SHOULD_RUN), :luc
...
- Workers waiting for work: 6
- Stopped workers: 2
FAILURE: Build failed with an exception.
* What went wrong:
Unable to make progress running work. There are items queued for execution
but none of them can be started
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
BUILD FAILED in 36s
111 actionable tasks: 105 executed, 6 up-to-date
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
> matter? Searching our build.gradle, I only see:
> includeBuild("dev-tools/missing-doclet")
> when searching for "includeBuild" but maybe it's other syntax too?
>
Look at settings.gradle -
includeBuild("build-tools/build-infra")
D.
> On Fri, Jul 12, 2024 at 6:01 AM Dawid Weiss <da...@gmail.com> wrote:
> >
> >
> > I've spent some time on this. The problem is a result of the change from
> buildSrc to an included build (composite subproject). I don't think this
> should matter much but task ordering from the composite affects and is
> tangled with the top level project tasks - I filed this issue and provided
> a link to an excellent comment explaining why running clean and other tasks
> at the same time confuses gradle's task execution graph:
> >
> > https://github.com/apache/lucene/issues/13567
> >
> > I don't think there is any fix for this - those included composite
> builds are a nice way of applying the same verification logic across
> multiple things... As the gradle employee mentioned, the workaround for now
> is to run gradlew clean separately from any other task (if you need it).
> >
> > Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: `./gradlew clean check` no longer works on main branch?
Posted by David Smiley <ds...@apache.org>.
What "composite subproject" / "included build" specifically is the
matter? Searching our build.gradle, I only see:
includeBuild("dev-tools/missing-doclet")
when searching for "includeBuild" but maybe it's other syntax too?
On Fri, Jul 12, 2024 at 6:01 AM Dawid Weiss <da...@gmail.com> wrote:
>
>
> I've spent some time on this. The problem is a result of the change from buildSrc to an included build (composite subproject). I don't think this should matter much but task ordering from the composite affects and is tangled with the top level project tasks - I filed this issue and provided a link to an excellent comment explaining why running clean and other tasks at the same time confuses gradle's task execution graph:
>
> https://github.com/apache/lucene/issues/13567
>
> I don't think there is any fix for this - those included composite builds are a nice way of applying the same verification logic across multiple things... As the gradle employee mentioned, the workaround for now is to run gradlew clean separately from any other task (if you need it).
>
> Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
I've spent some time on this. The problem is a result of the change from
buildSrc to an included build (composite subproject). I don't think this
should matter much but task ordering from the composite affects and is
tangled with the top level project tasks - I filed this issue and provided
a link to an excellent comment explaining why running clean and other tasks
at the same time confuses gradle's task execution graph:
https://github.com/apache/lucene/issues/13567
I don't think there is any fix for this - those included composite builds
are a nice way of applying the same verification logic across multiple
things... As the gradle employee mentioned, the workaround for now is to
run gradlew clean separately from any other task (if you need it).
Dawid
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
>
> Fair -- I was mainly just wondering if you (or anyone) knew/remembered off
> the top of your head something that had changed in the "architecture" of
> our gradle deps on the main branch (since this problem doesn't sem to
> affect branch_9x, eventhough it also uses the same version of gradle)
>
I've changed the dependency lock checker/ validation from Palantir's to my
own but I doubt it's the cause of the problem. More likely it's present on
9x too but doesn't manifest due to different ordering (hash maps,
whatever).
> Yeah, it's mainly something I do when switching branches -- but also out
> of muscle memory.
>
Incremental builds should work fine even when you're switching branches,
actually. I admit that my fool-proof way of restoring pristine state is
based on git, not gradle -
git clean -xfd .
this removes everything though (including the generated gradle.properties)
so it may not be for everyone.
> Maybe we should add an 'afterEvaluate' hook to complain if people try to
> combine "clean" with other tasks on the command line? :)
>
Nah, better to try to figure out what's going on - I'll file an issue and
take a look if I can work it out.
Dawid
Re: `./gradlew clean check` no longer works on main branch?
Posted by Chris Hostetter <ho...@fucit.org>.
: The build does need an overhaul - there are some problems with it that I
: know about - but it's not as simple as reviewing the dependencies you
: mentioned - there are also implicit dependencies that are derived from
: configurations dependencies, source sets, etc.
Fair -- I was mainly just wondering if you (or anyone) knew/remembered off
the top of your head something that had changed in the "architecture" of
our gradle deps on the main branch (since this problem doesn't sem to
affect branch_9x, eventhough it also uses the same version of gradle)
: in whatever order". So you may be better off setting up an alias in bash to
: run gradlew clean && gradlew xyz in sequence... Or - better yet - not run
: gradlew clean at all - the incremental build should work fine.
Yeah, it's mainly something I do when switching branches -- but also out
of muscle memory.
Maybe we should add an 'afterEvaluate' hook to complain if people try to
combine "clean" with other tasks on the command line? :)
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
> ...so maybe there is a way to avoid this problem in lucene by
> removing / re-thinking / tweaking some of dependsOn, mustRunAfter,
> or finalizedBy declarations somewhere?
The build does need an overhaul - there are some problems with it that I
know about - but it's not as simple as reviewing the dependencies you
mentioned - there are also implicit dependencies that are derived from
configurations dependencies, source sets, etc.
And even if it's a bug in the build - it should be reported as a circular
dependency by gradle, not as a cryptic internal error... I do have a plan
to work on the build in my spare time - to speed it up and clean up the odd
bits - but the spare time seems to be evaporating like ether.
Also, historically, running "gradle clean xyz" has been a headache since I
can remember because clean is inserted as a normal task into the task graph
and then manipulated together with the rest of the dependencies. It's not
"everything that depends on clean, followed by everything that depends on
xyz", as would be intuitive. It's "everything that depends on clean or xyz,
in whatever order". So you may be better off setting up an alias in bash to
run gradlew clean && gradlew xyz in sequence... Or - better yet - not run
gradlew clean at all - the incremental build should work fine.
D.
Re: `./gradlew clean check` no longer works on main branch?
Posted by Chris Hostetter <ho...@fucit.org>.
: Seems like a bug in gradle somewhere, Chris -
: https://github.com/gradle/gradle/issues/21325
Following some of the linked commits/issues/PRs from that URL, and reading
between the lines, it sounds like gradle 7.5 (and above) changed something
in how gradle analyzes what/where/when/how it can pipeline/parallelize
tasks such that it's now smarter in the common case, but can't detect
problems like we're seeing -- and the gradle devs seem to be playing whack
-a-mole trying to fix underlying problem(s)...
https://github.com/gradle/gradle/issues/21325
- fixed in 7.6 by pull/21643
https://github.com/gradle/gradle/issues/25951
- fixed in 8.4 by pull/26172
...the general "vibe" I get from the comments on these issues is that the
gradle devs seem to feel like these fixes / workarounds wouldn't be
neccessary if people's build.gradle files weren't so broken/smelly ...
to "blame the victim" -- IIUC for having circular task interactions...
https://github.com/gradle/gradle/issues/25951#issuecomment-1667761257
https://github.com/gradle/gradle/issues/25951#issuecomment-1674808161
...so maybe there is a way to avoid this problem in lucene by
removing / re-thinking / tweaking some of dependsOn, mustRunAfter,
or finalizedBy declarations somewhere?
I have no idea *where* -- but presumably it's in something in the
build.gradle files on main that isn't on branch_9x ?
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
Seems like a bug in gradle somewhere, Chris -
https://github.com/gradle/gradle/issues/21325
On Mon, Jul 8, 2024 at 10:28 PM Dawid Weiss <da...@gmail.com> wrote:
>
> Yep, same for me. Looks terrible. I don't know what this is, sorry.
>
> Dawid
>
>
> On Mon, Jul 8, 2024 at 9:30 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> It's been a bit since I built anything on the main branch, but it seems
>> like smething is now broken in the gradle task
>> depenendencies/parallelization if you try to run...
>>
>> ./gradlew clean check
>>
>> A truncated example of what the output looks like is below, the final
>> failure message being "Unable to make progress running work. There are
>> items queued for execution but none of them can be started"
>>
>> This problem doesn't happen if you run...
>>
>> ./gradlew clean; ./gradlew check
>>
>> ...so it seems like it has something to do with the task graph. It also
>> doesn't seem to happen on branch_9x (allthough there seem to be other
>> compilations on branch_9x at the moment)
>>
>> Anybody else seeing this?
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ----
>>
>> hossman@slate:~/lucene/lucene [j21] [*main] $ ./gradlew clean check
>>
>> > Task :errorProneSkipped
>> WARNING: errorprone disabled (skipped on builds not running inside CI
>> environments, pass -Pvalidation.errorprone=true to enable)
>>
>> > Task :randomizationInfo
>> Running tests with randomization seed: tests.seed=AE138305D034A14A
>> Unable to make progress running work. The following items are queued for
>> execution but none of them can be started:
>> - Build ':':
>> - Waiting for nodes:
>> - :lucene:analysis.tests:wipeTaskTemp (state=SHOULD_RUN,
>> dependencies=NOT_COMPLETE, group=finalizer
>> :lucene:analysis.tests:wipeTaskTemp ordinal: task group 1, delegate:
>> default group, no dependencies, finalizes=[:lucene:analysis.tests:test
>> (SHOULD_RUN)] )
>> - :lucene:analysis:kuromoji:check (state=SHOULD_RUN,
>> dependencies=NOT_COMPLETE, group=task group 1,
>> dependencies=[:lucene:analysis:kuromoji:ecjLint (SHOULD_RUN),
>> :lucene:analysis:kuromoji:forbiddenApis (SHOULD_RUN),
>> :lucene:analysis:kuromoji:licenses (SHOULD_RUN), :luc
>>
>> ...
>>
>> - Workers waiting for work: 6
>> - Stopped workers: 2
>>
>> FAILURE: Build failed with an exception.
>>
>> * What went wrong:
>> Unable to make progress running work. There are items queued for
>> execution
>> but none of them can be started
>>
>> * Try:
>> > Run with --stacktrace option to get the stack trace.
>> > Run with --info or --debug option to get more log output.
>> > Get more help at https://help.gradle.org.
>>
>> BUILD FAILED in 36s
>> 111 actionable tasks: 105 executed, 6 up-to-date
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: `./gradlew clean check` no longer works on main branch?
Posted by Dawid Weiss <da...@gmail.com>.
Yep, same for me. Looks terrible. I don't know what this is, sorry.
Dawid
On Mon, Jul 8, 2024 at 9:30 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> It's been a bit since I built anything on the main branch, but it seems
> like smething is now broken in the gradle task
> depenendencies/parallelization if you try to run...
>
> ./gradlew clean check
>
> A truncated example of what the output looks like is below, the final
> failure message being "Unable to make progress running work. There are
> items queued for execution but none of them can be started"
>
> This problem doesn't happen if you run...
>
> ./gradlew clean; ./gradlew check
>
> ...so it seems like it has something to do with the task graph. It also
> doesn't seem to happen on branch_9x (allthough there seem to be other
> compilations on branch_9x at the moment)
>
> Anybody else seeing this?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ----
>
> hossman@slate:~/lucene/lucene [j21] [*main] $ ./gradlew clean check
>
> > Task :errorProneSkipped
> WARNING: errorprone disabled (skipped on builds not running inside CI
> environments, pass -Pvalidation.errorprone=true to enable)
>
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=AE138305D034A14A
> Unable to make progress running work. The following items are queued for
> execution but none of them can be started:
> - Build ':':
> - Waiting for nodes:
> - :lucene:analysis.tests:wipeTaskTemp (state=SHOULD_RUN,
> dependencies=NOT_COMPLETE, group=finalizer
> :lucene:analysis.tests:wipeTaskTemp ordinal: task group 1, delegate:
> default group, no dependencies, finalizes=[:lucene:analysis.tests:test
> (SHOULD_RUN)] )
> - :lucene:analysis:kuromoji:check (state=SHOULD_RUN,
> dependencies=NOT_COMPLETE, group=task group 1,
> dependencies=[:lucene:analysis:kuromoji:ecjLint (SHOULD_RUN),
> :lucene:analysis:kuromoji:forbiddenApis (SHOULD_RUN),
> :lucene:analysis:kuromoji:licenses (SHOULD_RUN), :luc
>
> ...
>
> - Workers waiting for work: 6
> - Stopped workers: 2
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Unable to make progress running work. There are items queued for execution
> but none of them can be started
>
> * Try:
> > Run with --stacktrace option to get the stack trace.
> > Run with --info or --debug option to get more log output.
> > Get more help at https://help.gradle.org.
>
> BUILD FAILED in 36s
> 111 actionable tasks: 105 executed, 6 up-to-date
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>