You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Dawid Weiss (Jira)" <ji...@apache.org> on 2020/07/10 08:09:00 UTC
[jira] [Updated] (LUCENE-9077) Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dawid Weiss updated LUCENE-9077:
--------------------------------
Description:
This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
* (/) Repro-line for failed tests/ runs.
* (/) add a top-level README note about building with gradle (and the required JVM).
* (/) add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
* (/) add an equivalent of 'rat-sources' to precommit.
* (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
* (/) javadoc compilation
Hard-to-implement stuff already investigated:
* (/) (done) -*Printing console output of failed tests.* There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit *tons* of output so it's an overkill.-
* (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.
Of lesser importance:
* (/) Add an equivalent of 'documentation-lint" to precommit.
* (/) Do not require files to be committed before running precommit. (staged files are fine).
* (/) add rendering of javadocs (gradlew javadoc)
* (/) identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
* (/) Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution?
* (/) There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
* (/) Precommit doesn't catch unused imports
* Attach javadocs to maven publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should be fixed in the ant build instead.
* Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.
was:
This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
* (/) Repro-line for failed tests/ runs.
* (/) add a top-level README note about building with gradle (and the required JVM).
* (/) add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
* (/) add an equivalent of 'rat-sources' to precommit.
* (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
* (/) javadoc compilation
Hard-to-implement stuff already investigated:
* (/) (done) -*Printing console output of failed tests.* There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit *tons* of output so it's an overkill.-
* (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.
Of lesser importance:
* Add an equivalent of 'documentation-lint" to precommit.
* (/) Do not require files to be committed before running precommit. (staged files are fine).
* (/) add rendering of javadocs (gradlew javadoc)
* Attach javadocs to maven publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should be fixed in the ant build instead.
* (/) identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
* Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).
** Remove idea task and just import from the gradle model? One less thing to maintain.
* Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution?
* There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
* Nightly-smoke also has some python execution, not sure of the status.
* (/) Precommit doesn't catch unused imports
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.
> Gradle build
> ------------
>
> Key: LUCENE-9077
> URL: https://issues.apache.org/jira/browse/LUCENE-9077
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9077-javadoc-locale-en-US.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.
> The code lives on *gradle-master* branch. It is kept with sync with *master*. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:
> gradlew :help
> A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
> * (/) Apply forbiddenAPIs
> * (/) Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
> * (/) Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
> * (/) Port other settings and randomizations from common-build.xml
> * (/) Configure security policy/ sandboxing for tests.
> * (/) test's console output on -Ptests.verbose=true
> * (/) add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
> * (/) jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
> * (/) verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
> * (/) Repro-line for failed tests/ runs.
> * (/) add a top-level README note about building with gradle (and the required JVM).
> * (/) add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
> * (/) add an equivalent of 'rat-sources' to precommit.
> * (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
> * (/) javadoc compilation
> Hard-to-implement stuff already investigated:
> * (/) (done) -*Printing console output of failed tests.* There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit *tons* of output so it's an overkill.-
> * (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.
> Of lesser importance:
> * (/) Add an equivalent of 'documentation-lint" to precommit.
> * (/) Do not require files to be committed before running precommit. (staged files are fine).
> * (/) add rendering of javadocs (gradlew javadoc)
> * (/) identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
> * (/) Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
> * I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution?
> * (/) There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
> * (/) Precommit doesn't catch unused imports
> * Attach javadocs to maven publications.
> * Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
> * if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should be fixed in the ant build instead.
> * Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
> * Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).
>
> *{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:
> * *Do not try to do too many things at once*. If we deviate too far from master, the branch will be hard to merge.
> * *Do everything in baby-steps* and add small, independent build fragments replacing the old ant infrastructure.
> * *Try to engage people to run, test and contribute early*. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org