You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@grails.apache.org by James Daugherty <jd...@jdresources.net.INVALID> on 2025/04/24 21:06:14 UTC
[DISCUSS][HELP] Reproducible Builds
Hi Everyone,
As part of being given permission to publish a release under the ASF, we
have to meet the requirements of the ASF for publishing builds. During our
security audit, the following was noted:
----- BEGIN QUOTE ------
One thing that I didn't see explicitly called out is 'how' to review/verify
the signed artifacts. As you may have seen in the release policy, releases
must be verified on 'trusted' hardware (
https://www.apache.org/legal/release-policy.html#owned-controlled-hardware).
The GHA builders don't count as 'trusted' on their own. In practice, *this
means that as part of the vote/review phase, any artifacts that are signed
with ASF key material must be independently built from source and verified
to match the artifacts produced by the GHA builder *(
https://infra.apache.org/release-signing.html#automated-release-signing).
AFAIK Gradle is generally fairly mature when it comes to making bit-by-bit
reproducible artifacts, but this might require setting some options (such
as
https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives)
and checking for remaining sources of nondeterminism in your build - there
may be some surprises in there.
----- END QUOTE ------
I've spent today looking at the reproducibility of our builds. I've made
initial changes under this PR:
https://github.com/apache/grails-core/pull/14670 I've added a summary of
the changes to the PR itself. Please take a moment to review and express
any concerns.
The problem in our case is the AST transforms we're using cause the
compiled class files to be non-determinant and thus cannot be verified.
The following artifacts are affected by Grails transforms and prevent
reproducible builds:
grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
grails-controllers/build/libs/grails-controllers-7.0.0-SNAPSHOT.jar
grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT.jar
grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT.jar
grails-data-hibernate5/grails-plugin/build/libs/grails-data-hibernate5-7.0.0-SNAPSHOT.jar
grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
grails-datamapping-support/build/libs/grails-datamapping-support-7.0.0-SNAPSHOT.jar
grails-datamapping-tck-base/build/libs/grails-datamapping-tck-base-7.0.0-SNAPSHOT-javadoc.jar
grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT-javadoc.jar
grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
grails-domain-class/build/libs/grails-domain-class-7.0.0-SNAPSHOT.jar
grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
grails-interceptors/build/libs/grails-interceptors-7.0.0-SNAPSHOT.jar
grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
grails-web-databinding/build/libs/grails-web-databinding-7.0.0-SNAPSHOT.jar
grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
I investigated the `@Entity` annotation which uses the
EntityASTTransformation in the `grails-datamapping-tck-domains` project as
a start. It is what injects other traits (like GormValidateable). The
resulting class file is different every build. Doing a hex comparison on
the file shows sometimes simple method ordering issues. Testing traits in
isolation with Groovy 4 does not have the same behavior.
I'm planning to continue researching this, but if anyone has experience
with the AST transforms or the ordering problem, it would be helpful to
solve this. This is probably the largest blocker to us proceeding with an
ASF release.
If we can't get this working, we'll be forced to build Grails locally for
any published release.
-James
Re: [DISCUSS][HELP] Reproducible Builds
Posted by Mattias Reichel <ma...@gmail.com>.
I think we should keep this as a side branch for now.
If I understand correctly, we will still have issues with reproducibility,
so I don't see the value in merging it at the moment.
Merging it will make Groovy a moving target that might create problems that
are difficult to diagnose/understand when working on other tasks.
Den ons 7 maj 2025 kl 08:28 skrev Søren Berg Glasius <so...@glasius.dk>:
> No objections.
>
> Den tirs. 6. maj 2025 kl. 16.54 skrev James Daugherty
> <jd...@jdresources.net.invalid>:
>
> > I reached out to security inquiring if our work around would be
> > acceptable. That will help guide us on the priority of the groovy
> > changes.
> >
> > I haven't seen anyone object to merging the snapshot changes and we can
> > easily reverse them if needed. I intend to merge these later today
> unless
> > there are objections.
> >
> > -James
> >
> > On Mon, May 5, 2025 at 9:08 PM James Daugherty <
> jdaugherty@jdresources.net
> > >
> > wrote:
> >
> > > To go back to security fully, I think we'll need to make a script that
> > > downloads the jars, builds locally, and then compares. Currently the
> > > script locally just builds twice and compares.
> > >
> > > As a starter, I'll ask security if they're ok with diffing via the
> > > decompile approach we've been taking.
> > >
> > > On Mon, May 5, 2025 at 2:24 PM James Fredley <ja...@apache.org>
> > > wrote:
> > >
> > >> I think we should proceed with updating the ASF security team on the
> > >> progress so we receive updated feedback.
> > >>
> > >> Since we are working on 7.0.0-SNAPSHOT, will most likely need Groovy
> > >> 4.0.27 for Grails 7.0.0-M4 and already put the Apache Snapshot
> > Repository
> > >> in the right locations to pull Grails SNAPSHOTS and Groovy SNAPSHOTS,
> > I'd
> > >> be OK with Groovy 4.0.27-SNAPSHOT for the entire build for now and
> also
> > OK
> > >> just for documentation. We ran into issues running an earlier Groovy
> > >> SNAPSHOT last year, but the root causes for those issues are addressed
> > in
> > >> grails-core.
> > >>
> > >> I also think it would be fine to merge and then start a new PR once
> > >> addition updates can be made.
> > >>
> > >> On 2025/05/05 16:11:22 James Daugherty wrote:
> > >> > Paul was able to fix the groovydoc differences via groovy
> > >> 4.0.27-SNAPSHOT.
> > >> > I've updated the ticket; we're down to these jars differing:
> > >> >
> > >> > grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
> > >> > grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> > >> > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> > >> >
> > >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> > >> >
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> > >> >
> > grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> > >> >
> > >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> > >> >
> > >>
> >
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> > >> >
> > >> grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
> > >> >
> > >> > My latest PR is here:
> > >> https://github.com/apache/grails-core/issues/14679 It
> > >> > uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead
> of
> > >> just
> > >> > assigning 4.0.27 for documentation generation, I set it for the
> entire
> > >> > project default. This PR also fixes the comparison of the jars in
> the
> > >> > grails-gradle project & addresses some documentation issues (one
> > >> example:
> > >> > grails data docs aren't fully being generated).
> > >> >
> > >> > Of these jars, the differences look due to fields reordering from
> > traits
> > >> > (fields have annotations like @TraitBridge on them). I'm guessing
> > >> there's
> > >> > some issue related to how groovy processes the traits, but this
> could
> > be
> > >> > something we're doing (I haven't figured out the specifics, but
> > >> > ApplicationTagLib is the easiest case of this since none of our AST
> > >> > transforms appear to do anything to that file). I am under the
> > >> impression
> > >> > that Paul is continuing to look at this issue to see if he can help
> or
> > >> > identify a groovy specific issue.
> > >> >
> > >> > My current dilemma is if I should merge this PR. I could scale it
> > back
> > >> to
> > >> > use groovy 4.0.26 for compiling & only use the snapshot for
> > >> documentation
> > >> > generation. The problem is if this is a groovy issue, we'll need a
> > >> groovy
> > >> > 4.0.27 release anyhow. What are people's thoughts here? Should I
> > just
> > >> > leave this on a side branch for now? Should we depend entirely on
> the
> > >> > snapshot build? Should only documentation depend on the snapshot
> > build?
> > >> >
> > >> > Alternatively, I think we can respond to ASF security that we can
> > verify
> > >> > the build artifacts by using the same process we use to test
> > >> reproducible
> > >> > builds. Namely, we can diff any artifact that has a different
> > checksum
> > >> via
> > >> > the decompiler in IntelliJ to confirm the code is the same. This
> > >> solution
> > >> > isn't a permanent solution, but for the Milestone / RC builds, I
> think
> > >> it's
> > >> > acceptable.
> > >> >
> > >> > -James
> > >> >
> > >> > On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <
> > >> jdaugherty@jdresources.net>
> > >> > wrote:
> > >> >
> > >> > > Hi Everyone,
> > >> > >
> > >> > > I summarized my progress thus far under this ticket:
> > >> > > https://github.com/apache/grails-core/issues/14679
> > >> > >
> > >> > > I checked in several scripts (see ticket) that assist in easily
> > >> testing
> > >> > > the jar files for differences.
> > >> > >
> > >> > > From what I can tell, there is an ordering issue when multiple,
> > >> inherited
> > >> > > traits are involved. I've found a simplified scenario in the
> > >> grails-gsp
> > >> > > plugin (ApplicationTagLib) that doesn't appear to be transformed
> by
> > >> our
> > >> > > code in a material way, and yet the difference always deals with
> the
> > >> copied
> > >> > > trait methods being in a random order. I'm at a loss for how to
> > >> further
> > >> > > debug this issue. From what I can tell, Grails isn't copying
> those
> > >> > > methods. I would assume Groovy is.
> > >> > >
> > >> > > The steps to reproduce this:
> > >> > > 1. etc/bin/test-reproducible-builds.sh
> > >> > > 2. etc/bin/extract-build-artifact.sh
> > >> > > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > >> > > 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
> > >> > > `etc/bin/results` and right click to select `Compare Directories`
> > >> > >
> > >> > > I intend to send an email to the groovy dev list to see if they
> can
> > >> help
> > >> > > me understand how I can debug this further.
> > >> > >
> > >> > > If we can't resolve these issues fully in Grails 7, the good news
> is
> > >> the
> > >> > > scripts I've checked in can be used to "verify" that the jars were
> > >> built
> > >> > > correctly - we can build locally and manually diff the artifacts
> to
> > >> confirm
> > >> > > they're correct. This is the work around I plan to suggest to ASF
> > >> security.
> > >> > >
> > >> > > -James
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
> > >> > > jdaugherty@jdresources.net> wrote:
> > >> > >
> > >> > >> Thank you Jochen. I believe I've switched to order stable
> > >> collections
> > >> > >> throughout the AST code base. Unfortunately, I'm still seeing
> > >> issues.
> > >> > >>
> > >> > >> I've narrowed the differences down to the following:
> > >> > >>
> > >>
> > grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >>
> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >>
> > grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> > >> > >>
> > grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >>
> >
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> > >> > >>
> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >>
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >>
> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >>
> > grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >>
> > grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
> > >> > >>
> > >> > >>
> > >>
> >
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> > >> > >>
> > >> > >> Since the javadoc jars now make up 13 of the remaining 26 jar
> > >> > >> differences, I was hoping someone can help here:
> > >> > >>
> > >> > >> The javadoc jars above are interesting because the source files &
> > >> > >> compiled classes no longer differ for projects like
> `grails-core`.
> > >> These
> > >> > >> jars actually contain the groovydoc (for java + groovy classes).
> > In
> > >> the
> > >> > >> case of `grails-core`, the example difference is the
> > >> > >> groovy.lang.Script#getBinding() and
> groovy.lang.Script#setBinding()
> > >> methods
> > >> > >> are reordered.
> > >> > >> Since this is groovydoc generating this, do you know if there are
> > >> known
> > >> > >> defined orders in groovydoc? This is from the gradle task, so I
> > >> assume
> > >> > >> this is being done with groovy 3's groovydoc. Is this something
> > >> addressed
> > >> > >> in groovydoc under groovy 4?
> > >> > >>
> > >> > >> The source file in question is rather simple. The methods being
> > >> > >> reordered are from the parent class (which is in Groovy instead
> of
> > >> > >> Grails). Here's the source:
> > >> > >>
> > >> > >> @CompileStatic
> > >> > >> abstract class SettingsFile extends Script {
> > >> > >>
> > >> > >>
> > >> > >> void include(String[] projectPaths) {
> > >> > >> binding.setVariable("projectPaths", projectPaths)
> > >> > >> }
> > >> > >>
> > >> > >> void includeFlat(String[] projectPaths) {
> > >> > >> binding.setVariable("projectPaths", projectPaths)
> > >> > >> }
> > >> > >>
> > >> > >> def methodMissing(String name, args) {
> > >> > >> // ignore
> > >> > >> }
> > >> > >>
> > >> > >> def propertyMissing(String name) {
> > >> > >> // ignore
> > >> > >> }
> > >> > >> }
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
> > >> > >> <bl...@gmx.org.invalid> wrote:
> > >> > >>
> > >> > >>> On 24.04.25 23:06, James Daugherty wrote:
> > >> > >>> [...]
> > >> > >>> > I'm planning to continue researching this, but if anyone has
> > >> experience
> > >> > >>> > with the AST transforms or the ordering problem, it would be
> > >> helpful to
> > >> > >>> > solve this. This is probably the largest blocker to us
> > proceeding
> > >> > >>> with an
> > >> > >>> > ASF release.
> > >> > >>>
> > >> > >>> as for the ordering. Groovy wants in general to use the order of
> > >> > >>> declaration in the source code. What we are talking about are
> most
> > >> > >>> likely methods added by transforms. And if it is one time stable
> > >> and one
> > >> > >>> time not, then I guess Maps are involved. ClassNode has several
> > >> > >>> getDeclaredMethodsXY methods, which are based on a methods maps,
> > >> while
> > >> > >>> getMethods or getMethodsList is based on the methods list.
> > >> > >>>
> > >> > >>> If you then have something like
> > >> > >>>
> > >> > >>> classNode.getAllDeclaredMethods().findAll(it annnotated with
> > >> XY).each
> > >> > >>> {doSomethingWith(it)}
> > >> > >>>
> > >> > >>> and there is more than one method annotated, then the order
> might
> > be
> > >> > >>> broken between builds. Just to give one example.
> > >> > >>>
> > >> > >>>
> > >> > >>> I would suggest to go through all transforms and check on the
> > usage
> > >> of
> > >> > >>> the getDeclaredMethods methods and then avoid them to use for
> > >> iteration.
> > >> > >>> They are ok for lookup of course.
> > >> > >>>
> > >> > >>> bye Jochen
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >
> >
>
>
> --
>
> Med venlig hilsen,
> Søren Berg Glasius
>
> Hedevej 1, Gl. Rye, 8680 Ry
> Mobile: +45 40 44 91 88
> --- Press ESC once to quit - twice to save the changes.
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by Søren Berg Glasius <so...@glasius.dk>.
No objections.
Den tirs. 6. maj 2025 kl. 16.54 skrev James Daugherty
<jd...@jdresources.net.invalid>:
> I reached out to security inquiring if our work around would be
> acceptable. That will help guide us on the priority of the groovy
> changes.
>
> I haven't seen anyone object to merging the snapshot changes and we can
> easily reverse them if needed. I intend to merge these later today unless
> there are objections.
>
> -James
>
> On Mon, May 5, 2025 at 9:08 PM James Daugherty <jdaugherty@jdresources.net
> >
> wrote:
>
> > To go back to security fully, I think we'll need to make a script that
> > downloads the jars, builds locally, and then compares. Currently the
> > script locally just builds twice and compares.
> >
> > As a starter, I'll ask security if they're ok with diffing via the
> > decompile approach we've been taking.
> >
> > On Mon, May 5, 2025 at 2:24 PM James Fredley <ja...@apache.org>
> > wrote:
> >
> >> I think we should proceed with updating the ASF security team on the
> >> progress so we receive updated feedback.
> >>
> >> Since we are working on 7.0.0-SNAPSHOT, will most likely need Groovy
> >> 4.0.27 for Grails 7.0.0-M4 and already put the Apache Snapshot
> Repository
> >> in the right locations to pull Grails SNAPSHOTS and Groovy SNAPSHOTS,
> I'd
> >> be OK with Groovy 4.0.27-SNAPSHOT for the entire build for now and also
> OK
> >> just for documentation. We ran into issues running an earlier Groovy
> >> SNAPSHOT last year, but the root causes for those issues are addressed
> in
> >> grails-core.
> >>
> >> I also think it would be fine to merge and then start a new PR once
> >> addition updates can be made.
> >>
> >> On 2025/05/05 16:11:22 James Daugherty wrote:
> >> > Paul was able to fix the groovydoc differences via groovy
> >> 4.0.27-SNAPSHOT.
> >> > I've updated the ticket; we're down to these jars differing:
> >> >
> >> > grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
> >> > grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> >> > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> >> >
> >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> >> > grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> >> >
> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> >> >
> >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> >> >
> >>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> >> >
> >> grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
> >> >
> >> > My latest PR is here:
> >> https://github.com/apache/grails-core/issues/14679 It
> >> > uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead of
> >> just
> >> > assigning 4.0.27 for documentation generation, I set it for the entire
> >> > project default. This PR also fixes the comparison of the jars in the
> >> > grails-gradle project & addresses some documentation issues (one
> >> example:
> >> > grails data docs aren't fully being generated).
> >> >
> >> > Of these jars, the differences look due to fields reordering from
> traits
> >> > (fields have annotations like @TraitBridge on them). I'm guessing
> >> there's
> >> > some issue related to how groovy processes the traits, but this could
> be
> >> > something we're doing (I haven't figured out the specifics, but
> >> > ApplicationTagLib is the easiest case of this since none of our AST
> >> > transforms appear to do anything to that file). I am under the
> >> impression
> >> > that Paul is continuing to look at this issue to see if he can help or
> >> > identify a groovy specific issue.
> >> >
> >> > My current dilemma is if I should merge this PR. I could scale it
> back
> >> to
> >> > use groovy 4.0.26 for compiling & only use the snapshot for
> >> documentation
> >> > generation. The problem is if this is a groovy issue, we'll need a
> >> groovy
> >> > 4.0.27 release anyhow. What are people's thoughts here? Should I
> just
> >> > leave this on a side branch for now? Should we depend entirely on the
> >> > snapshot build? Should only documentation depend on the snapshot
> build?
> >> >
> >> > Alternatively, I think we can respond to ASF security that we can
> verify
> >> > the build artifacts by using the same process we use to test
> >> reproducible
> >> > builds. Namely, we can diff any artifact that has a different
> checksum
> >> via
> >> > the decompiler in IntelliJ to confirm the code is the same. This
> >> solution
> >> > isn't a permanent solution, but for the Milestone / RC builds, I think
> >> it's
> >> > acceptable.
> >> >
> >> > -James
> >> >
> >> > On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <
> >> jdaugherty@jdresources.net>
> >> > wrote:
> >> >
> >> > > Hi Everyone,
> >> > >
> >> > > I summarized my progress thus far under this ticket:
> >> > > https://github.com/apache/grails-core/issues/14679
> >> > >
> >> > > I checked in several scripts (see ticket) that assist in easily
> >> testing
> >> > > the jar files for differences.
> >> > >
> >> > > From what I can tell, there is an ordering issue when multiple,
> >> inherited
> >> > > traits are involved. I've found a simplified scenario in the
> >> grails-gsp
> >> > > plugin (ApplicationTagLib) that doesn't appear to be transformed by
> >> our
> >> > > code in a material way, and yet the difference always deals with the
> >> copied
> >> > > trait methods being in a random order. I'm at a loss for how to
> >> further
> >> > > debug this issue. From what I can tell, Grails isn't copying those
> >> > > methods. I would assume Groovy is.
> >> > >
> >> > > The steps to reproduce this:
> >> > > 1. etc/bin/test-reproducible-builds.sh
> >> > > 2. etc/bin/extract-build-artifact.sh
> >> > > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> >> > > 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
> >> > > `etc/bin/results` and right click to select `Compare Directories`
> >> > >
> >> > > I intend to send an email to the groovy dev list to see if they can
> >> help
> >> > > me understand how I can debug this further.
> >> > >
> >> > > If we can't resolve these issues fully in Grails 7, the good news is
> >> the
> >> > > scripts I've checked in can be used to "verify" that the jars were
> >> built
> >> > > correctly - we can build locally and manually diff the artifacts to
> >> confirm
> >> > > they're correct. This is the work around I plan to suggest to ASF
> >> security.
> >> > >
> >> > > -James
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
> >> > > jdaugherty@jdresources.net> wrote:
> >> > >
> >> > >> Thank you Jochen. I believe I've switched to order stable
> >> collections
> >> > >> throughout the AST code base. Unfortunately, I'm still seeing
> >> issues.
> >> > >>
> >> > >> I've narrowed the differences down to the following:
> >> > >>
> >>
> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> >> > >>
> >>
> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> >> > >>
> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >>
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> >> > >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> >> > >>
> >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> >> > >>
> >> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >>
> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
> >> > >> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >>
> >>
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> >> > >>
> >>
> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
> >> > >>
> >> > >>
> >>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> >> > >>
> >> > >> Since the javadoc jars now make up 13 of the remaining 26 jar
> >> > >> differences, I was hoping someone can help here:
> >> > >>
> >> > >> The javadoc jars above are interesting because the source files &
> >> > >> compiled classes no longer differ for projects like `grails-core`.
> >> These
> >> > >> jars actually contain the groovydoc (for java + groovy classes).
> In
> >> the
> >> > >> case of `grails-core`, the example difference is the
> >> > >> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding()
> >> methods
> >> > >> are reordered.
> >> > >> Since this is groovydoc generating this, do you know if there are
> >> known
> >> > >> defined orders in groovydoc? This is from the gradle task, so I
> >> assume
> >> > >> this is being done with groovy 3's groovydoc. Is this something
> >> addressed
> >> > >> in groovydoc under groovy 4?
> >> > >>
> >> > >> The source file in question is rather simple. The methods being
> >> > >> reordered are from the parent class (which is in Groovy instead of
> >> > >> Grails). Here's the source:
> >> > >>
> >> > >> @CompileStatic
> >> > >> abstract class SettingsFile extends Script {
> >> > >>
> >> > >>
> >> > >> void include(String[] projectPaths) {
> >> > >> binding.setVariable("projectPaths", projectPaths)
> >> > >> }
> >> > >>
> >> > >> void includeFlat(String[] projectPaths) {
> >> > >> binding.setVariable("projectPaths", projectPaths)
> >> > >> }
> >> > >>
> >> > >> def methodMissing(String name, args) {
> >> > >> // ignore
> >> > >> }
> >> > >>
> >> > >> def propertyMissing(String name) {
> >> > >> // ignore
> >> > >> }
> >> > >> }
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
> >> > >> <bl...@gmx.org.invalid> wrote:
> >> > >>
> >> > >>> On 24.04.25 23:06, James Daugherty wrote:
> >> > >>> [...]
> >> > >>> > I'm planning to continue researching this, but if anyone has
> >> experience
> >> > >>> > with the AST transforms or the ordering problem, it would be
> >> helpful to
> >> > >>> > solve this. This is probably the largest blocker to us
> proceeding
> >> > >>> with an
> >> > >>> > ASF release.
> >> > >>>
> >> > >>> as for the ordering. Groovy wants in general to use the order of
> >> > >>> declaration in the source code. What we are talking about are most
> >> > >>> likely methods added by transforms. And if it is one time stable
> >> and one
> >> > >>> time not, then I guess Maps are involved. ClassNode has several
> >> > >>> getDeclaredMethodsXY methods, which are based on a methods maps,
> >> while
> >> > >>> getMethods or getMethodsList is based on the methods list.
> >> > >>>
> >> > >>> If you then have something like
> >> > >>>
> >> > >>> classNode.getAllDeclaredMethods().findAll(it annnotated with
> >> XY).each
> >> > >>> {doSomethingWith(it)}
> >> > >>>
> >> > >>> and there is more than one method annotated, then the order might
> be
> >> > >>> broken between builds. Just to give one example.
> >> > >>>
> >> > >>>
> >> > >>> I would suggest to go through all transforms and check on the
> usage
> >> of
> >> > >>> the getDeclaredMethods methods and then avoid them to use for
> >> iteration.
> >> > >>> They are ok for lookup of course.
> >> > >>>
> >> > >>> bye Jochen
> >> > >>>
> >> > >>
> >> >
> >>
> >
>
--
Med venlig hilsen,
Søren Berg Glasius
Hedevej 1, Gl. Rye, 8680 Ry
Mobile: +45 40 44 91 88
--- Press ESC once to quit - twice to save the changes.
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
I reached out to security inquiring if our work around would be
acceptable. That will help guide us on the priority of the groovy
changes.
I haven't seen anyone object to merging the snapshot changes and we can
easily reverse them if needed. I intend to merge these later today unless
there are objections.
-James
On Mon, May 5, 2025 at 9:08 PM James Daugherty <jd...@jdresources.net>
wrote:
> To go back to security fully, I think we'll need to make a script that
> downloads the jars, builds locally, and then compares. Currently the
> script locally just builds twice and compares.
>
> As a starter, I'll ask security if they're ok with diffing via the
> decompile approach we've been taking.
>
> On Mon, May 5, 2025 at 2:24 PM James Fredley <ja...@apache.org>
> wrote:
>
>> I think we should proceed with updating the ASF security team on the
>> progress so we receive updated feedback.
>>
>> Since we are working on 7.0.0-SNAPSHOT, will most likely need Groovy
>> 4.0.27 for Grails 7.0.0-M4 and already put the Apache Snapshot Repository
>> in the right locations to pull Grails SNAPSHOTS and Groovy SNAPSHOTS, I'd
>> be OK with Groovy 4.0.27-SNAPSHOT for the entire build for now and also OK
>> just for documentation. We ran into issues running an earlier Groovy
>> SNAPSHOT last year, but the root causes for those issues are addressed in
>> grails-core.
>>
>> I also think it would be fine to merge and then start a new PR once
>> addition updates can be made.
>>
>> On 2025/05/05 16:11:22 James Daugherty wrote:
>> > Paul was able to fix the groovydoc differences via groovy
>> 4.0.27-SNAPSHOT.
>> > I've updated the ticket; we're down to these jars differing:
>> >
>> > grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
>> >
>> grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
>> > grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
>> >
>> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
>> >
>> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
>> > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
>> >
>> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
>> >
>> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
>> > grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
>> > grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
>> >
>> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
>> >
>> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
>> >
>> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
>> >
>> grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
>> >
>> > My latest PR is here:
>> https://github.com/apache/grails-core/issues/14679 It
>> > uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead of
>> just
>> > assigning 4.0.27 for documentation generation, I set it for the entire
>> > project default. This PR also fixes the comparison of the jars in the
>> > grails-gradle project & addresses some documentation issues (one
>> example:
>> > grails data docs aren't fully being generated).
>> >
>> > Of these jars, the differences look due to fields reordering from traits
>> > (fields have annotations like @TraitBridge on them). I'm guessing
>> there's
>> > some issue related to how groovy processes the traits, but this could be
>> > something we're doing (I haven't figured out the specifics, but
>> > ApplicationTagLib is the easiest case of this since none of our AST
>> > transforms appear to do anything to that file). I am under the
>> impression
>> > that Paul is continuing to look at this issue to see if he can help or
>> > identify a groovy specific issue.
>> >
>> > My current dilemma is if I should merge this PR. I could scale it back
>> to
>> > use groovy 4.0.26 for compiling & only use the snapshot for
>> documentation
>> > generation. The problem is if this is a groovy issue, we'll need a
>> groovy
>> > 4.0.27 release anyhow. What are people's thoughts here? Should I just
>> > leave this on a side branch for now? Should we depend entirely on the
>> > snapshot build? Should only documentation depend on the snapshot build?
>> >
>> > Alternatively, I think we can respond to ASF security that we can verify
>> > the build artifacts by using the same process we use to test
>> reproducible
>> > builds. Namely, we can diff any artifact that has a different checksum
>> via
>> > the decompiler in IntelliJ to confirm the code is the same. This
>> solution
>> > isn't a permanent solution, but for the Milestone / RC builds, I think
>> it's
>> > acceptable.
>> >
>> > -James
>> >
>> > On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <
>> jdaugherty@jdresources.net>
>> > wrote:
>> >
>> > > Hi Everyone,
>> > >
>> > > I summarized my progress thus far under this ticket:
>> > > https://github.com/apache/grails-core/issues/14679
>> > >
>> > > I checked in several scripts (see ticket) that assist in easily
>> testing
>> > > the jar files for differences.
>> > >
>> > > From what I can tell, there is an ordering issue when multiple,
>> inherited
>> > > traits are involved. I've found a simplified scenario in the
>> grails-gsp
>> > > plugin (ApplicationTagLib) that doesn't appear to be transformed by
>> our
>> > > code in a material way, and yet the difference always deals with the
>> copied
>> > > trait methods being in a random order. I'm at a loss for how to
>> further
>> > > debug this issue. From what I can tell, Grails isn't copying those
>> > > methods. I would assume Groovy is.
>> > >
>> > > The steps to reproduce this:
>> > > 1. etc/bin/test-reproducible-builds.sh
>> > > 2. etc/bin/extract-build-artifact.sh
>> > > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
>> > > 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
>> > > `etc/bin/results` and right click to select `Compare Directories`
>> > >
>> > > I intend to send an email to the groovy dev list to see if they can
>> help
>> > > me understand how I can debug this further.
>> > >
>> > > If we can't resolve these issues fully in Grails 7, the good news is
>> the
>> > > scripts I've checked in can be used to "verify" that the jars were
>> built
>> > > correctly - we can build locally and manually diff the artifacts to
>> confirm
>> > > they're correct. This is the work around I plan to suggest to ASF
>> security.
>> > >
>> > > -James
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
>> > > jdaugherty@jdresources.net> wrote:
>> > >
>> > >> Thank you Jochen. I believe I've switched to order stable
>> collections
>> > >> throughout the AST code base. Unfortunately, I'm still seeing
>> issues.
>> > >>
>> > >> I've narrowed the differences down to the following:
>> > >>
>> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
>> > >>
>> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
>> > >> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
>> > >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
>> > >>
>> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
>> > >>
>> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
>> > >>
>> > >>
>> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
>> > >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
>> > >>
>> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
>> > >>
>> > >>
>> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
>> > >>
>> > >> Since the javadoc jars now make up 13 of the remaining 26 jar
>> > >> differences, I was hoping someone can help here:
>> > >>
>> > >> The javadoc jars above are interesting because the source files &
>> > >> compiled classes no longer differ for projects like `grails-core`.
>> These
>> > >> jars actually contain the groovydoc (for java + groovy classes). In
>> the
>> > >> case of `grails-core`, the example difference is the
>> > >> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding()
>> methods
>> > >> are reordered.
>> > >> Since this is groovydoc generating this, do you know if there are
>> known
>> > >> defined orders in groovydoc? This is from the gradle task, so I
>> assume
>> > >> this is being done with groovy 3's groovydoc. Is this something
>> addressed
>> > >> in groovydoc under groovy 4?
>> > >>
>> > >> The source file in question is rather simple. The methods being
>> > >> reordered are from the parent class (which is in Groovy instead of
>> > >> Grails). Here's the source:
>> > >>
>> > >> @CompileStatic
>> > >> abstract class SettingsFile extends Script {
>> > >>
>> > >>
>> > >> void include(String[] projectPaths) {
>> > >> binding.setVariable("projectPaths", projectPaths)
>> > >> }
>> > >>
>> > >> void includeFlat(String[] projectPaths) {
>> > >> binding.setVariable("projectPaths", projectPaths)
>> > >> }
>> > >>
>> > >> def methodMissing(String name, args) {
>> > >> // ignore
>> > >> }
>> > >>
>> > >> def propertyMissing(String name) {
>> > >> // ignore
>> > >> }
>> > >> }
>> > >>
>> > >>
>> > >>
>> > >> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
>> > >> <bl...@gmx.org.invalid> wrote:
>> > >>
>> > >>> On 24.04.25 23:06, James Daugherty wrote:
>> > >>> [...]
>> > >>> > I'm planning to continue researching this, but if anyone has
>> experience
>> > >>> > with the AST transforms or the ordering problem, it would be
>> helpful to
>> > >>> > solve this. This is probably the largest blocker to us proceeding
>> > >>> with an
>> > >>> > ASF release.
>> > >>>
>> > >>> as for the ordering. Groovy wants in general to use the order of
>> > >>> declaration in the source code. What we are talking about are most
>> > >>> likely methods added by transforms. And if it is one time stable
>> and one
>> > >>> time not, then I guess Maps are involved. ClassNode has several
>> > >>> getDeclaredMethodsXY methods, which are based on a methods maps,
>> while
>> > >>> getMethods or getMethodsList is based on the methods list.
>> > >>>
>> > >>> If you then have something like
>> > >>>
>> > >>> classNode.getAllDeclaredMethods().findAll(it annnotated with
>> XY).each
>> > >>> {doSomethingWith(it)}
>> > >>>
>> > >>> and there is more than one method annotated, then the order might be
>> > >>> broken between builds. Just to give one example.
>> > >>>
>> > >>>
>> > >>> I would suggest to go through all transforms and check on the usage
>> of
>> > >>> the getDeclaredMethods methods and then avoid them to use for
>> iteration.
>> > >>> They are ok for lookup of course.
>> > >>>
>> > >>> bye Jochen
>> > >>>
>> > >>
>> >
>>
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
To go back to security fully, I think we'll need to make a script that
downloads the jars, builds locally, and then compares. Currently the
script locally just builds twice and compares.
As a starter, I'll ask security if they're ok with diffing via the
decompile approach we've been taking.
On Mon, May 5, 2025 at 2:24 PM James Fredley <ja...@apache.org>
wrote:
> I think we should proceed with updating the ASF security team on the
> progress so we receive updated feedback.
>
> Since we are working on 7.0.0-SNAPSHOT, will most likely need Groovy
> 4.0.27 for Grails 7.0.0-M4 and already put the Apache Snapshot Repository
> in the right locations to pull Grails SNAPSHOTS and Groovy SNAPSHOTS, I'd
> be OK with Groovy 4.0.27-SNAPSHOT for the entire build for now and also OK
> just for documentation. We ran into issues running an earlier Groovy
> SNAPSHOT last year, but the root causes for those issues are addressed in
> grails-core.
>
> I also think it would be fine to merge and then start a new PR once
> addition updates can be made.
>
> On 2025/05/05 16:11:22 James Daugherty wrote:
> > Paul was able to fix the groovydoc differences via groovy
> 4.0.27-SNAPSHOT.
> > I've updated the ticket; we're down to these jars differing:
> >
> > grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> >
> grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
> > grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> >
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> >
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> >
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> > grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> > grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> > grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> >
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> >
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> >
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> >
> grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
> >
> > My latest PR is here: https://github.com/apache/grails-core/issues/14679
> It
> > uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead of
> just
> > assigning 4.0.27 for documentation generation, I set it for the entire
> > project default. This PR also fixes the comparison of the jars in the
> > grails-gradle project & addresses some documentation issues (one example:
> > grails data docs aren't fully being generated).
> >
> > Of these jars, the differences look due to fields reordering from traits
> > (fields have annotations like @TraitBridge on them). I'm guessing
> there's
> > some issue related to how groovy processes the traits, but this could be
> > something we're doing (I haven't figured out the specifics, but
> > ApplicationTagLib is the easiest case of this since none of our AST
> > transforms appear to do anything to that file). I am under the
> impression
> > that Paul is continuing to look at this issue to see if he can help or
> > identify a groovy specific issue.
> >
> > My current dilemma is if I should merge this PR. I could scale it back
> to
> > use groovy 4.0.26 for compiling & only use the snapshot for documentation
> > generation. The problem is if this is a groovy issue, we'll need a
> groovy
> > 4.0.27 release anyhow. What are people's thoughts here? Should I just
> > leave this on a side branch for now? Should we depend entirely on the
> > snapshot build? Should only documentation depend on the snapshot build?
> >
> > Alternatively, I think we can respond to ASF security that we can verify
> > the build artifacts by using the same process we use to test reproducible
> > builds. Namely, we can diff any artifact that has a different checksum
> via
> > the decompiler in IntelliJ to confirm the code is the same. This
> solution
> > isn't a permanent solution, but for the Milestone / RC builds, I think
> it's
> > acceptable.
> >
> > -James
> >
> > On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <
> jdaugherty@jdresources.net>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I summarized my progress thus far under this ticket:
> > > https://github.com/apache/grails-core/issues/14679
> > >
> > > I checked in several scripts (see ticket) that assist in easily testing
> > > the jar files for differences.
> > >
> > > From what I can tell, there is an ordering issue when multiple,
> inherited
> > > traits are involved. I've found a simplified scenario in the
> grails-gsp
> > > plugin (ApplicationTagLib) that doesn't appear to be transformed by our
> > > code in a material way, and yet the difference always deals with the
> copied
> > > trait methods being in a random order. I'm at a loss for how to
> further
> > > debug this issue. From what I can tell, Grails isn't copying those
> > > methods. I would assume Groovy is.
> > >
> > > The steps to reproduce this:
> > > 1. etc/bin/test-reproducible-builds.sh
> > > 2. etc/bin/extract-build-artifact.sh
> > > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > > 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
> > > `etc/bin/results` and right click to select `Compare Directories`
> > >
> > > I intend to send an email to the groovy dev list to see if they can
> help
> > > me understand how I can debug this further.
> > >
> > > If we can't resolve these issues fully in Grails 7, the good news is
> the
> > > scripts I've checked in can be used to "verify" that the jars were
> built
> > > correctly - we can build locally and manually diff the artifacts to
> confirm
> > > they're correct. This is the work around I plan to suggest to ASF
> security.
> > >
> > > -James
> > >
> > >
> > >
> > >
> > > On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
> > > jdaugherty@jdresources.net> wrote:
> > >
> > >> Thank you Jochen. I believe I've switched to order stable collections
> > >> throughout the AST code base. Unfortunately, I'm still seeing issues.
> > >>
> > >> I've narrowed the differences down to the following:
> > >>
> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> > >>
> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> > >> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> > >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> > >>
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> > >>
> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> > >>
> > >>
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> > >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> > >>
> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
> > >>
> > >>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> > >>
> > >> Since the javadoc jars now make up 13 of the remaining 26 jar
> > >> differences, I was hoping someone can help here:
> > >>
> > >> The javadoc jars above are interesting because the source files &
> > >> compiled classes no longer differ for projects like `grails-core`.
> These
> > >> jars actually contain the groovydoc (for java + groovy classes). In
> the
> > >> case of `grails-core`, the example difference is the
> > >> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding()
> methods
> > >> are reordered.
> > >> Since this is groovydoc generating this, do you know if there are
> known
> > >> defined orders in groovydoc? This is from the gradle task, so I
> assume
> > >> this is being done with groovy 3's groovydoc. Is this something
> addressed
> > >> in groovydoc under groovy 4?
> > >>
> > >> The source file in question is rather simple. The methods being
> > >> reordered are from the parent class (which is in Groovy instead of
> > >> Grails). Here's the source:
> > >>
> > >> @CompileStatic
> > >> abstract class SettingsFile extends Script {
> > >>
> > >>
> > >> void include(String[] projectPaths) {
> > >> binding.setVariable("projectPaths", projectPaths)
> > >> }
> > >>
> > >> void includeFlat(String[] projectPaths) {
> > >> binding.setVariable("projectPaths", projectPaths)
> > >> }
> > >>
> > >> def methodMissing(String name, args) {
> > >> // ignore
> > >> }
> > >>
> > >> def propertyMissing(String name) {
> > >> // ignore
> > >> }
> > >> }
> > >>
> > >>
> > >>
> > >> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
> > >> <bl...@gmx.org.invalid> wrote:
> > >>
> > >>> On 24.04.25 23:06, James Daugherty wrote:
> > >>> [...]
> > >>> > I'm planning to continue researching this, but if anyone has
> experience
> > >>> > with the AST transforms or the ordering problem, it would be
> helpful to
> > >>> > solve this. This is probably the largest blocker to us proceeding
> > >>> with an
> > >>> > ASF release.
> > >>>
> > >>> as for the ordering. Groovy wants in general to use the order of
> > >>> declaration in the source code. What we are talking about are most
> > >>> likely methods added by transforms. And if it is one time stable and
> one
> > >>> time not, then I guess Maps are involved. ClassNode has several
> > >>> getDeclaredMethodsXY methods, which are based on a methods maps,
> while
> > >>> getMethods or getMethodsList is based on the methods list.
> > >>>
> > >>> If you then have something like
> > >>>
> > >>> classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
> > >>> {doSomethingWith(it)}
> > >>>
> > >>> and there is more than one method annotated, then the order might be
> > >>> broken between builds. Just to give one example.
> > >>>
> > >>>
> > >>> I would suggest to go through all transforms and check on the usage
> of
> > >>> the getDeclaredMethods methods and then avoid them to use for
> iteration.
> > >>> They are ok for lookup of course.
> > >>>
> > >>> bye Jochen
> > >>>
> > >>
> >
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Fredley <ja...@apache.org>.
I think we should proceed with updating the ASF security team on the progress so we receive updated feedback.
Since we are working on 7.0.0-SNAPSHOT, will most likely need Groovy 4.0.27 for Grails 7.0.0-M4 and already put the Apache Snapshot Repository in the right locations to pull Grails SNAPSHOTS and Groovy SNAPSHOTS, I'd be OK with Groovy 4.0.27-SNAPSHOT for the entire build for now and also OK just for documentation. We ran into issues running an earlier Groovy SNAPSHOT last year, but the root causes for those issues are addressed in grails-core.
I also think it would be fine to merge and then start a new PR once addition updates can be made.
On 2025/05/05 16:11:22 James Daugherty wrote:
> Paul was able to fix the groovydoc differences via groovy 4.0.27-SNAPSHOT.
> I've updated the ticket; we're down to these jars differing:
>
> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
>
> My latest PR is here: https://github.com/apache/grails-core/issues/14679 It
> uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead of just
> assigning 4.0.27 for documentation generation, I set it for the entire
> project default. This PR also fixes the comparison of the jars in the
> grails-gradle project & addresses some documentation issues (one example:
> grails data docs aren't fully being generated).
>
> Of these jars, the differences look due to fields reordering from traits
> (fields have annotations like @TraitBridge on them). I'm guessing there's
> some issue related to how groovy processes the traits, but this could be
> something we're doing (I haven't figured out the specifics, but
> ApplicationTagLib is the easiest case of this since none of our AST
> transforms appear to do anything to that file). I am under the impression
> that Paul is continuing to look at this issue to see if he can help or
> identify a groovy specific issue.
>
> My current dilemma is if I should merge this PR. I could scale it back to
> use groovy 4.0.26 for compiling & only use the snapshot for documentation
> generation. The problem is if this is a groovy issue, we'll need a groovy
> 4.0.27 release anyhow. What are people's thoughts here? Should I just
> leave this on a side branch for now? Should we depend entirely on the
> snapshot build? Should only documentation depend on the snapshot build?
>
> Alternatively, I think we can respond to ASF security that we can verify
> the build artifacts by using the same process we use to test reproducible
> builds. Namely, we can diff any artifact that has a different checksum via
> the decompiler in IntelliJ to confirm the code is the same. This solution
> isn't a permanent solution, but for the Milestone / RC builds, I think it's
> acceptable.
>
> -James
>
> On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <jd...@jdresources.net>
> wrote:
>
> > Hi Everyone,
> >
> > I summarized my progress thus far under this ticket:
> > https://github.com/apache/grails-core/issues/14679
> >
> > I checked in several scripts (see ticket) that assist in easily testing
> > the jar files for differences.
> >
> > From what I can tell, there is an ordering issue when multiple, inherited
> > traits are involved. I've found a simplified scenario in the grails-gsp
> > plugin (ApplicationTagLib) that doesn't appear to be transformed by our
> > code in a material way, and yet the difference always deals with the copied
> > trait methods being in a random order. I'm at a loss for how to further
> > debug this issue. From what I can tell, Grails isn't copying those
> > methods. I would assume Groovy is.
> >
> > The steps to reproduce this:
> > 1. etc/bin/test-reproducible-builds.sh
> > 2. etc/bin/extract-build-artifact.sh
> > grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> > 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
> > `etc/bin/results` and right click to select `Compare Directories`
> >
> > I intend to send an email to the groovy dev list to see if they can help
> > me understand how I can debug this further.
> >
> > If we can't resolve these issues fully in Grails 7, the good news is the
> > scripts I've checked in can be used to "verify" that the jars were built
> > correctly - we can build locally and manually diff the artifacts to confirm
> > they're correct. This is the work around I plan to suggest to ASF security.
> >
> > -James
> >
> >
> >
> >
> > On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
> > jdaugherty@jdresources.net> wrote:
> >
> >> Thank you Jochen. I believe I've switched to order stable collections
> >> throughout the AST code base. Unfortunately, I'm still seeing issues.
> >>
> >> I've narrowed the differences down to the following:
> >> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> >> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
> >>
> >> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
> >>
> >> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> >> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
> >>
> >> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
> >>
> >> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> >>
> >> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> >> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> >> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
> >>
> >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> >> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> >> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
> >>
> >> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
> >>
> >> Since the javadoc jars now make up 13 of the remaining 26 jar
> >> differences, I was hoping someone can help here:
> >>
> >> The javadoc jars above are interesting because the source files &
> >> compiled classes no longer differ for projects like `grails-core`. These
> >> jars actually contain the groovydoc (for java + groovy classes). In the
> >> case of `grails-core`, the example difference is the
> >> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding() methods
> >> are reordered.
> >> Since this is groovydoc generating this, do you know if there are known
> >> defined orders in groovydoc? This is from the gradle task, so I assume
> >> this is being done with groovy 3's groovydoc. Is this something addressed
> >> in groovydoc under groovy 4?
> >>
> >> The source file in question is rather simple. The methods being
> >> reordered are from the parent class (which is in Groovy instead of
> >> Grails). Here's the source:
> >>
> >> @CompileStatic
> >> abstract class SettingsFile extends Script {
> >>
> >>
> >> void include(String[] projectPaths) {
> >> binding.setVariable("projectPaths", projectPaths)
> >> }
> >>
> >> void includeFlat(String[] projectPaths) {
> >> binding.setVariable("projectPaths", projectPaths)
> >> }
> >>
> >> def methodMissing(String name, args) {
> >> // ignore
> >> }
> >>
> >> def propertyMissing(String name) {
> >> // ignore
> >> }
> >> }
> >>
> >>
> >>
> >> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
> >> <bl...@gmx.org.invalid> wrote:
> >>
> >>> On 24.04.25 23:06, James Daugherty wrote:
> >>> [...]
> >>> > I'm planning to continue researching this, but if anyone has experience
> >>> > with the AST transforms or the ordering problem, it would be helpful to
> >>> > solve this. This is probably the largest blocker to us proceeding
> >>> with an
> >>> > ASF release.
> >>>
> >>> as for the ordering. Groovy wants in general to use the order of
> >>> declaration in the source code. What we are talking about are most
> >>> likely methods added by transforms. And if it is one time stable and one
> >>> time not, then I guess Maps are involved. ClassNode has several
> >>> getDeclaredMethodsXY methods, which are based on a methods maps, while
> >>> getMethods or getMethodsList is based on the methods list.
> >>>
> >>> If you then have something like
> >>>
> >>> classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
> >>> {doSomethingWith(it)}
> >>>
> >>> and there is more than one method annotated, then the order might be
> >>> broken between builds. Just to give one example.
> >>>
> >>>
> >>> I would suggest to go through all transforms and check on the usage of
> >>> the getDeclaredMethods methods and then avoid them to use for iteration.
> >>> They are ok for lookup of course.
> >>>
> >>> bye Jochen
> >>>
> >>
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
Paul was able to fix the groovydoc differences via groovy 4.0.27-SNAPSHOT.
I've updated the ticket; we're down to these jars differing:
grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
grails-datamapping-tck/build/libs/grails-datamapping-tck-7.0.0-SNAPSHOT.jar
grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
grails-wrapper-impl/build/libs/grails-wrapper-impl-7.0.0-SNAPSHOT.jar
My latest PR is here: https://github.com/apache/grails-core/issues/14679 It
uses the 4.0.27 groovy snasphot to fix the javadoc issue. Instead of just
assigning 4.0.27 for documentation generation, I set it for the entire
project default. This PR also fixes the comparison of the jars in the
grails-gradle project & addresses some documentation issues (one example:
grails data docs aren't fully being generated).
Of these jars, the differences look due to fields reordering from traits
(fields have annotations like @TraitBridge on them). I'm guessing there's
some issue related to how groovy processes the traits, but this could be
something we're doing (I haven't figured out the specifics, but
ApplicationTagLib is the easiest case of this since none of our AST
transforms appear to do anything to that file). I am under the impression
that Paul is continuing to look at this issue to see if he can help or
identify a groovy specific issue.
My current dilemma is if I should merge this PR. I could scale it back to
use groovy 4.0.26 for compiling & only use the snapshot for documentation
generation. The problem is if this is a groovy issue, we'll need a groovy
4.0.27 release anyhow. What are people's thoughts here? Should I just
leave this on a side branch for now? Should we depend entirely on the
snapshot build? Should only documentation depend on the snapshot build?
Alternatively, I think we can respond to ASF security that we can verify
the build artifacts by using the same process we use to test reproducible
builds. Namely, we can diff any artifact that has a different checksum via
the decompiler in IntelliJ to confirm the code is the same. This solution
isn't a permanent solution, but for the Milestone / RC builds, I think it's
acceptable.
-James
On Mon, Apr 28, 2025 at 10:15 PM James Daugherty <jd...@jdresources.net>
wrote:
> Hi Everyone,
>
> I summarized my progress thus far under this ticket:
> https://github.com/apache/grails-core/issues/14679
>
> I checked in several scripts (see ticket) that assist in easily testing
> the jar files for differences.
>
> From what I can tell, there is an ordering issue when multiple, inherited
> traits are involved. I've found a simplified scenario in the grails-gsp
> plugin (ApplicationTagLib) that doesn't appear to be transformed by our
> code in a material way, and yet the difference always deals with the copied
> trait methods being in a random order. I'm at a loss for how to further
> debug this issue. From what I can tell, Grails isn't copying those
> methods. I would assume Groovy is.
>
> The steps to reproduce this:
> 1. etc/bin/test-reproducible-builds.sh
> 2. etc/bin/extract-build-artifact.sh
> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> 3. select `firstArtifact` and `secondArtifact` in IntelliJ under
> `etc/bin/results` and right click to select `Compare Directories`
>
> I intend to send an email to the groovy dev list to see if they can help
> me understand how I can debug this further.
>
> If we can't resolve these issues fully in Grails 7, the good news is the
> scripts I've checked in can be used to "verify" that the jars were built
> correctly - we can build locally and manually diff the artifacts to confirm
> they're correct. This is the work around I plan to suggest to ASF security.
>
> -James
>
>
>
>
> On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <
> jdaugherty@jdresources.net> wrote:
>
>> Thank you Jochen. I believe I've switched to order stable collections
>> throughout the AST code base. Unfortunately, I'm still seeing issues.
>>
>> I've narrowed the differences down to the following:
>> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
>> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
>> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
>> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
>> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
>>
>> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
>>
>> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
>> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
>> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
>> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
>>
>> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
>>
>> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
>> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
>>
>> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
>> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
>> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
>> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
>> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
>> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
>> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
>>
>> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
>> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
>> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
>>
>> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
>>
>> Since the javadoc jars now make up 13 of the remaining 26 jar
>> differences, I was hoping someone can help here:
>>
>> The javadoc jars above are interesting because the source files &
>> compiled classes no longer differ for projects like `grails-core`. These
>> jars actually contain the groovydoc (for java + groovy classes). In the
>> case of `grails-core`, the example difference is the
>> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding() methods
>> are reordered.
>> Since this is groovydoc generating this, do you know if there are known
>> defined orders in groovydoc? This is from the gradle task, so I assume
>> this is being done with groovy 3's groovydoc. Is this something addressed
>> in groovydoc under groovy 4?
>>
>> The source file in question is rather simple. The methods being
>> reordered are from the parent class (which is in Groovy instead of
>> Grails). Here's the source:
>>
>> @CompileStatic
>> abstract class SettingsFile extends Script {
>>
>>
>> void include(String[] projectPaths) {
>> binding.setVariable("projectPaths", projectPaths)
>> }
>>
>> void includeFlat(String[] projectPaths) {
>> binding.setVariable("projectPaths", projectPaths)
>> }
>>
>> def methodMissing(String name, args) {
>> // ignore
>> }
>>
>> def propertyMissing(String name) {
>> // ignore
>> }
>> }
>>
>>
>>
>> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou
>> <bl...@gmx.org.invalid> wrote:
>>
>>> On 24.04.25 23:06, James Daugherty wrote:
>>> [...]
>>> > I'm planning to continue researching this, but if anyone has experience
>>> > with the AST transforms or the ordering problem, it would be helpful to
>>> > solve this. This is probably the largest blocker to us proceeding
>>> with an
>>> > ASF release.
>>>
>>> as for the ordering. Groovy wants in general to use the order of
>>> declaration in the source code. What we are talking about are most
>>> likely methods added by transforms. And if it is one time stable and one
>>> time not, then I guess Maps are involved. ClassNode has several
>>> getDeclaredMethodsXY methods, which are based on a methods maps, while
>>> getMethods or getMethodsList is based on the methods list.
>>>
>>> If you then have something like
>>>
>>> classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
>>> {doSomethingWith(it)}
>>>
>>> and there is more than one method annotated, then the order might be
>>> broken between builds. Just to give one example.
>>>
>>>
>>> I would suggest to go through all transforms and check on the usage of
>>> the getDeclaredMethods methods and then avoid them to use for iteration.
>>> They are ok for lookup of course.
>>>
>>> bye Jochen
>>>
>>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
Hi Everyone,
I summarized my progress thus far under this ticket:
https://github.com/apache/grails-core/issues/14679
I checked in several scripts (see ticket) that assist in easily testing the
jar files for differences.
From what I can tell, there is an ordering issue when multiple, inherited
traits are involved. I've found a simplified scenario in the grails-gsp
plugin (ApplicationTagLib) that doesn't appear to be transformed by our
code in a material way, and yet the difference always deals with the copied
trait methods being in a random order. I'm at a loss for how to further
debug this issue. From what I can tell, Grails isn't copying those
methods. I would assume Groovy is.
The steps to reproduce this:
1. etc/bin/test-reproducible-builds.sh
2. etc/bin/extract-build-artifact.sh
grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
3. select `firstArtifact` and `secondArtifact` in IntelliJ under
`etc/bin/results` and right click to select `Compare Directories`
I intend to send an email to the groovy dev list to see if they can help me
understand how I can debug this further.
If we can't resolve these issues fully in Grails 7, the good news is the
scripts I've checked in can be used to "verify" that the jars were built
correctly - we can build locally and manually diff the artifacts to confirm
they're correct. This is the work around I plan to suggest to ASF security.
-James
On Mon, Apr 28, 2025 at 11:58 AM James Daugherty <jd...@jdresources.net>
wrote:
> Thank you Jochen. I believe I've switched to order stable collections
> throughout the AST code base. Unfortunately, I'm still seeing issues.
>
> I've narrowed the differences down to the following:
> grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
> grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
> grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
> grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
> grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
>
> grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
>
> grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
> grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
> grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
> grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
>
> grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
>
> grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
> grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
> grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
> grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
> grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
> grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
> grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
> grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
>
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
> grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
> grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
>
> grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
>
> Since the javadoc jars now make up 13 of the remaining 26 jar differences,
> I was hoping someone can help here:
>
> The javadoc jars above are interesting because the source files & compiled
> classes no longer differ for projects like `grails-core`. These jars
> actually contain the groovydoc (for java + groovy classes). In the case of
> `grails-core`, the example difference is the
> groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding() methods
> are reordered.
> Since this is groovydoc generating this, do you know if there are known
> defined orders in groovydoc? This is from the gradle task, so I assume
> this is being done with groovy 3's groovydoc. Is this something addressed
> in groovydoc under groovy 4?
>
> The source file in question is rather simple. The methods being reordered
> are from the parent class (which is in Groovy instead of Grails). Here's
> the source:
>
> @CompileStatic
> abstract class SettingsFile extends Script {
>
>
> void include(String[] projectPaths) {
> binding.setVariable("projectPaths", projectPaths)
> }
>
> void includeFlat(String[] projectPaths) {
> binding.setVariable("projectPaths", projectPaths)
> }
>
> def methodMissing(String name, args) {
> // ignore
> }
>
> def propertyMissing(String name) {
> // ignore
> }
> }
>
>
>
> On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou <bl...@gmx.org.invalid>
> wrote:
>
>> On 24.04.25 23:06, James Daugherty wrote:
>> [...]
>> > I'm planning to continue researching this, but if anyone has experience
>> > with the AST transforms or the ordering problem, it would be helpful to
>> > solve this. This is probably the largest blocker to us proceeding with
>> an
>> > ASF release.
>>
>> as for the ordering. Groovy wants in general to use the order of
>> declaration in the source code. What we are talking about are most
>> likely methods added by transforms. And if it is one time stable and one
>> time not, then I guess Maps are involved. ClassNode has several
>> getDeclaredMethodsXY methods, which are based on a methods maps, while
>> getMethods or getMethodsList is based on the methods list.
>>
>> If you then have something like
>>
>> classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
>> {doSomethingWith(it)}
>>
>> and there is more than one method annotated, then the order might be
>> broken between builds. Just to give one example.
>>
>>
>> I would suggest to go through all transforms and check on the usage of
>> the getDeclaredMethods methods and then avoid them to use for iteration.
>> They are ok for lookup of course.
>>
>> bye Jochen
>>
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
Thank you Jochen. I believe I've switched to order stable collections
throughout the AST code base. Unfortunately, I'm still seeing issues.
I've narrowed the differences down to the following:
grails-async/core/build/libs/grails-async-core-7.0.0-SNAPSHOT-javadoc.jar
grails-bootstrap/build/libs/grails-bootstrap-7.0.0-SNAPSHOT-javadoc.jar
grails-cache/build/libs/grails-cache-7.0.0-SNAPSHOT.jar
grails-converters/build/libs/grails-converters-7.0.0-SNAPSHOT-javadoc.jar
grails-core/build/libs/grails-core-7.0.0-SNAPSHOT-javadoc.jar
grails-data-hibernate5/core/build/libs/grails-data-hibernate5-core-7.0.0-SNAPSHOT-javadoc.jar
grails-data-hibernate5/dbmigration/build/libs/grails-data-hibernate5-dbmigration-7.0.0-SNAPSHOT-javadoc.jar
grails-datamapping-core/build/libs/grails-datamapping-core-7.0.0-SNAPSHOT-javadoc.jar
grails-datamapping-tck-domains/build/libs/grails-datamapping-tck-domains-7.0.0-SNAPSHOT.jar
grails-datamapping-tck-tests/build/libs/grails-datamapping-tck-tests-7.0.0-SNAPSHOT.jar
grails-datamapping-validation/build/libs/grails-datamapping-validation-7.0.0-SNAPSHOT-javadoc.jar
grails-datastore-core/build/libs/grails-datastore-core-7.0.0-SNAPSHOT-javadoc.jar
grails-fields/build/libs/grails-fields-7.0.0-SNAPSHOT.jar
grails-gsp/core/build/libs/grails-gsp-core-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/grails-sitemesh3/build/libs/grails-sitemesh3-7.0.0-SNAPSHOT.jar
grails-gsp/grails-taglib/build/libs/grails-taglib-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/grails-web-gsp-taglib/build/libs/grails-web-gsp-taglib-7.0.0-SNAPSHOT.jar
grails-gsp/grails-web-gsp/build/libs/grails-web-gsp-7.0.0-SNAPSHOT-javadoc.jar
grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar
grails-rest-transforms/build/libs/grails-rest-transforms-7.0.0-SNAPSHOT.jar
grails-scaffolding/build/libs/grails-scaffolding-7.0.0-SNAPSHOT.jar
grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT-javadoc.jar
grails-shell-cli/build/libs/grails-shell-cli-7.0.0-SNAPSHOT.jar
grails-test-core/build/libs/grails-test-core-7.0.0-SNAPSHOT-javadoc.jar
grails-views-core/build/libs/grails-views-core-7.0.0-SNAPSHOT-javadoc.jar
grails-views-gson/build/libs/grails-views-gson-7.0.0-SNAPSHOT.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT-javadoc.jar
grails-views-markup/build/libs/grails-views-markup-7.0.0-SNAPSHOT.jar
grails-web-common/build/libs/grails-web-common-7.0.0-SNAPSHOT-javadoc.jar
grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT-javadoc.jar
grails-web-url-mappings/build/libs/grails-web-url-mappings-7.0.0-SNAPSHOT.jar
Since the javadoc jars now make up 13 of the remaining 26 jar differences,
I was hoping someone can help here:
The javadoc jars above are interesting because the source files & compiled
classes no longer differ for projects like `grails-core`. These jars
actually contain the groovydoc (for java + groovy classes). In the case of
`grails-core`, the example difference is the
groovy.lang.Script#getBinding() and groovy.lang.Script#setBinding() methods
are reordered.
Since this is groovydoc generating this, do you know if there are known
defined orders in groovydoc? This is from the gradle task, so I assume
this is being done with groovy 3's groovydoc. Is this something addressed
in groovydoc under groovy 4?
The source file in question is rather simple. The methods being reordered
are from the parent class (which is in Groovy instead of Grails). Here's
the source:
@CompileStatic
abstract class SettingsFile extends Script {
void include(String[] projectPaths) {
binding.setVariable("projectPaths", projectPaths)
}
void includeFlat(String[] projectPaths) {
binding.setVariable("projectPaths", projectPaths)
}
def methodMissing(String name, args) {
// ignore
}
def propertyMissing(String name) {
// ignore
}
}
On Sat, Apr 26, 2025 at 5:32 AM Jochen Theodorou <bl...@gmx.org.invalid>
wrote:
> On 24.04.25 23:06, James Daugherty wrote:
> [...]
> > I'm planning to continue researching this, but if anyone has experience
> > with the AST transforms or the ordering problem, it would be helpful to
> > solve this. This is probably the largest blocker to us proceeding with
> an
> > ASF release.
>
> as for the ordering. Groovy wants in general to use the order of
> declaration in the source code. What we are talking about are most
> likely methods added by transforms. And if it is one time stable and one
> time not, then I guess Maps are involved. ClassNode has several
> getDeclaredMethodsXY methods, which are based on a methods maps, while
> getMethods or getMethodsList is based on the methods list.
>
> If you then have something like
>
> classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
> {doSomethingWith(it)}
>
> and there is more than one method annotated, then the order might be
> broken between builds. Just to give one example.
>
>
> I would suggest to go through all transforms and check on the usage of
> the getDeclaredMethods methods and then avoid them to use for iteration.
> They are ok for lookup of course.
>
> bye Jochen
>
Re: [DISCUSS][HELP] Reproducible Builds
Posted by Jochen Theodorou <bl...@gmx.org.INVALID>.
On 24.04.25 23:06, James Daugherty wrote:
[...]
> I'm planning to continue researching this, but if anyone has experience
> with the AST transforms or the ordering problem, it would be helpful to
> solve this. This is probably the largest blocker to us proceeding with an
> ASF release.
as for the ordering. Groovy wants in general to use the order of
declaration in the source code. What we are talking about are most
likely methods added by transforms. And if it is one time stable and one
time not, then I guess Maps are involved. ClassNode has several
getDeclaredMethodsXY methods, which are based on a methods maps, while
getMethods or getMethodsList is based on the methods list.
If you then have something like
classNode.getAllDeclaredMethods().findAll(it annnotated with XY).each
{doSomethingWith(it)}
and there is more than one method annotated, then the order might be
broken between builds. Just to give one example.
I would suggest to go through all transforms and check on the usage of
the getDeclaredMethods methods and then avoid them to use for iteration.
They are ok for lookup of course.
bye Jochen
Today's (5/14/2025) Weekly Apache Grails Planning Meeting Reminder
Posted by James Fredley <ja...@apache.org>.
This week is the later time week.
Google Meet Link is always: https://meet.google.com/her-tjpt-xmf
Notes: https://docs.google.com/document/d/1L590r6Z_xNQ48l31Yolsir75dUgAorEnTWmW1kGqJrw/edit?tab=t.0#heading=h.yzvjivh6d3yb
Meeting Time:
This week: https://www.worldtimebuddy.com/?qm=1&lid=8,6,5,12,30,1819729,2207259&h=8&date=2025-5-14&sln=14-15&hf=1
Next week: https://www.worldtimebuddy.com/?qm=1&lid=8,6,5,12,30,1819729,2207259&h=8&date=2025-5-21&sln=7-8&hf=1
Today's Weekly Apache Grails Planning Meeting Reminder
Posted by James Fredley <ja...@apache.org>.
This week is the early week.
Google Meet Link is always: https://meet.google.com/her-tjpt-xmf
Notes: https://docs.google.com/document/d/1L590r6Z_xNQ48l31Yolsir75dUgAorEnTWmW1kGqJrw/edit?tab=t.0#heading=h.yzvjivh6d3yb
Meeting Time:
This week: https://www.worldtimebuddy.com/?qm=1&lid=8,6,5,12,30,1819729,2207259&h=8&date=2025-5-7&sln=7-8&hf=1
Next week: https://www.worldtimebuddy.com/?qm=1&lid=8,6,5,12,30,1819729,2207259&h=8&date=2025-5-14&sln=14-15&hf=1
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
The largest benefit: if repo.grails.org gets removed tomorrow, we know what
will break and why. Today, that repo points to so many things that I don't
think we have a good understanding of what's required for various grails
applications.
The second benefit: I know of several old libraries that are accessible in
that repo (ones that went through licensing changes that aren't on maven
central, i.e. iText) and we need to make sure people aren't still resolving
them as part of a plugin or a secondary dependency. Forcing a change to a
new location forces people to update older libraries. Grails 7 seems to be
the best time to do that.
-James
On Sat, May 3, 2025 at 5:58 PM Mattias Reichel <ma...@gmail.com>
wrote:
> We could do that, but I don't think I fully understand the benefit of doing
> so.
> What are the implications of old artifacts being resolvable?
> I guess a fresh start for Grails 7+ artifacts would not hurt given the age
> and state of that repository.
>
> Den lör 3 maj 2025 kl 14:47 skrev James Daugherty
> <jd...@jdresources.net.invalid>:
>
> > @Mattias, what are your thoughts about creating a new repository on
> > repo.grails.org to isolate legacy artifacts?
> >
> > On Sat, May 3, 2025 at 7:52 AM Mattias Reichel <
> mattias.reichel@gmail.com>
> > wrote:
> >
> > > I think we should keep using repo.grails.org for now (below the other
> > > repos, in the repositories block), and await JFrogs answer, if we can
> > keep
> > > using it.
> > >
> > > /Mattias
> > >
> > > Den lör 3 maj 2025 kl 10:42 skrev Gianluca Sartori <
> g.sartori@gmail.com
> > >:
> > >
> > > > Hi James & James,
> > > >
> > > > I agree we should move to repo.gradle.org/gradle/libs-releases, for
> > all
> > > > the
> > > > technical reasons you've provided, but also to make Grails more
> future
> > > > proof and communicate it as more "integrated" with relevant
> > technologies
> > > > (if that makes any sense, it does in my mind).
> > > >
> > > > Gianluca Sartori
> > > > --
> > > > https://dueuno.com
> > > >
> > > >
> > > > On Sat, 3 May 2025 at 01:23, James Daugherty
> > > > <jd...@jdresources.net.invalid> wrote:
> > > >
> > > > > I would like to see us try to exclude repo.grails.org/grails/core
> > from
> > > > our
> > > > > default applications if at all possible.
> > > > >
> > > > > There are several reasons:
> > > > >
> > > > > 1. The age of that repo: it has so many dependencies in it.
> > > > > 2. It has both snapshot & release artifacts (I realize that the
> > > snapshot
> > > > > versions aren't supposed to be chosen, but people make mistakes and
> > it
> > > > > happens).
> > > > > 3. We're relying on Cloudflare to continue to give us "free"
> services
> > > for
> > > > > CDN / certificate management. By adopting more public repos, we
> can
> > > > lessen
> > > > > this burden in case it's switched off. (I believe it's traffic
> sits
> > > > around
> > > > > 1.5TB/month currently).
> > > > > 4. It's an external resource from Apache.
> > > > >
> > > > > I realize this goal is easier said than done, but I know this repo
> is
> > > > also
> > > > > currently required for:
> > > > >
> > > > > 1. grails-plugins github org
> > > > > 2. gradle tooling api
> > > > > 3. gpc github org
> > > > > 4. legacy artifacts - some of which have since had a license change
> > to
> > > > what
> > > > > some consider unacceptable (so this is the only source)
> > > > >
> > > > > I believe all of these can be worked around. Here are my thoughts
> on
> > > > each:
> > > > > 1. I'm happy to do the work to start publishing to central.
> > > > > 2. You have given us 2 possible solutions. I lean towards just
> > adding
> > > > the
> > > > > repo.gradle.org/gradle/libs-releases location. Alternatively, we
> > can
> > > > > create a dedicated repo on repo.grails.org so that it doesn't
> > include
> > > > the
> > > > > legacy artifacts. I'm less a fan of Netbeans since they're
> > repackaging
> > > > it
> > > > > and it could technically be different. We also have to wait for
> them
> > > to
> > > > > upgrade gradle (possibly).
> > > > > 3. Like #1, I can get GPC publishing to central snapshots or we can
> > > > publish
> > > > > them to their github repos and people can add as they need.
> > > > > 4. This is really my #1 concern, but given that 7 has upgraded just
> > > about
> > > > > every version, I don't see a need for this. If people are still
> > using
> > > > > legacy artifacts, we need to know. This is more of a reason to not
> > use
> > > > it.
> > > > >
> > > > > It would be nice to also include snapshot repos only on snapshot
> > > builds,
> > > > > while using release only repos for official releases. Especially
> now
> > > > that
> > > > > we've gotten rid of the pre-release workflow.
> > > > >
> > > > > -James
> > > > >
> > > > >
> > > > >
> > > > > On Fri, May 2, 2025 at 12:54 PM James Fredley <
> > jamesfredley@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Previously we solved for the org.gradle:gradle-tooling-api
> > dependency
> > > > by
> > > > > > adding a remote repo on repo.grails.org so that it is available
> > via
> > > > > >
> > > https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
> > > > > >
> > > > > > org.gradle:gradle-tooling-api is currently a dependency on
> > > > > > grails-shell-cli only
> > > > > >
> > > > > > As we move the main projects away from repo.grails.org, I think
> we
> > > > will
> > > > > > want to solve this another way. This might be more of a Grails
> > 7.1.0
> > > > or
> > > > > > 8.0.0 decision.
> > > > > >
> > > > > > 1. Add an additional repository to the grails-core projects and
> end
> > > > > > generated projects
> > > > > > https://repo.gradle.org/gradle/libs-releases/
> > > > > >
> > > > > > 2. Use the republished versions from NetBeans
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by Mattias Reichel <ma...@gmail.com>.
We could do that, but I don't think I fully understand the benefit of doing
so.
What are the implications of old artifacts being resolvable?
I guess a fresh start for Grails 7+ artifacts would not hurt given the age
and state of that repository.
Den lör 3 maj 2025 kl 14:47 skrev James Daugherty
<jd...@jdresources.net.invalid>:
> @Mattias, what are your thoughts about creating a new repository on
> repo.grails.org to isolate legacy artifacts?
>
> On Sat, May 3, 2025 at 7:52 AM Mattias Reichel <ma...@gmail.com>
> wrote:
>
> > I think we should keep using repo.grails.org for now (below the other
> > repos, in the repositories block), and await JFrogs answer, if we can
> keep
> > using it.
> >
> > /Mattias
> >
> > Den lör 3 maj 2025 kl 10:42 skrev Gianluca Sartori <g.sartori@gmail.com
> >:
> >
> > > Hi James & James,
> > >
> > > I agree we should move to repo.gradle.org/gradle/libs-releases, for
> all
> > > the
> > > technical reasons you've provided, but also to make Grails more future
> > > proof and communicate it as more "integrated" with relevant
> technologies
> > > (if that makes any sense, it does in my mind).
> > >
> > > Gianluca Sartori
> > > --
> > > https://dueuno.com
> > >
> > >
> > > On Sat, 3 May 2025 at 01:23, James Daugherty
> > > <jd...@jdresources.net.invalid> wrote:
> > >
> > > > I would like to see us try to exclude repo.grails.org/grails/core
> from
> > > our
> > > > default applications if at all possible.
> > > >
> > > > There are several reasons:
> > > >
> > > > 1. The age of that repo: it has so many dependencies in it.
> > > > 2. It has both snapshot & release artifacts (I realize that the
> > snapshot
> > > > versions aren't supposed to be chosen, but people make mistakes and
> it
> > > > happens).
> > > > 3. We're relying on Cloudflare to continue to give us "free" services
> > for
> > > > CDN / certificate management. By adopting more public repos, we can
> > > lessen
> > > > this burden in case it's switched off. (I believe it's traffic sits
> > > around
> > > > 1.5TB/month currently).
> > > > 4. It's an external resource from Apache.
> > > >
> > > > I realize this goal is easier said than done, but I know this repo is
> > > also
> > > > currently required for:
> > > >
> > > > 1. grails-plugins github org
> > > > 2. gradle tooling api
> > > > 3. gpc github org
> > > > 4. legacy artifacts - some of which have since had a license change
> to
> > > what
> > > > some consider unacceptable (so this is the only source)
> > > >
> > > > I believe all of these can be worked around. Here are my thoughts on
> > > each:
> > > > 1. I'm happy to do the work to start publishing to central.
> > > > 2. You have given us 2 possible solutions. I lean towards just
> adding
> > > the
> > > > repo.gradle.org/gradle/libs-releases location. Alternatively, we
> can
> > > > create a dedicated repo on repo.grails.org so that it doesn't
> include
> > > the
> > > > legacy artifacts. I'm less a fan of Netbeans since they're
> repackaging
> > > it
> > > > and it could technically be different. We also have to wait for them
> > to
> > > > upgrade gradle (possibly).
> > > > 3. Like #1, I can get GPC publishing to central snapshots or we can
> > > publish
> > > > them to their github repos and people can add as they need.
> > > > 4. This is really my #1 concern, but given that 7 has upgraded just
> > about
> > > > every version, I don't see a need for this. If people are still
> using
> > > > legacy artifacts, we need to know. This is more of a reason to not
> use
> > > it.
> > > >
> > > > It would be nice to also include snapshot repos only on snapshot
> > builds,
> > > > while using release only repos for official releases. Especially now
> > > that
> > > > we've gotten rid of the pre-release workflow.
> > > >
> > > > -James
> > > >
> > > >
> > > >
> > > > On Fri, May 2, 2025 at 12:54 PM James Fredley <
> jamesfredley@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Previously we solved for the org.gradle:gradle-tooling-api
> dependency
> > > by
> > > > > adding a remote repo on repo.grails.org so that it is available
> via
> > > > >
> > https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
> > > > >
> > > > > org.gradle:gradle-tooling-api is currently a dependency on
> > > > > grails-shell-cli only
> > > > >
> > > > > As we move the main projects away from repo.grails.org, I think we
> > > will
> > > > > want to solve this another way. This might be more of a Grails
> 7.1.0
> > > or
> > > > > 8.0.0 decision.
> > > > >
> > > > > 1. Add an additional repository to the grails-core projects and end
> > > > > generated projects
> > > > > https://repo.gradle.org/gradle/libs-releases/
> > > > >
> > > > > 2. Use the republished versions from NetBeans
> > > > >
> > > > >
> > > >
> > >
> >
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
@Mattias, what are your thoughts about creating a new repository on
repo.grails.org to isolate legacy artifacts?
On Sat, May 3, 2025 at 7:52 AM Mattias Reichel <ma...@gmail.com>
wrote:
> I think we should keep using repo.grails.org for now (below the other
> repos, in the repositories block), and await JFrogs answer, if we can keep
> using it.
>
> /Mattias
>
> Den lör 3 maj 2025 kl 10:42 skrev Gianluca Sartori <g....@gmail.com>:
>
> > Hi James & James,
> >
> > I agree we should move to repo.gradle.org/gradle/libs-releases, for all
> > the
> > technical reasons you've provided, but also to make Grails more future
> > proof and communicate it as more "integrated" with relevant technologies
> > (if that makes any sense, it does in my mind).
> >
> > Gianluca Sartori
> > --
> > https://dueuno.com
> >
> >
> > On Sat, 3 May 2025 at 01:23, James Daugherty
> > <jd...@jdresources.net.invalid> wrote:
> >
> > > I would like to see us try to exclude repo.grails.org/grails/core from
> > our
> > > default applications if at all possible.
> > >
> > > There are several reasons:
> > >
> > > 1. The age of that repo: it has so many dependencies in it.
> > > 2. It has both snapshot & release artifacts (I realize that the
> snapshot
> > > versions aren't supposed to be chosen, but people make mistakes and it
> > > happens).
> > > 3. We're relying on Cloudflare to continue to give us "free" services
> for
> > > CDN / certificate management. By adopting more public repos, we can
> > lessen
> > > this burden in case it's switched off. (I believe it's traffic sits
> > around
> > > 1.5TB/month currently).
> > > 4. It's an external resource from Apache.
> > >
> > > I realize this goal is easier said than done, but I know this repo is
> > also
> > > currently required for:
> > >
> > > 1. grails-plugins github org
> > > 2. gradle tooling api
> > > 3. gpc github org
> > > 4. legacy artifacts - some of which have since had a license change to
> > what
> > > some consider unacceptable (so this is the only source)
> > >
> > > I believe all of these can be worked around. Here are my thoughts on
> > each:
> > > 1. I'm happy to do the work to start publishing to central.
> > > 2. You have given us 2 possible solutions. I lean towards just adding
> > the
> > > repo.gradle.org/gradle/libs-releases location. Alternatively, we can
> > > create a dedicated repo on repo.grails.org so that it doesn't include
> > the
> > > legacy artifacts. I'm less a fan of Netbeans since they're repackaging
> > it
> > > and it could technically be different. We also have to wait for them
> to
> > > upgrade gradle (possibly).
> > > 3. Like #1, I can get GPC publishing to central snapshots or we can
> > publish
> > > them to their github repos and people can add as they need.
> > > 4. This is really my #1 concern, but given that 7 has upgraded just
> about
> > > every version, I don't see a need for this. If people are still using
> > > legacy artifacts, we need to know. This is more of a reason to not use
> > it.
> > >
> > > It would be nice to also include snapshot repos only on snapshot
> builds,
> > > while using release only repos for official releases. Especially now
> > that
> > > we've gotten rid of the pre-release workflow.
> > >
> > > -James
> > >
> > >
> > >
> > > On Fri, May 2, 2025 at 12:54 PM James Fredley <jamesfredley@apache.org
> >
> > > wrote:
> > >
> > > > Previously we solved for the org.gradle:gradle-tooling-api dependency
> > by
> > > > adding a remote repo on repo.grails.org so that it is available via
> > > >
> https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
> > > >
> > > > org.gradle:gradle-tooling-api is currently a dependency on
> > > > grails-shell-cli only
> > > >
> > > > As we move the main projects away from repo.grails.org, I think we
> > will
> > > > want to solve this another way. This might be more of a Grails 7.1.0
> > or
> > > > 8.0.0 decision.
> > > >
> > > > 1. Add an additional repository to the grails-core projects and end
> > > > generated projects
> > > > https://repo.gradle.org/gradle/libs-releases/
> > > >
> > > > 2. Use the republished versions from NetBeans
> > > >
> > > >
> > >
> >
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by Mattias Reichel <ma...@gmail.com>.
I think we should keep using repo.grails.org for now (below the other
repos, in the repositories block), and await JFrogs answer, if we can keep
using it.
/Mattias
Den lör 3 maj 2025 kl 10:42 skrev Gianluca Sartori <g....@gmail.com>:
> Hi James & James,
>
> I agree we should move to repo.gradle.org/gradle/libs-releases, for all
> the
> technical reasons you've provided, but also to make Grails more future
> proof and communicate it as more "integrated" with relevant technologies
> (if that makes any sense, it does in my mind).
>
> Gianluca Sartori
> --
> https://dueuno.com
>
>
> On Sat, 3 May 2025 at 01:23, James Daugherty
> <jd...@jdresources.net.invalid> wrote:
>
> > I would like to see us try to exclude repo.grails.org/grails/core from
> our
> > default applications if at all possible.
> >
> > There are several reasons:
> >
> > 1. The age of that repo: it has so many dependencies in it.
> > 2. It has both snapshot & release artifacts (I realize that the snapshot
> > versions aren't supposed to be chosen, but people make mistakes and it
> > happens).
> > 3. We're relying on Cloudflare to continue to give us "free" services for
> > CDN / certificate management. By adopting more public repos, we can
> lessen
> > this burden in case it's switched off. (I believe it's traffic sits
> around
> > 1.5TB/month currently).
> > 4. It's an external resource from Apache.
> >
> > I realize this goal is easier said than done, but I know this repo is
> also
> > currently required for:
> >
> > 1. grails-plugins github org
> > 2. gradle tooling api
> > 3. gpc github org
> > 4. legacy artifacts - some of which have since had a license change to
> what
> > some consider unacceptable (so this is the only source)
> >
> > I believe all of these can be worked around. Here are my thoughts on
> each:
> > 1. I'm happy to do the work to start publishing to central.
> > 2. You have given us 2 possible solutions. I lean towards just adding
> the
> > repo.gradle.org/gradle/libs-releases location. Alternatively, we can
> > create a dedicated repo on repo.grails.org so that it doesn't include
> the
> > legacy artifacts. I'm less a fan of Netbeans since they're repackaging
> it
> > and it could technically be different. We also have to wait for them to
> > upgrade gradle (possibly).
> > 3. Like #1, I can get GPC publishing to central snapshots or we can
> publish
> > them to their github repos and people can add as they need.
> > 4. This is really my #1 concern, but given that 7 has upgraded just about
> > every version, I don't see a need for this. If people are still using
> > legacy artifacts, we need to know. This is more of a reason to not use
> it.
> >
> > It would be nice to also include snapshot repos only on snapshot builds,
> > while using release only repos for official releases. Especially now
> that
> > we've gotten rid of the pre-release workflow.
> >
> > -James
> >
> >
> >
> > On Fri, May 2, 2025 at 12:54 PM James Fredley <ja...@apache.org>
> > wrote:
> >
> > > Previously we solved for the org.gradle:gradle-tooling-api dependency
> by
> > > adding a remote repo on repo.grails.org so that it is available via
> > > https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
> > >
> > > org.gradle:gradle-tooling-api is currently a dependency on
> > > grails-shell-cli only
> > >
> > > As we move the main projects away from repo.grails.org, I think we
> will
> > > want to solve this another way. This might be more of a Grails 7.1.0
> or
> > > 8.0.0 decision.
> > >
> > > 1. Add an additional repository to the grails-core projects and end
> > > generated projects
> > > https://repo.gradle.org/gradle/libs-releases/
> > >
> > > 2. Use the republished versions from NetBeans
> > >
> > >
> >
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by Gianluca Sartori <g....@gmail.com>.
Hi James & James,
I agree we should move to repo.gradle.org/gradle/libs-releases, for all the
technical reasons you've provided, but also to make Grails more future
proof and communicate it as more "integrated" with relevant technologies
(if that makes any sense, it does in my mind).
Gianluca Sartori
--
https://dueuno.com
On Sat, 3 May 2025 at 01:23, James Daugherty
<jd...@jdresources.net.invalid> wrote:
> I would like to see us try to exclude repo.grails.org/grails/core from our
> default applications if at all possible.
>
> There are several reasons:
>
> 1. The age of that repo: it has so many dependencies in it.
> 2. It has both snapshot & release artifacts (I realize that the snapshot
> versions aren't supposed to be chosen, but people make mistakes and it
> happens).
> 3. We're relying on Cloudflare to continue to give us "free" services for
> CDN / certificate management. By adopting more public repos, we can lessen
> this burden in case it's switched off. (I believe it's traffic sits around
> 1.5TB/month currently).
> 4. It's an external resource from Apache.
>
> I realize this goal is easier said than done, but I know this repo is also
> currently required for:
>
> 1. grails-plugins github org
> 2. gradle tooling api
> 3. gpc github org
> 4. legacy artifacts - some of which have since had a license change to what
> some consider unacceptable (so this is the only source)
>
> I believe all of these can be worked around. Here are my thoughts on each:
> 1. I'm happy to do the work to start publishing to central.
> 2. You have given us 2 possible solutions. I lean towards just adding the
> repo.gradle.org/gradle/libs-releases location. Alternatively, we can
> create a dedicated repo on repo.grails.org so that it doesn't include the
> legacy artifacts. I'm less a fan of Netbeans since they're repackaging it
> and it could technically be different. We also have to wait for them to
> upgrade gradle (possibly).
> 3. Like #1, I can get GPC publishing to central snapshots or we can publish
> them to their github repos and people can add as they need.
> 4. This is really my #1 concern, but given that 7 has upgraded just about
> every version, I don't see a need for this. If people are still using
> legacy artifacts, we need to know. This is more of a reason to not use it.
>
> It would be nice to also include snapshot repos only on snapshot builds,
> while using release only repos for official releases. Especially now that
> we've gotten rid of the pre-release workflow.
>
> -James
>
>
>
> On Fri, May 2, 2025 at 12:54 PM James Fredley <ja...@apache.org>
> wrote:
>
> > Previously we solved for the org.gradle:gradle-tooling-api dependency by
> > adding a remote repo on repo.grails.org so that it is available via
> > https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
> >
> > org.gradle:gradle-tooling-api is currently a dependency on
> > grails-shell-cli only
> >
> > As we move the main projects away from repo.grails.org, I think we will
> > want to solve this another way. This might be more of a Grails 7.1.0 or
> > 8.0.0 decision.
> >
> > 1. Add an additional repository to the grails-core projects and end
> > generated projects
> > https://repo.gradle.org/gradle/libs-releases/
> >
> > 2. Use the republished versions from NetBeans
> >
> >
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
> >
> >
> >
> >
> >
>
Re: [DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by James Daugherty <jd...@jdresources.net.INVALID>.
I would like to see us try to exclude repo.grails.org/grails/core from our
default applications if at all possible.
There are several reasons:
1. The age of that repo: it has so many dependencies in it.
2. It has both snapshot & release artifacts (I realize that the snapshot
versions aren't supposed to be chosen, but people make mistakes and it
happens).
3. We're relying on Cloudflare to continue to give us "free" services for
CDN / certificate management. By adopting more public repos, we can lessen
this burden in case it's switched off. (I believe it's traffic sits around
1.5TB/month currently).
4. It's an external resource from Apache.
I realize this goal is easier said than done, but I know this repo is also
currently required for:
1. grails-plugins github org
2. gradle tooling api
3. gpc github org
4. legacy artifacts - some of which have since had a license change to what
some consider unacceptable (so this is the only source)
I believe all of these can be worked around. Here are my thoughts on each:
1. I'm happy to do the work to start publishing to central.
2. You have given us 2 possible solutions. I lean towards just adding the
repo.gradle.org/gradle/libs-releases location. Alternatively, we can
create a dedicated repo on repo.grails.org so that it doesn't include the
legacy artifacts. I'm less a fan of Netbeans since they're repackaging it
and it could technically be different. We also have to wait for them to
upgrade gradle (possibly).
3. Like #1, I can get GPC publishing to central snapshots or we can publish
them to their github repos and people can add as they need.
4. This is really my #1 concern, but given that 7 has upgraded just about
every version, I don't see a need for this. If people are still using
legacy artifacts, we need to know. This is more of a reason to not use it.
It would be nice to also include snapshot repos only on snapshot builds,
while using release only repos for official releases. Especially now that
we've gotten rid of the pre-release workflow.
-James
On Fri, May 2, 2025 at 12:54 PM James Fredley <ja...@apache.org>
wrote:
> Previously we solved for the org.gradle:gradle-tooling-api dependency by
> adding a remote repo on repo.grails.org so that it is available via
> https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
>
> org.gradle:gradle-tooling-api is currently a dependency on
> grails-shell-cli only
>
> As we move the main projects away from repo.grails.org, I think we will
> want to solve this another way. This might be more of a Grails 7.1.0 or
> 8.0.0 decision.
>
> 1. Add an additional repository to the grails-core projects and end
> generated projects
> https://repo.gradle.org/gradle/libs-releases/
>
> 2. Use the republished versions from NetBeans
>
> https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions
>
>
>
>
>
[DISCUSS] Handling org.gradle:gradle-tooling-api which is not published to mavenCentral
Posted by James Fredley <ja...@apache.org>.
Previously we solved for the org.gradle:gradle-tooling-api dependency by adding a remote repo on repo.grails.org so that it is available via https://repo.grails.org/ui/native/core/org/gradle/gradle-tooling-api/
org.gradle:gradle-tooling-api is currently a dependency on grails-shell-cli only
As we move the main projects away from repo.grails.org, I think we will want to solve this another way. This might be more of a Grails 7.1.0 or 8.0.0 decision.
1. Add an additional repository to the grails-core projects and end generated projects
https://repo.gradle.org/gradle/libs-releases/
2. Use the republished versions from NetBeans
https://central.sonatype.com/artifact/org.netbeans.external/gradle-tooling-api/versions