Re: Discussion on Build System improvements

[Avik Ganguly <av...@fynarfin.io>]

>
> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a
> horror to find out which ones failed, and our best bet is to text-search
> for "FAILED" in the 17000+ lines of output.

+1

Items like this should be high priority in CI improvements epic. The
overall failed test history, flaky test history, log tracing problem
(actual result is often not sufficient to quickly debug and linking logs to
a failed test case is important) were historically connected to getting PRs
merged in time. CircleCI has helped us with the first two parts of this
problem.

just by splitting up the code as is and moving some classes to different
> project modules is not really taking into account the interdependencies...
> if we don't address those then the project modules are not really helpful
> (actually it's adds more work to the Gradle builds)

I had overestimated what Fineract modularization design efforts have
achieved. Would be interesting to know more about the design here as
"splitting up the code as is" can be helped with more clarity.


On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
cheetah@monkeysintown.com> wrote:

> Kristof,
>
> ... great overview, so first: thanks for that. Some notes...
>
> Repeatable builds: we should consider this as a major bug and not just
> "gloss over" it by executing the build multiple times... stuff like this
> should halt all current development on other features and everyone (or at
> least one person) should focus on fixing this as soon as possible. I would
> consider this also a show stopper for a release.
>
> Dirty checks: we were actually at one point (2+ years?) not too bad here,
> but it seems that this is now actually worse than before... my bet is that
> we introduced some performance hogs by splitting up the code base into
> separate project modules; this didn't really fix the main problem of the
> source code (too many interdependencies between packages aka "modules");
> just by splitting up the code as is and moving some classes to different
> project modules is not really taking into account the interdependencies...
> if we don't address those then the project modules are not really helpful
> (actually it's adds more work to the Gradle builds), neither for publishing
> them on Maven central nor chopping up the code into smaller pieces and
> taking full advantage of caching.
>
> Caching: disabling caching is just not such a good idea... at all...
> again: I would consider this a major bug and halt all machines until it's
> fixed; also: consider this a show stopper for releases.
>
> Console readability: we've talked about this already on occasions (there
> might be even a mailing list message)... on every upgrade of the
> dependencies (including the quality assurance tools like Errorprone etc.)
> the screws get tightened (read: rules get stricter); I remember that after
> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
> enforcements, because the reported style and quality checks literally
> exploded; holding back on Errorprone (and similar) upgrades is obviously
> not a solution; I tried to be conservative (just disabled those that
> brought the build to a halt aka that were reported as errors), but as we
> can see there are enough warnings left to drive everyone crazy; btw:
> disabling the rules with certain annotations is neither a solution, that's
> also avoiding to fix the quality issues.
>
> Suggestions:
>
>    - let's make an effort to reduce the interdependencies between
>    packages aka "modules" before we rip the code apart and move it to separate
>    project modules; if we can't do that (backwards compatibility, not allowed
>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>    no added value of having classes in different jar files when it hurts build
>    performance
>    - let's spend time and fix those style and quality issues, not just
>    pile feature on feature on feature... I am pretty sure that everyone can
>    grab a handful of those warnings and fix them as part of their PRs (even if
>    it's not one that is explicitly dedicated to quality assurance), in other
>    words: everyone look a bit left and right to see what ELSE you can fix/add
>    in addition to your tasks (chores vs cherry picking)... fixing everything
>    in one PR is obviously not practical; we'll have a thousand files
>    changed... this will interrupt everyone else quite significantly and is a
>    nightmare to review
>    - there might be less than a handful of Gradle build constraints that
>    are again too paranoid and mess up the build; I think it's worth looking at
>    the OpenAPI yaml build... that one had issues in the past with proper dirty
>    checks, but there might be other/new constraints that are equally unhelpful
>
> My 2 cents
>
> Aleks
>
>
>
> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> After working with Fineract for some months now, I collected several
>> issues related to the Gradle build system we have in place. I categorized
>> my pain points, and would like to initiate a discussion on the topic with
>> the community. My usage scenarios include building locally with
>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>> build from the web UI.
>>
>> The problems I gathered fall into the following categories:
>>
>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails for
>> the first time with:
>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>> FineractProperties to the 'fineract' property
>> and simply trying it again always makes it wor0k.
>>
>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline character
>> into the Loan class) and restarting Fineract makes it "build" for 17-18
>> seconds (checking all modules to be up-to-date) before starting it again.
>> This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating up to
>> 800% CPU and spinning up all the vents in my rig while doing so.
>>
>> - *INTEGRATION TESTS*: the same dirty check problem happens when working
>> with tests as well. Adding a blank line to an existing test class and
>> re-running the test again happens after a ~20 seconds "rebuild" process.
>> This leads to painfully slow code-test cycles at local development.
>>
>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the Gradle
>> daemon repeatedly grows up to 8Gb memory usage. That causes issues with dev
>> boxes having 16Gb or even less RAM.. Even using my high-spec dev box, I
>> needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>> -Dorg.gradle.priority=low" to keep my desktop responsible during local
>> builds.
>>
>> - *CACHING*: I see many team members and mailing list members
>> recommending `--no-build-cache --no-daemon` settings when invoking
>> gradle, to avoid issues with our build. This kind of denies even any
>> potential adventages of using Gradle over more stable build tools. Note,
>> that I do not use these settings (so my issues are definetely not caused by
>> this), but I keep getting this recommendation for a reason.
>>
>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting running
>> gradle with `--console=plain` to get some chance to read the logs in a
>> sensible way. Still, when eg. 2 tests fail out of the 1100+ on a Github
>> build, it's a horror to find out which ones failed, and our best bet is to
>> text-search for "FAILED" in the 17000+ lines of output.
>>
>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>> for a perfect example. Here at line 1489 Gradle shows
>> > Task :fineract-loan:compileJava FAILED
>> and shows a couple of warnings below, but the actual error is shown at
>> line 1393 which appears to be part of
>> > Task :fineract-document:modernizer
>> which is clearly misleading, and your best bet to find the actual error
>> on Github is to search for "error: ", but NOT using Ctrl+F, but the
>> search box at the top right corner. That's ridiculous - a sensible build
>> tool should communicate build errors in a straightforward way.
>>
>> I'd like to ask you to share your experiences and pain points with the
>> current build that we have and use every day. Once we have a clearer
>> understanding of the areas that need improvement, we can discuss potential
>> strategies to revise our build system and ensure it meets basic
>> requirements.To be fair, please also make sure you share any recent
>> positive experiences related to Fineract builds, if you have any.
>>
>> Thanks,
>> Kristof
>>
>>

Re: Discussion on Build System improvements

[Kristof Jozsa <kj...@apache.org>]

Hi James,

Apologies for the delayed reply - things were a bit quieter here over the
holidays.
Thanks for putting this together and for the kind words. I don't have
anything further to add for 2025—the summary you've written covers it well.
I appreciate you highlighting this for the board report.

Best,
Kristof

On Tue, Dec 30, 2025 at 9:56 PM James Dailey <jd...@apache.org> wrote:

> Kristof
>
> I would like to highlight this work completed in 2025 as part of the
> project's January report to the ASF Board.  To my mind, this showed some
> important things:
>
>    1. This was a collaborative approach on the project with multiple devs
>    chiming in to offer ideas and converging on treating Build Reliability and
>    Developer productivity as critical concepts.
>    2. Your "build is fragile and slow" analysis led you to pinpointing a
>    key Gradle plugin wiring issue - and specific ideas how to resolve.
>    3. You went the extra mile in speaking to the Gradle core team to get
>    to specific recommendations for them to implement in their code.
>    4. Build improvements were noticed and celebrated.
>    5. You took on additional things to work on and they got done.
>
> Anything to add for 2025 on this thread?
>
> I don't know if it was properly celebrated, but at one point you mentioned
> the GE (Gradle Enterprise) build observability tool.  That is now called
> Develocity, an Apache silver partner, and Fineract reports are showing
> build stats and trends.
>
> https://develocity.apache.org/scans/trends?search.relativeStartTime=P90D&search.rootProjectNames=fineract&search.timeZoneId=America%2FLos_Angeles&trends.timeResolution=week.
>  (choosing weekly to get a wider view)
>
> I think that's a great outcome - not only are the build time trends
> looking good, we know they are looking good.
>
> Thanks,
>
>
>
> On Mon, Jan 20, 2025 at 5:59 PM VICTOR MANUEL ROMERO RODRIGUEZ <
> victor.romero@fintecheando.mx> wrote:
>
>> @Józsa Kristóf <kr...@dpc.hu>
>>
>> The parameters used for building the Apache Fineract and the hardware
>> specs used for reaching the numbers have been measured using the GitHub
>> Actions environment.
>>
>> Has it been tested using another platform ?
>>
>> I have reviewed the Github Action execution times and times are very
>> impressive .
>>
>> https://github.com/apache/fineract/actions/runs/12836398561
>> https://github.com/apache/fineract/actions/runs/12836398564
>>
>> I have tested on a local dev computer and works very fine.
>>
>> Regards
>>
>> Víctor Romero
>>
>>
>> El lun, 20 ene 2025 a las 14:59, James Dailey (<ja...@gmail.com>)
>> escribió:
>>
>>> Awesome !
>>>
>>> On Mon, Jan 20, 2025 at 12:30 PM Ádám Sághy <ad...@gmail.com> wrote:
>>>
>>>> Hi Kristof,
>>>>
>>>> I really appreciate all the work you put into this.
>>>>
>>>> I am happy to say it is working, it's definitely building faster!
>>>>
>>>> I was using whole day these changes, built (after gradle clean) and
>>>> rebuilt multiple times the backend, deployed locally back and forth and
>>>> there was no error, no issues.
>>>>
>>>> Good job!
>>>>
>>>> Regards
>>>> Adam
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 17 Jan 2025, at 22:15, Kristof Jozsa <kr...@gmail.com>
>>>> wrote:
>>>>
>>>> 
>>>> Hi all,
>>>>
>>>> I'd like to revive this thread. While I've been relatively quiet on the
>>>> topic here on the mailing list, those who work with me in person know that
>>>> I've spent countless hours over the past few months trying to address the
>>>> most serious pain points in our ever-growing Gradle build. I won't bore you
>>>> with all the details, but I’ve had several attempts—some that didn’t work,
>>>> some that partially worked, and I had to restart from scratch several times
>>>> also.
>>>>
>>>> In my most recent attempt, mostly during the holidays and since, I
>>>> started fresh from a recent apache/develop HEAD with a narrowed focus:
>>>> - enable caching for the longest-running, most critical tasks.
>>>> - introduce the new :fineract-provider:devRun task
>>>> - avoid breaking any existing functionality
>>>>
>>>> The second goal probably deserves some explanation. Most of us build
>>>> and run Fineract *several times per hour* during development,
>>>> typically using the :fineract-provider:bootRun task. I identified
>>>> several plugins that run every time (e.g., Checkstyle, Spotless,
>>>> ErrorProne, Javadoc, SpotBugs, and Modernizer) and decided to exclude them
>>>> from the new :fineract-provider:devRun task. This new task is
>>>> otherwise a clone of bootRun and should achieve the same outcome but with
>>>> improved efficiency.
>>>>
>>>> To share some of my latest measurements with this PR, here are the
>>>> rebuild times of the project (right after a successful build) excluding
>>>> tests on both the current apache/develop HEAD and my branch for a
>>>> direct comparison:
>>>> *Rebuild on apache/develop:*
>>>> BUILD SUCCESSFUL in 4m 20s
>>>> 524 actionable tasks: 186 executed, 338 up-to-date
>>>>
>>>>
>>>> *Rebuild on #devrun branch:*BUILD SUCCESSFUL in 27s
>>>> 606 actionable tasks: 88 executed, 518 up-to-date
>>>> You can also check the apache github page to see improved build times,
>>>> see eg.
>>>> https://github.com/apache/fineract/actions/workflows/build-docker-postgresql.yml
>>>> - the Docker Postgres job's build time decreased from 13-14mins to 8m30s..
>>>> but there are similar advantages across all build jobs triggered by the
>>>> PR's build.
>>>>
>>>> This PR is currently in draft form and can be found here:
>>>> https://github.com/apache/fineract/pull/4272. At this point, I’d like
>>>> to *ask the community for help* with testing. Please pull the changes,
>>>> test them, and let me know if they help or if they break anything.
>>>> Experiment with using the new devRun task for local development and
>>>> see if it improves your daily workflow. Feel free to share your feedback
>>>> here on the mailing list or with me directly on the Mifos Slack for more
>>>> interactive discussions.
>>>>
>>>> Please note that this PR is definitely a first step (though I believe a
>>>> highly valuable one), and not the last one related to improving our build.
>>>> There are still countless opportunities to simplify, clean up, remove
>>>> obsolete dependencies/plugins, rewrite the build files to the new KTS
>>>> format (I started, but it got abandoned along the way), etc.. Feel free to
>>>> suggest additional improvements and let's prioritize.. And as always, any
>>>> help is greatly appreciated.
>>>>
>>>> Thanks much,
>>>> Kristof
>>>>
>>>>
>>>> On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic <
>>>> cheetah@monkeysintown.com> wrote:
>>>>
>>>>> ... well done Kristof! Thanks for taking the time and giving this in
>>>>> depth view on the performance bottlenecks!
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Today I had a very lengthy discussion in person with a core Gradle
>>>>>> engineer. We looked at our Fineract build, and the various issues we
>>>>>> gathered, and also looked for reasons and possible ways to get around them.
>>>>>> Let me summarize the hints and suggestions we received:
>>>>>>
>>>>>> *- The swagger plugin in fineract-provider module*
>>>>>> This one seems to be the root of one of our primary pain points. This
>>>>>> is wired in the *fineract-provider* module, and it's done in a quite
>>>>>> cumbersome way.. but for a reason. The reason seems to be that this plugin
>>>>>> looks plain incorrect, because of its hard-wired dependencies, as clearly
>>>>>> shown in its source at
>>>>>> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
>>>>>> The problem with this plugin and its wiring is that it forces Gradle to
>>>>>> rebuild this module every time, and it's a very lengthy move..
>>>>>>
>>>>>> The suggested way would be to include swagger's generated resources
>>>>>> using the processResources task, but approaching it that way causes
>>>>>> a circular dependency, because the plugin's incorrectly wired
>>>>>> dependencies.. To get this fixed, we need to raise an issue at the plugin
>>>>>> and get it fixed first. An alternative approach would be splitting the
>>>>>> swagger resource generation to its own module inside Fineract, and accept
>>>>>> the fact that it corrupts Gradle's cache mechanism (so that this new module
>>>>>> would always get re-built in every case).
>>>>>>
>>>>>> The cache issue can also be seen in Develocity:
>>>>>> [image: image.png]
>>>>>>
>>>>>> We concluded that I'll try to raise an issue at the Swagger Github
>>>>>> about the plugins incorrect ways of doing things, and they'll also
>>>>>> emphasize it from Gradle's side, hopefully getting it fixed mid-term.
>>>>>>
>>>>>> *- Console log's readabilty*
>>>>>> Gradle team understands these concerns, and are working on improving
>>>>>> these parts already. They are considering adding information to each line
>>>>>> which helps to understand where the piece of information is coming from.
>>>>>> That could help demystifing logs like this (and cleaning up the related
>>>>>> parts in the build):
>>>>>> [image: image.png]
>>>>>> They are also considering gathering all error reports at the end of
>>>>>> the builds (again), so we won't have to hunt the errors from the flood of
>>>>>> 897 warning messages. Speaking of which:
>>>>>>
>>>>>> *- Suggestion: remove warning messages for headless builds*
>>>>>> We discussed the importance of warning messages, but their point is
>>>>>> that either those are important (so could be errors as well), or not
>>>>>> important, so we should silence them. At least for the headless builds
>>>>>> running on CI servers, because no one will care about those warnings
>>>>>> anyway. That doesn't perfectly fit my view, but for hundreds of warnings
>>>>>> being there for many years, they have a point...
>>>>>>
>>>>>> *- IntelliJ Build View readability*
>>>>>> They also consider improving the UI on IntelliJ's Build view, this
>>>>>> one:
>>>>>> [image: image.png]
>>>>>> (today I learnt that lines nested under bootRun are exceptions
>>>>>> gathered and picked up from the console logs.. Personally I think they are
>>>>>> not useful at all, especially with all of the stacktrace removed from the
>>>>>> detail view on the right side, and not related to the build either...)
>>>>>>
>>>>>> *- Suggestion: switch to the Kotlin DSL and clean up builds*
>>>>>> Gradle team thinks the Kotlin DSL is the way forward, and we should
>>>>>> seriously consider switching our build from Groovy to Kotlin, especially if
>>>>>> we are aiming to clean up our build somewhat meanwhile. There are also
>>>>>> parts which look deprecated and possibly no longer required, like
>>>>>> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
>>>>>> `println` statements while we're at it..
>>>>>>
>>>>>> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
>>>>>> running*
>>>>>> Our Fineract build tries to utilize the --scan feature already,
>>>>>> which provides deep insights of what's happening, why it's happening and
>>>>>> how long it takes. The only issue is that these scans get uploaded to
>>>>>> ge.apache.org by default, and it's down... I got connected to
>>>>>> another Gradle team member, who mentioned that ASF Infra hosts
>>>>>> ge.apache.org, and there are recurring availability issues with it,
>>>>>> but they (Gradle team) are happy to help. They also drafted a plan forward,
>>>>>> but I'd prefer to discuss it personally with the AST team member usually
>>>>>> taking care of this host.. @Aleksandar maybe you know who can touch these
>>>>>> parts? Let's discuss this on Slack..
>>>>>>
>>>>>> That's all I could push these issues forward today. Hope that helps
>>>>>> in the long run...
>>>>>>
>>>>>> Kristof
>>>>>>
>>>>>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>
>>>>>>> ... the performance issues stem from "dependsOn"... and similar...
>>>>>>> there are also neat features in Gradle where you can precompile pretty much
>>>>>>> all of the build script (and "configuration" cache it) which would boost
>>>>>>> the build process by a lot (disclosure: I am using this technique in
>>>>>>> projects with 3x of Fineract's project modules).
>>>>>>>
>>>>>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>>>>>>> caching... which is really a big deal performance wise... BUT: to fully
>>>>>>> take advantage of that feature all plugins used must support that (and of
>>>>>>> course some don't). Still, configuration caching would be the "cherry on
>>>>>>> top"... our main problem is "dependsOn" (explicit and implicit).
>>>>>>>
>>>>>>> I can see the declarative style of Maven (having used it since
>>>>>>> before it was released 1.0)... but I would still bet my money on Gradle...
>>>>>>> and you are right, it would be a lot of work to create a Maven build from
>>>>>>> scratch.
>>>>>>>
>>>>>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <
>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>
>>>>>>>> I believe many of our issues stem from our current Gradle build
>>>>>>>> being overly complex. One of my main concerns with Gradle is that it allows
>>>>>>>> customization to evolve completely out of control. Following this
>>>>>>>> principle, I would be opposed to adding new elements to our build.
>>>>>>>>
>>>>>>>> Instead, I’d like to see the current build configuration simplified
>>>>>>>> as much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>>>>>>> always preferred Maven’s declarative approach over the programmatic style
>>>>>>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>>>>>>> could dedicate some time to reducing the custom behavior in our build, I
>>>>>>>> believe it would resolve many of the issues we’re currently facing.
>>>>>>>> An alternative would be turning back to a more maintainable,
>>>>>>>> declarative approach, but crafting a new build from scratch covering the
>>>>>>>> complete current functionality would be a hell of a work..
>>>>>>>>
>>>>>>>> Kristof
>>>>>>>>
>>>>>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>>>>>> victor.romero@fintecheando.mx> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> What about adding observability to the building process? In this
>>>>>>>>> way we can have information about the root causes of the issues during the
>>>>>>>>> building time.
>>>>>>>>>
>>>>>>>>> A OSS suggestion:
>>>>>>>>>
>>>>>>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> Victor
>>>>>>>>>
>>>>>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<
>>>>>>>>> jamespdailey@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> +1  agree
>>>>>>>>>> How do we tackle this ?
>>>>>>>>>>
>>>>>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build,
>>>>>>>>>>>> it's a horror to find out which ones failed, and our best bet is to
>>>>>>>>>>>> text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>> Items like this should be high priority in CI improvements epic.
>>>>>>>>>>> The overall failed test history, flaky test history, log tracing problem
>>>>>>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>>>>>>> problem.
>>>>>>>>>>>
>>>>>>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>>>>>>
>>>>>>>>>>> I had overestimated what Fineract modularization design efforts
>>>>>>>>>>> have achieved. Would be interesting to know more about the design here as
>>>>>>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Kristof,
>>>>>>>>>>>>
>>>>>>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>>>>>>
>>>>>>>>>>>> Repeatable builds: we should consider this as a major bug and
>>>>>>>>>>>> not just "gloss over" it by executing the build multiple times... stuff
>>>>>>>>>>>> like this should halt all current development on other features and
>>>>>>>>>>>> everyone (or at least one person) should focus on fixing this as soon as
>>>>>>>>>>>> possible. I would consider this also a show stopper for a release.
>>>>>>>>>>>>
>>>>>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too
>>>>>>>>>>>> bad here, but it seems that this is now actually worse than before... my
>>>>>>>>>>>> bet is that we introduced some performance hogs by splitting up the code
>>>>>>>>>>>> base into separate project modules; this didn't really fix the main problem
>>>>>>>>>>>> of the source code (too many interdependencies between packages aka
>>>>>>>>>>>> "modules"); just by splitting up the code as is and moving some classes to
>>>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds),
>>>>>>>>>>>> neither for publishing them on Maven central nor chopping up the code into
>>>>>>>>>>>> smaller pieces and taking full advantage of caching.
>>>>>>>>>>>>
>>>>>>>>>>>> Caching: disabling caching is just not such a good idea... at
>>>>>>>>>>>> all... again: I would consider this a major bug and halt all machines until
>>>>>>>>>>>> it's fixed; also: consider this a show stopper for releases.
>>>>>>>>>>>>
>>>>>>>>>>>> Console readability: we've talked about this already on
>>>>>>>>>>>> occasions (there might be even a mailing list message)... on every upgrade
>>>>>>>>>>>> of the dependencies (including the quality assurance tools like Errorprone
>>>>>>>>>>>> etc.) the screws get tightened (read: rules get stricter); I remember that
>>>>>>>>>>>> after one upgrade (maybe last year, maybe longer ago) I had to disable some
>>>>>>>>>>>> rule enforcements, because the reported style and quality checks literally
>>>>>>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>>>>>>> also avoiding to fix the quality issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Suggestions:
>>>>>>>>>>>>
>>>>>>>>>>>>    - let's make an effort to reduce the interdependencies
>>>>>>>>>>>>    between packages aka "modules" before we rip the code apart and move it to
>>>>>>>>>>>>    separate project modules; if we can't do that (backwards compatibility, not
>>>>>>>>>>>>    allowed to "break eggs" etc.) then let's leave the code in one place as
>>>>>>>>>>>>    is... I see no added value of having classes in different jar files when it
>>>>>>>>>>>>    hurts build performance
>>>>>>>>>>>>    - let's spend time and fix those style and quality issues,
>>>>>>>>>>>>    not just pile feature on feature on feature... I am pretty sure that
>>>>>>>>>>>>    everyone can grab a handful of those warnings and fix them as part of their
>>>>>>>>>>>>    PRs (even if it's not one that is explicitly dedicated to quality
>>>>>>>>>>>>    assurance), in other words: everyone look a bit left and right to see what
>>>>>>>>>>>>    ELSE you can fix/add in addition to your tasks (chores vs cherry
>>>>>>>>>>>>    picking)... fixing everything in one PR is obviously not practical; we'll
>>>>>>>>>>>>    have a thousand files changed... this will interrupt everyone else quite
>>>>>>>>>>>>    significantly and is a nightmare to review
>>>>>>>>>>>>    - there might be less than a handful of Gradle build
>>>>>>>>>>>>    constraints that are again too paranoid and mess up the build; I think it's
>>>>>>>>>>>>    worth looking at the OpenAPI yaml build... that one had issues in the past
>>>>>>>>>>>>    with proper dirty checks, but there might be other/new constraints that are
>>>>>>>>>>>>    equally unhelpful
>>>>>>>>>>>>
>>>>>>>>>>>> My 2 cents
>>>>>>>>>>>>
>>>>>>>>>>>> Aleks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>>>>>>> build from the web UI.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>>>>>>> fails for the first time with:
>>>>>>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens
>>>>>>>>>>>>> when working with tests as well. Adding a blank line to an existing test
>>>>>>>>>>>>> class and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>>>>>>> development.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases
>>>>>>>>>>>>> the Gradle daemon repeatedly grows up to 8Gb memory usage. That causes
>>>>>>>>>>>>> issues with dev boxes having 16Gb or even less RAM.. Even using my
>>>>>>>>>>>>> high-spec dev box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible
>>>>>>>>>>>>> during local builds.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>>>>>>> recommending `--no-build-cache --no-daemon` settings when
>>>>>>>>>>>>> invoking gradle, to avoid issues with our build. This kind of denies even
>>>>>>>>>>>>> any potential adventages of using Gradle over more stable build tools.
>>>>>>>>>>>>> Note, that I do not use these settings (so my issues are definetely not
>>>>>>>>>>>>> caused by this), but I keep getting this recommendation for a reason.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs
>>>>>>>>>>>>> suggesting running gradle with `--console=plain` to get some
>>>>>>>>>>>>> chance to read the logs in a sensible way. Still, when eg. 2 tests fail out
>>>>>>>>>>>>> of the 1100+ on a Github build, it's a horror to find out which ones
>>>>>>>>>>>>> failed, and our best bet is to text-search for "FAILED" in the 17000+ lines
>>>>>>>>>>>>> of output.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point,
>>>>>>>>>>>>> check
>>>>>>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>>>>>>> and shows a couple of warnings below, but the actual error is
>>>>>>>>>>>>> shown at line 1393 which appears to be part of
>>>>>>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>>>>>>> which is clearly misleading, and your best bet to find the
>>>>>>>>>>>>> actual error on Github is to search for "error: ", but NOT
>>>>>>>>>>>>> using Ctrl+F, but the search box at the top right corner. That's ridiculous
>>>>>>>>>>>>> - a sensible build tool should communicate build errors in a
>>>>>>>>>>>>> straightforward way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to ask you to share your experiences and pain points
>>>>>>>>>>>>> with the current build that we have and use every day. Once we have a
>>>>>>>>>>>>> clearer understanding of the areas that need improvement, we can discuss
>>>>>>>>>>>>> potential strategies to revise our build system and ensure it meets basic
>>>>>>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Kristof
>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: Discussion on Build System improvements

[James Dailey <jd...@apache.org>]

Kristof

I would like to highlight this work completed in 2025 as part of the
project's January report to the ASF Board.  To my mind, this showed some
important things:

   1. This was a collaborative approach on the project with multiple devs
   chiming in to offer ideas and converging on treating Build Reliability and
   Developer productivity as critical concepts.
   2. Your "build is fragile and slow" analysis led you to pinpointing a
   key Gradle plugin wiring issue - and specific ideas how to resolve.
   3. You went the extra mile in speaking to the Gradle core team to get to
   specific recommendations for them to implement in their code.
   4. Build improvements were noticed and celebrated.
   5. You took on additional things to work on and they got done.

Anything to add for 2025 on this thread?

I don't know if it was properly celebrated, but at one point you mentioned
the GE (Gradle Enterprise) build observability tool.  That is now called
Develocity, an Apache silver partner, and Fineract reports are showing
build stats and trends.

https://develocity.apache.org/scans/trends?search.relativeStartTime=P90D&search.rootProjectNames=fineract&search.timeZoneId=America%2FLos_Angeles&trends.timeResolution=week.
 (choosing weekly to get a wider view)

I think that's a great outcome - not only are the build time trends looking
good, we know they are looking good.

Thanks,



On Mon, Jan 20, 2025 at 5:59 PM VICTOR MANUEL ROMERO RODRIGUEZ <
victor.romero@fintecheando.mx> wrote:

> @Józsa Kristóf <kr...@dpc.hu>
>
> The parameters used for building the Apache Fineract and the hardware
> specs used for reaching the numbers have been measured using the GitHub
> Actions environment.
>
> Has it been tested using another platform ?
>
> I have reviewed the Github Action execution times and times are very
> impressive .
>
> https://github.com/apache/fineract/actions/runs/12836398561
> https://github.com/apache/fineract/actions/runs/12836398564
>
> I have tested on a local dev computer and works very fine.
>
> Regards
>
> Víctor Romero
>
>
> El lun, 20 ene 2025 a las 14:59, James Dailey (<ja...@gmail.com>)
> escribió:
>
>> Awesome !
>>
>> On Mon, Jan 20, 2025 at 12:30 PM Ádám Sághy <ad...@gmail.com> wrote:
>>
>>> Hi Kristof,
>>>
>>> I really appreciate all the work you put into this.
>>>
>>> I am happy to say it is working, it's definitely building faster!
>>>
>>> I was using whole day these changes, built (after gradle clean) and
>>> rebuilt multiple times the backend, deployed locally back and forth and
>>> there was no error, no issues.
>>>
>>> Good job!
>>>
>>> Regards
>>> Adam
>>>
>>> Sent from my iPhone
>>>
>>> On 17 Jan 2025, at 22:15, Kristof Jozsa <kr...@gmail.com> wrote:
>>>
>>> 
>>> Hi all,
>>>
>>> I'd like to revive this thread. While I've been relatively quiet on the
>>> topic here on the mailing list, those who work with me in person know that
>>> I've spent countless hours over the past few months trying to address the
>>> most serious pain points in our ever-growing Gradle build. I won't bore you
>>> with all the details, but I’ve had several attempts—some that didn’t work,
>>> some that partially worked, and I had to restart from scratch several times
>>> also.
>>>
>>> In my most recent attempt, mostly during the holidays and since, I
>>> started fresh from a recent apache/develop HEAD with a narrowed focus:
>>> - enable caching for the longest-running, most critical tasks.
>>> - introduce the new :fineract-provider:devRun task
>>> - avoid breaking any existing functionality
>>>
>>> The second goal probably deserves some explanation. Most of us build and
>>> run Fineract *several times per hour* during development, typically
>>> using the :fineract-provider:bootRun task. I identified several plugins
>>> that run every time (e.g., Checkstyle, Spotless, ErrorProne, Javadoc,
>>> SpotBugs, and Modernizer) and decided to exclude them from the new
>>> :fineract-provider:devRun task. This new task is otherwise a clone of
>>> bootRun and should achieve the same outcome but with improved efficiency.
>>>
>>> To share some of my latest measurements with this PR, here are the
>>> rebuild times of the project (right after a successful build) excluding
>>> tests on both the current apache/develop HEAD and my branch for a
>>> direct comparison:
>>> *Rebuild on apache/develop:*
>>> BUILD SUCCESSFUL in 4m 20s
>>> 524 actionable tasks: 186 executed, 338 up-to-date
>>>
>>>
>>> *Rebuild on #devrun branch:*BUILD SUCCESSFUL in 27s
>>> 606 actionable tasks: 88 executed, 518 up-to-date
>>> You can also check the apache github page to see improved build times,
>>> see eg.
>>> https://github.com/apache/fineract/actions/workflows/build-docker-postgresql.yml
>>> - the Docker Postgres job's build time decreased from 13-14mins to 8m30s..
>>> but there are similar advantages across all build jobs triggered by the
>>> PR's build.
>>>
>>> This PR is currently in draft form and can be found here:
>>> https://github.com/apache/fineract/pull/4272. At this point, I’d like
>>> to *ask the community for help* with testing. Please pull the changes,
>>> test them, and let me know if they help or if they break anything.
>>> Experiment with using the new devRun task for local development and see
>>> if it improves your daily workflow. Feel free to share your feedback here
>>> on the mailing list or with me directly on the Mifos Slack for more
>>> interactive discussions.
>>>
>>> Please note that this PR is definitely a first step (though I believe a
>>> highly valuable one), and not the last one related to improving our build.
>>> There are still countless opportunities to simplify, clean up, remove
>>> obsolete dependencies/plugins, rewrite the build files to the new KTS
>>> format (I started, but it got abandoned along the way), etc.. Feel free to
>>> suggest additional improvements and let's prioritize.. And as always, any
>>> help is greatly appreciated.
>>>
>>> Thanks much,
>>> Kristof
>>>
>>>
>>> On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic <
>>> cheetah@monkeysintown.com> wrote:
>>>
>>>> ... well done Kristof! Thanks for taking the time and giving this in
>>>> depth view on the performance bottlenecks!
>>>>
>>>> Cheers
>>>>
>>>> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Today I had a very lengthy discussion in person with a core Gradle
>>>>> engineer. We looked at our Fineract build, and the various issues we
>>>>> gathered, and also looked for reasons and possible ways to get around them.
>>>>> Let me summarize the hints and suggestions we received:
>>>>>
>>>>> *- The swagger plugin in fineract-provider module*
>>>>> This one seems to be the root of one of our primary pain points. This
>>>>> is wired in the *fineract-provider* module, and it's done in a quite
>>>>> cumbersome way.. but for a reason. The reason seems to be that this plugin
>>>>> looks plain incorrect, because of its hard-wired dependencies, as clearly
>>>>> shown in its source at
>>>>> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
>>>>> The problem with this plugin and its wiring is that it forces Gradle to
>>>>> rebuild this module every time, and it's a very lengthy move..
>>>>>
>>>>> The suggested way would be to include swagger's generated resources
>>>>> using the processResources task, but approaching it that way causes a
>>>>> circular dependency, because the plugin's incorrectly wired dependencies..
>>>>> To get this fixed, we need to raise an issue at the plugin and get it fixed
>>>>> first. An alternative approach would be splitting the swagger resource
>>>>> generation to its own module inside Fineract, and accept the fact that it
>>>>> corrupts Gradle's cache mechanism (so that this new module would always get
>>>>> re-built in every case).
>>>>>
>>>>> The cache issue can also be seen in Develocity:
>>>>> [image: image.png]
>>>>>
>>>>> We concluded that I'll try to raise an issue at the Swagger Github
>>>>> about the plugins incorrect ways of doing things, and they'll also
>>>>> emphasize it from Gradle's side, hopefully getting it fixed mid-term.
>>>>>
>>>>> *- Console log's readabilty*
>>>>> Gradle team understands these concerns, and are working on improving
>>>>> these parts already. They are considering adding information to each line
>>>>> which helps to understand where the piece of information is coming from.
>>>>> That could help demystifing logs like this (and cleaning up the related
>>>>> parts in the build):
>>>>> [image: image.png]
>>>>> They are also considering gathering all error reports at the end of
>>>>> the builds (again), so we won't have to hunt the errors from the flood of
>>>>> 897 warning messages. Speaking of which:
>>>>>
>>>>> *- Suggestion: remove warning messages for headless builds*
>>>>> We discussed the importance of warning messages, but their point is
>>>>> that either those are important (so could be errors as well), or not
>>>>> important, so we should silence them. At least for the headless builds
>>>>> running on CI servers, because no one will care about those warnings
>>>>> anyway. That doesn't perfectly fit my view, but for hundreds of warnings
>>>>> being there for many years, they have a point...
>>>>>
>>>>> *- IntelliJ Build View readability*
>>>>> They also consider improving the UI on IntelliJ's Build view, this one:
>>>>> [image: image.png]
>>>>> (today I learnt that lines nested under bootRun are exceptions
>>>>> gathered and picked up from the console logs.. Personally I think they are
>>>>> not useful at all, especially with all of the stacktrace removed from the
>>>>> detail view on the right side, and not related to the build either...)
>>>>>
>>>>> *- Suggestion: switch to the Kotlin DSL and clean up builds*
>>>>> Gradle team thinks the Kotlin DSL is the way forward, and we should
>>>>> seriously consider switching our build from Groovy to Kotlin, especially if
>>>>> we are aiming to clean up our build somewhat meanwhile. There are also
>>>>> parts which look deprecated and possibly no longer required, like
>>>>> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
>>>>> `println` statements while we're at it..
>>>>>
>>>>> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
>>>>> running*
>>>>> Our Fineract build tries to utilize the --scan feature already, which
>>>>> provides deep insights of what's happening, why it's happening and how long
>>>>> it takes. The only issue is that these scans get uploaded to
>>>>> ge.apache.org by default, and it's down... I got connected to another
>>>>> Gradle team member, who mentioned that ASF Infra hosts ge.apache.org,
>>>>> and there are recurring availability issues with it, but they (Gradle team)
>>>>> are happy to help. They also drafted a plan forward, but I'd prefer to
>>>>> discuss it personally with the AST team member usually taking care of this
>>>>> host.. @Aleksandar maybe you know who can touch these parts? Let's discuss
>>>>> this on Slack..
>>>>>
>>>>> That's all I could push these issues forward today. Hope that helps in
>>>>> the long run...
>>>>>
>>>>> Kristof
>>>>>
>>>>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
>>>>> cheetah@monkeysintown.com> wrote:
>>>>>
>>>>>> ... the performance issues stem from "dependsOn"... and similar...
>>>>>> there are also neat features in Gradle where you can precompile pretty much
>>>>>> all of the build script (and "configuration" cache it) which would boost
>>>>>> the build process by a lot (disclosure: I am using this technique in
>>>>>> projects with 3x of Fineract's project modules).
>>>>>>
>>>>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>>>>>> caching... which is really a big deal performance wise... BUT: to fully
>>>>>> take advantage of that feature all plugins used must support that (and of
>>>>>> course some don't). Still, configuration caching would be the "cherry on
>>>>>> top"... our main problem is "dependsOn" (explicit and implicit).
>>>>>>
>>>>>> I can see the declarative style of Maven (having used it since before
>>>>>> it was released 1.0)... but I would still bet my money on Gradle... and you
>>>>>> are right, it would be a lot of work to create a Maven build from scratch.
>>>>>>
>>>>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I believe many of our issues stem from our current Gradle build
>>>>>>> being overly complex. One of my main concerns with Gradle is that it allows
>>>>>>> customization to evolve completely out of control. Following this
>>>>>>> principle, I would be opposed to adding new elements to our build.
>>>>>>>
>>>>>>> Instead, I’d like to see the current build configuration simplified
>>>>>>> as much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>>>>>> always preferred Maven’s declarative approach over the programmatic style
>>>>>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>>>>>> could dedicate some time to reducing the custom behavior in our build, I
>>>>>>> believe it would resolve many of the issues we’re currently facing.
>>>>>>> An alternative would be turning back to a more maintainable,
>>>>>>> declarative approach, but crafting a new build from scratch covering the
>>>>>>> complete current functionality would be a hell of a work..
>>>>>>>
>>>>>>> Kristof
>>>>>>>
>>>>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>>>>> victor.romero@fintecheando.mx> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> What about adding observability to the building process? In this
>>>>>>>> way we can have information about the root causes of the issues during the
>>>>>>>> building time.
>>>>>>>>
>>>>>>>> A OSS suggestion:
>>>>>>>>
>>>>>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Victor
>>>>>>>>
>>>>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<
>>>>>>>> jamespdailey@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> +1  agree
>>>>>>>>> How do we tackle this ?
>>>>>>>>>
>>>>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build,
>>>>>>>>>>> it's a horror to find out which ones failed, and our best bet is to
>>>>>>>>>>> text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>> Items like this should be high priority in CI improvements epic.
>>>>>>>>>> The overall failed test history, flaky test history, log tracing problem
>>>>>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>>>>>> problem.
>>>>>>>>>>
>>>>>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>>>>>
>>>>>>>>>> I had overestimated what Fineract modularization design efforts
>>>>>>>>>> have achieved. Would be interesting to know more about the design here as
>>>>>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Kristof,
>>>>>>>>>>>
>>>>>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>>>>>
>>>>>>>>>>> Repeatable builds: we should consider this as a major bug and
>>>>>>>>>>> not just "gloss over" it by executing the build multiple times... stuff
>>>>>>>>>>> like this should halt all current development on other features and
>>>>>>>>>>> everyone (or at least one person) should focus on fixing this as soon as
>>>>>>>>>>> possible. I would consider this also a show stopper for a release.
>>>>>>>>>>>
>>>>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too
>>>>>>>>>>> bad here, but it seems that this is now actually worse than before... my
>>>>>>>>>>> bet is that we introduced some performance hogs by splitting up the code
>>>>>>>>>>> base into separate project modules; this didn't really fix the main problem
>>>>>>>>>>> of the source code (too many interdependencies between packages aka
>>>>>>>>>>> "modules"); just by splitting up the code as is and moving some classes to
>>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds),
>>>>>>>>>>> neither for publishing them on Maven central nor chopping up the code into
>>>>>>>>>>> smaller pieces and taking full advantage of caching.
>>>>>>>>>>>
>>>>>>>>>>> Caching: disabling caching is just not such a good idea... at
>>>>>>>>>>> all... again: I would consider this a major bug and halt all machines until
>>>>>>>>>>> it's fixed; also: consider this a show stopper for releases.
>>>>>>>>>>>
>>>>>>>>>>> Console readability: we've talked about this already on
>>>>>>>>>>> occasions (there might be even a mailing list message)... on every upgrade
>>>>>>>>>>> of the dependencies (including the quality assurance tools like Errorprone
>>>>>>>>>>> etc.) the screws get tightened (read: rules get stricter); I remember that
>>>>>>>>>>> after one upgrade (maybe last year, maybe longer ago) I had to disable some
>>>>>>>>>>> rule enforcements, because the reported style and quality checks literally
>>>>>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>>>>>> also avoiding to fix the quality issues.
>>>>>>>>>>>
>>>>>>>>>>> Suggestions:
>>>>>>>>>>>
>>>>>>>>>>>    - let's make an effort to reduce the interdependencies
>>>>>>>>>>>    between packages aka "modules" before we rip the code apart and move it to
>>>>>>>>>>>    separate project modules; if we can't do that (backwards compatibility, not
>>>>>>>>>>>    allowed to "break eggs" etc.) then let's leave the code in one place as
>>>>>>>>>>>    is... I see no added value of having classes in different jar files when it
>>>>>>>>>>>    hurts build performance
>>>>>>>>>>>    - let's spend time and fix those style and quality issues,
>>>>>>>>>>>    not just pile feature on feature on feature... I am pretty sure that
>>>>>>>>>>>    everyone can grab a handful of those warnings and fix them as part of their
>>>>>>>>>>>    PRs (even if it's not one that is explicitly dedicated to quality
>>>>>>>>>>>    assurance), in other words: everyone look a bit left and right to see what
>>>>>>>>>>>    ELSE you can fix/add in addition to your tasks (chores vs cherry
>>>>>>>>>>>    picking)... fixing everything in one PR is obviously not practical; we'll
>>>>>>>>>>>    have a thousand files changed... this will interrupt everyone else quite
>>>>>>>>>>>    significantly and is a nightmare to review
>>>>>>>>>>>    - there might be less than a handful of Gradle build
>>>>>>>>>>>    constraints that are again too paranoid and mess up the build; I think it's
>>>>>>>>>>>    worth looking at the OpenAPI yaml build... that one had issues in the past
>>>>>>>>>>>    with proper dirty checks, but there might be other/new constraints that are
>>>>>>>>>>>    equally unhelpful
>>>>>>>>>>>
>>>>>>>>>>> My 2 cents
>>>>>>>>>>>
>>>>>>>>>>> Aleks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>>>>>> build from the web UI.
>>>>>>>>>>>>
>>>>>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>>>>>
>>>>>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>>>>>> fails for the first time with:
>>>>>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>>>>>
>>>>>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>>>>>
>>>>>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens
>>>>>>>>>>>> when working with tests as well. Adding a blank line to an existing test
>>>>>>>>>>>> class and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>>>>>> development.
>>>>>>>>>>>>
>>>>>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases
>>>>>>>>>>>> the Gradle daemon repeatedly grows up to 8Gb memory usage. That causes
>>>>>>>>>>>> issues with dev boxes having 16Gb or even less RAM.. Even using my
>>>>>>>>>>>> high-spec dev box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible
>>>>>>>>>>>> during local builds.
>>>>>>>>>>>>
>>>>>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>>>>>> recommending `--no-build-cache --no-daemon` settings when
>>>>>>>>>>>> invoking gradle, to avoid issues with our build. This kind of denies even
>>>>>>>>>>>> any potential adventages of using Gradle over more stable build tools.
>>>>>>>>>>>> Note, that I do not use these settings (so my issues are definetely not
>>>>>>>>>>>> caused by this), but I keep getting this recommendation for a reason.
>>>>>>>>>>>>
>>>>>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs
>>>>>>>>>>>> suggesting running gradle with `--console=plain` to get some
>>>>>>>>>>>> chance to read the logs in a sensible way. Still, when eg. 2 tests fail out
>>>>>>>>>>>> of the 1100+ on a Github build, it's a horror to find out which ones
>>>>>>>>>>>> failed, and our best bet is to text-search for "FAILED" in the 17000+ lines
>>>>>>>>>>>> of output.
>>>>>>>>>>>>
>>>>>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point,
>>>>>>>>>>>> check
>>>>>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>>>>>> and shows a couple of warnings below, but the actual error is
>>>>>>>>>>>> shown at line 1393 which appears to be part of
>>>>>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>>>>>> which is clearly misleading, and your best bet to find the
>>>>>>>>>>>> actual error on Github is to search for "error: ", but NOT
>>>>>>>>>>>> using Ctrl+F, but the search box at the top right corner. That's ridiculous
>>>>>>>>>>>> - a sensible build tool should communicate build errors in a
>>>>>>>>>>>> straightforward way.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to ask you to share your experiences and pain points
>>>>>>>>>>>> with the current build that we have and use every day. Once we have a
>>>>>>>>>>>> clearer understanding of the areas that need improvement, we can discuss
>>>>>>>>>>>> potential strategies to revise our build system and ensure it meets basic
>>>>>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Kristof
>>>>>>>>>>>>
>>>>>>>>>>>>

Re: Discussion on Build System improvements

[VICTOR MANUEL ROMERO RODRIGUEZ <vi...@fintecheando.mx>]

@Józsa Kristóf <kr...@dpc.hu>

The parameters used for building the Apache Fineract and the hardware specs
used for reaching the numbers have been measured using the GitHub Actions
environment.

Has it been tested using another platform ?

I have reviewed the Github Action execution times and times are very
impressive .

https://github.com/apache/fineract/actions/runs/12836398561
https://github.com/apache/fineract/actions/runs/12836398564

I have tested on a local dev computer and works very fine.

Regards

Víctor Romero


El lun, 20 ene 2025 a las 14:59, James Dailey (<ja...@gmail.com>)
escribió:

> Awesome !
>
> On Mon, Jan 20, 2025 at 12:30 PM Ádám Sághy <ad...@gmail.com> wrote:
>
>> Hi Kristof,
>>
>> I really appreciate all the work you put into this.
>>
>> I am happy to say it is working, it's definitely building faster!
>>
>> I was using whole day these changes, built (after gradle clean) and
>> rebuilt multiple times the backend, deployed locally back and forth and
>> there was no error, no issues.
>>
>> Good job!
>>
>> Regards
>> Adam
>>
>> Sent from my iPhone
>>
>> On 17 Jan 2025, at 22:15, Kristof Jozsa <kr...@gmail.com> wrote:
>>
>> 
>> Hi all,
>>
>> I'd like to revive this thread. While I've been relatively quiet on the
>> topic here on the mailing list, those who work with me in person know that
>> I've spent countless hours over the past few months trying to address the
>> most serious pain points in our ever-growing Gradle build. I won't bore you
>> with all the details, but I’ve had several attempts—some that didn’t work,
>> some that partially worked, and I had to restart from scratch several times
>> also.
>>
>> In my most recent attempt, mostly during the holidays and since, I
>> started fresh from a recent apache/develop HEAD with a narrowed focus:
>> - enable caching for the longest-running, most critical tasks.
>> - introduce the new :fineract-provider:devRun task
>> - avoid breaking any existing functionality
>>
>> The second goal probably deserves some explanation. Most of us build and
>> run Fineract *several times per hour* during development, typically
>> using the :fineract-provider:bootRun task. I identified several plugins
>> that run every time (e.g., Checkstyle, Spotless, ErrorProne, Javadoc,
>> SpotBugs, and Modernizer) and decided to exclude them from the new
>> :fineract-provider:devRun task. This new task is otherwise a clone of
>> bootRun and should achieve the same outcome but with improved efficiency.
>>
>> To share some of my latest measurements with this PR, here are the
>> rebuild times of the project (right after a successful build) excluding
>> tests on both the current apache/develop HEAD and my branch for a direct
>> comparison:
>> *Rebuild on apache/develop:*
>> BUILD SUCCESSFUL in 4m 20s
>> 524 actionable tasks: 186 executed, 338 up-to-date
>>
>>
>> *Rebuild on #devrun branch:*BUILD SUCCESSFUL in 27s
>> 606 actionable tasks: 88 executed, 518 up-to-date
>> You can also check the apache github page to see improved build times,
>> see eg.
>> https://github.com/apache/fineract/actions/workflows/build-docker-postgresql.yml
>> - the Docker Postgres job's build time decreased from 13-14mins to 8m30s..
>> but there are similar advantages across all build jobs triggered by the
>> PR's build.
>>
>> This PR is currently in draft form and can be found here:
>> https://github.com/apache/fineract/pull/4272. At this point, I’d like to *ask
>> the community for help* with testing. Please pull the changes, test
>> them, and let me know if they help or if they break anything. Experiment
>> with using the new devRun task for local development and see if it
>> improves your daily workflow. Feel free to share your feedback here on the
>> mailing list or with me directly on the Mifos Slack for more interactive
>> discussions.
>>
>> Please note that this PR is definitely a first step (though I believe a
>> highly valuable one), and not the last one related to improving our build.
>> There are still countless opportunities to simplify, clean up, remove
>> obsolete dependencies/plugins, rewrite the build files to the new KTS
>> format (I started, but it got abandoned along the way), etc.. Feel free to
>> suggest additional improvements and let's prioritize.. And as always, any
>> help is greatly appreciated.
>>
>> Thanks much,
>> Kristof
>>
>>
>> On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic <
>> cheetah@monkeysintown.com> wrote:
>>
>>> ... well done Kristof! Thanks for taking the time and giving this in
>>> depth view on the performance bottlenecks!
>>>
>>> Cheers
>>>
>>> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
>>> wrote:
>>>
>>>> Today I had a very lengthy discussion in person with a core Gradle
>>>> engineer. We looked at our Fineract build, and the various issues we
>>>> gathered, and also looked for reasons and possible ways to get around them.
>>>> Let me summarize the hints and suggestions we received:
>>>>
>>>> *- The swagger plugin in fineract-provider module*
>>>> This one seems to be the root of one of our primary pain points. This
>>>> is wired in the *fineract-provider* module, and it's done in a quite
>>>> cumbersome way.. but for a reason. The reason seems to be that this plugin
>>>> looks plain incorrect, because of its hard-wired dependencies, as clearly
>>>> shown in its source at
>>>> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
>>>> The problem with this plugin and its wiring is that it forces Gradle to
>>>> rebuild this module every time, and it's a very lengthy move..
>>>>
>>>> The suggested way would be to include swagger's generated resources
>>>> using the processResources task, but approaching it that way causes a
>>>> circular dependency, because the plugin's incorrectly wired dependencies..
>>>> To get this fixed, we need to raise an issue at the plugin and get it fixed
>>>> first. An alternative approach would be splitting the swagger resource
>>>> generation to its own module inside Fineract, and accept the fact that it
>>>> corrupts Gradle's cache mechanism (so that this new module would always get
>>>> re-built in every case).
>>>>
>>>> The cache issue can also be seen in Develocity:
>>>> [image: image.png]
>>>>
>>>> We concluded that I'll try to raise an issue at the Swagger Github
>>>> about the plugins incorrect ways of doing things, and they'll also
>>>> emphasize it from Gradle's side, hopefully getting it fixed mid-term.
>>>>
>>>> *- Console log's readabilty*
>>>> Gradle team understands these concerns, and are working on improving
>>>> these parts already. They are considering adding information to each line
>>>> which helps to understand where the piece of information is coming from.
>>>> That could help demystifing logs like this (and cleaning up the related
>>>> parts in the build):
>>>> [image: image.png]
>>>> They are also considering gathering all error reports at the end of the
>>>> builds (again), so we won't have to hunt the errors from the flood of 897
>>>> warning messages. Speaking of which:
>>>>
>>>> *- Suggestion: remove warning messages for headless builds*
>>>> We discussed the importance of warning messages, but their point is
>>>> that either those are important (so could be errors as well), or not
>>>> important, so we should silence them. At least for the headless builds
>>>> running on CI servers, because no one will care about those warnings
>>>> anyway. That doesn't perfectly fit my view, but for hundreds of warnings
>>>> being there for many years, they have a point...
>>>>
>>>> *- IntelliJ Build View readability*
>>>> They also consider improving the UI on IntelliJ's Build view, this one:
>>>> [image: image.png]
>>>> (today I learnt that lines nested under bootRun are exceptions gathered
>>>> and picked up from the console logs.. Personally I think they are not
>>>> useful at all, especially with all of the stacktrace removed from the
>>>> detail view on the right side, and not related to the build either...)
>>>>
>>>> *- Suggestion: switch to the Kotlin DSL and clean up builds*
>>>> Gradle team thinks the Kotlin DSL is the way forward, and we should
>>>> seriously consider switching our build from Groovy to Kotlin, especially if
>>>> we are aiming to clean up our build somewhat meanwhile. There are also
>>>> parts which look deprecated and possibly no longer required, like
>>>> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
>>>> `println` statements while we're at it..
>>>>
>>>> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
>>>> running*
>>>> Our Fineract build tries to utilize the --scan feature already, which
>>>> provides deep insights of what's happening, why it's happening and how long
>>>> it takes. The only issue is that these scans get uploaded to
>>>> ge.apache.org by default, and it's down... I got connected to another
>>>> Gradle team member, who mentioned that ASF Infra hosts ge.apache.org,
>>>> and there are recurring availability issues with it, but they (Gradle team)
>>>> are happy to help. They also drafted a plan forward, but I'd prefer to
>>>> discuss it personally with the AST team member usually taking care of this
>>>> host.. @Aleksandar maybe you know who can touch these parts? Let's discuss
>>>> this on Slack..
>>>>
>>>> That's all I could push these issues forward today. Hope that helps in
>>>> the long run...
>>>>
>>>> Kristof
>>>>
>>>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
>>>> cheetah@monkeysintown.com> wrote:
>>>>
>>>>> ... the performance issues stem from "dependsOn"... and similar...
>>>>> there are also neat features in Gradle where you can precompile pretty much
>>>>> all of the build script (and "configuration" cache it) which would boost
>>>>> the build process by a lot (disclosure: I am using this technique in
>>>>> projects with 3x of Fineract's project modules).
>>>>>
>>>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>>>>> caching... which is really a big deal performance wise... BUT: to fully
>>>>> take advantage of that feature all plugins used must support that (and of
>>>>> course some don't). Still, configuration caching would be the "cherry on
>>>>> top"... our main problem is "dependsOn" (explicit and implicit).
>>>>>
>>>>> I can see the declarative style of Maven (having used it since before
>>>>> it was released 1.0)... but I would still bet my money on Gradle... and you
>>>>> are right, it would be a lot of work to create a Maven build from scratch.
>>>>>
>>>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I believe many of our issues stem from our current Gradle build being
>>>>>> overly complex. One of my main concerns with Gradle is that it allows
>>>>>> customization to evolve completely out of control. Following this
>>>>>> principle, I would be opposed to adding new elements to our build.
>>>>>>
>>>>>> Instead, I’d like to see the current build configuration simplified
>>>>>> as much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>>>>> always preferred Maven’s declarative approach over the programmatic style
>>>>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>>>>> could dedicate some time to reducing the custom behavior in our build, I
>>>>>> believe it would resolve many of the issues we’re currently facing.
>>>>>> An alternative would be turning back to a more maintainable,
>>>>>> declarative approach, but crafting a new build from scratch covering the
>>>>>> complete current functionality would be a hell of a work..
>>>>>>
>>>>>> Kristof
>>>>>>
>>>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>>>> victor.romero@fintecheando.mx> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> What about adding observability to the building process? In this
>>>>>>> way we can have information about the root causes of the issues during the
>>>>>>> building time.
>>>>>>>
>>>>>>> A OSS suggestion:
>>>>>>>
>>>>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Victor
>>>>>>>
>>>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<
>>>>>>> jamespdailey@gmail.com>) escribió:
>>>>>>>
>>>>>>>> +1  agree
>>>>>>>> How do we tackle this ?
>>>>>>>>
>>>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build,
>>>>>>>>>> it's a horror to find out which ones failed, and our best bet is to
>>>>>>>>>> text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> Items like this should be high priority in CI improvements epic.
>>>>>>>>> The overall failed test history, flaky test history, log tracing problem
>>>>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>>>>
>>>>>>>>> I had overestimated what Fineract modularization design efforts
>>>>>>>>> have achieved. Would be interesting to know more about the design here as
>>>>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>>>>
>>>>>>>>>> Kristof,
>>>>>>>>>>
>>>>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>>>>
>>>>>>>>>> Repeatable builds: we should consider this as a major bug and not
>>>>>>>>>> just "gloss over" it by executing the build multiple times... stuff like
>>>>>>>>>> this should halt all current development on other features and everyone (or
>>>>>>>>>> at least one person) should focus on fixing this as soon as possible. I
>>>>>>>>>> would consider this also a show stopper for a release.
>>>>>>>>>>
>>>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too
>>>>>>>>>> bad here, but it seems that this is now actually worse than before... my
>>>>>>>>>> bet is that we introduced some performance hogs by splitting up the code
>>>>>>>>>> base into separate project modules; this didn't really fix the main problem
>>>>>>>>>> of the source code (too many interdependencies between packages aka
>>>>>>>>>> "modules"); just by splitting up the code as is and moving some classes to
>>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds),
>>>>>>>>>> neither for publishing them on Maven central nor chopping up the code into
>>>>>>>>>> smaller pieces and taking full advantage of caching.
>>>>>>>>>>
>>>>>>>>>> Caching: disabling caching is just not such a good idea... at
>>>>>>>>>> all... again: I would consider this a major bug and halt all machines until
>>>>>>>>>> it's fixed; also: consider this a show stopper for releases.
>>>>>>>>>>
>>>>>>>>>> Console readability: we've talked about this already on occasions
>>>>>>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>>>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>>>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>>>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>>>>>>> enforcements, because the reported style and quality checks literally
>>>>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>>>>> also avoiding to fix the quality issues.
>>>>>>>>>>
>>>>>>>>>> Suggestions:
>>>>>>>>>>
>>>>>>>>>>    - let's make an effort to reduce the interdependencies
>>>>>>>>>>    between packages aka "modules" before we rip the code apart and move it to
>>>>>>>>>>    separate project modules; if we can't do that (backwards compatibility, not
>>>>>>>>>>    allowed to "break eggs" etc.) then let's leave the code in one place as
>>>>>>>>>>    is... I see no added value of having classes in different jar files when it
>>>>>>>>>>    hurts build performance
>>>>>>>>>>    - let's spend time and fix those style and quality issues,
>>>>>>>>>>    not just pile feature on feature on feature... I am pretty sure that
>>>>>>>>>>    everyone can grab a handful of those warnings and fix them as part of their
>>>>>>>>>>    PRs (even if it's not one that is explicitly dedicated to quality
>>>>>>>>>>    assurance), in other words: everyone look a bit left and right to see what
>>>>>>>>>>    ELSE you can fix/add in addition to your tasks (chores vs cherry
>>>>>>>>>>    picking)... fixing everything in one PR is obviously not practical; we'll
>>>>>>>>>>    have a thousand files changed... this will interrupt everyone else quite
>>>>>>>>>>    significantly and is a nightmare to review
>>>>>>>>>>    - there might be less than a handful of Gradle build
>>>>>>>>>>    constraints that are again too paranoid and mess up the build; I think it's
>>>>>>>>>>    worth looking at the OpenAPI yaml build... that one had issues in the past
>>>>>>>>>>    with proper dirty checks, but there might be other/new constraints that are
>>>>>>>>>>    equally unhelpful
>>>>>>>>>>
>>>>>>>>>> My 2 cents
>>>>>>>>>>
>>>>>>>>>> Aleks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>>>>> build from the web UI.
>>>>>>>>>>>
>>>>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>>>>
>>>>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>>>>> fails for the first time with:
>>>>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>>>>
>>>>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>>>>
>>>>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens
>>>>>>>>>>> when working with tests as well. Adding a blank line to an existing test
>>>>>>>>>>> class and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>>>>> development.
>>>>>>>>>>>
>>>>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible
>>>>>>>>>>> during local builds.
>>>>>>>>>>>
>>>>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>>>>> recommending `--no-build-cache --no-daemon` settings when
>>>>>>>>>>> invoking gradle, to avoid issues with our build. This kind of denies even
>>>>>>>>>>> any potential adventages of using Gradle over more stable build tools.
>>>>>>>>>>> Note, that I do not use these settings (so my issues are definetely not
>>>>>>>>>>> caused by this), but I keep getting this recommendation for a reason.
>>>>>>>>>>>
>>>>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>>>>>>> running gradle with `--console=plain` to get some chance to
>>>>>>>>>>> read the logs in a sensible way. Still, when eg. 2 tests fail out of the
>>>>>>>>>>> 1100+ on a Github build, it's a horror to find out which ones failed, and
>>>>>>>>>>> our best bet is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>>>
>>>>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point,
>>>>>>>>>>> check
>>>>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>>>>> and shows a couple of warnings below, but the actual error is
>>>>>>>>>>> shown at line 1393 which appears to be part of
>>>>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>>>>> which is clearly misleading, and your best bet to find the
>>>>>>>>>>> actual error on Github is to search for "error: ", but NOT
>>>>>>>>>>> using Ctrl+F, but the search box at the top right corner. That's ridiculous
>>>>>>>>>>> - a sensible build tool should communicate build errors in a
>>>>>>>>>>> straightforward way.
>>>>>>>>>>>
>>>>>>>>>>> I'd like to ask you to share your experiences and pain points
>>>>>>>>>>> with the current build that we have and use every day. Once we have a
>>>>>>>>>>> clearer understanding of the areas that need improvement, we can discuss
>>>>>>>>>>> potential strategies to revise our build system and ensure it meets basic
>>>>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Kristof
>>>>>>>>>>>
>>>>>>>>>>>

Re: Discussion on Build System improvements

[James Dailey <ja...@gmail.com>]

Awesome !

On Mon, Jan 20, 2025 at 12:30 PM Ádám Sághy <ad...@gmail.com> wrote:

> Hi Kristof,
>
> I really appreciate all the work you put into this.
>
> I am happy to say it is working, it's definitely building faster!
>
> I was using whole day these changes, built (after gradle clean) and
> rebuilt multiple times the backend, deployed locally back and forth and
> there was no error, no issues.
>
> Good job!
>
> Regards
> Adam
>
> Sent from my iPhone
>
> On 17 Jan 2025, at 22:15, Kristof Jozsa <kr...@gmail.com> wrote:
>
> 
> Hi all,
>
> I'd like to revive this thread. While I've been relatively quiet on the
> topic here on the mailing list, those who work with me in person know that
> I've spent countless hours over the past few months trying to address the
> most serious pain points in our ever-growing Gradle build. I won't bore you
> with all the details, but I’ve had several attempts—some that didn’t work,
> some that partially worked, and I had to restart from scratch several times
> also.
>
> In my most recent attempt, mostly during the holidays and since, I started
> fresh from a recent apache/develop HEAD with a narrowed focus:
> - enable caching for the longest-running, most critical tasks.
> - introduce the new :fineract-provider:devRun task
> - avoid breaking any existing functionality
>
> The second goal probably deserves some explanation. Most of us build and
> run Fineract *several times per hour* during development, typically using
> the :fineract-provider:bootRun task. I identified several plugins that
> run every time (e.g., Checkstyle, Spotless, ErrorProne, Javadoc, SpotBugs,
> and Modernizer) and decided to exclude them from the new
> :fineract-provider:devRun task. This new task is otherwise a clone of
> bootRun and should achieve the same outcome but with improved efficiency.
>
> To share some of my latest measurements with this PR, here are the rebuild
> times of the project (right after a successful build) excluding tests on
> both the current apache/develop HEAD and my branch for a direct
> comparison:
> *Rebuild on apache/develop:*
> BUILD SUCCESSFUL in 4m 20s
> 524 actionable tasks: 186 executed, 338 up-to-date
>
>
> *Rebuild on #devrun branch:*BUILD SUCCESSFUL in 27s
> 606 actionable tasks: 88 executed, 518 up-to-date
> You can also check the apache github page to see improved build times, see
> eg.
> https://github.com/apache/fineract/actions/workflows/build-docker-postgresql.yml
> - the Docker Postgres job's build time decreased from 13-14mins to 8m30s..
> but there are similar advantages across all build jobs triggered by the
> PR's build.
>
> This PR is currently in draft form and can be found here:
> https://github.com/apache/fineract/pull/4272. At this point, I’d like to *ask
> the community for help* with testing. Please pull the changes, test them,
> and let me know if they help or if they break anything. Experiment with
> using the new devRun task for local development and see if it improves
> your daily workflow. Feel free to share your feedback here on the mailing
> list or with me directly on the Mifos Slack for more interactive
> discussions.
>
> Please note that this PR is definitely a first step (though I believe a
> highly valuable one), and not the last one related to improving our build.
> There are still countless opportunities to simplify, clean up, remove
> obsolete dependencies/plugins, rewrite the build files to the new KTS
> format (I started, but it got abandoned along the way), etc.. Feel free to
> suggest additional improvements and let's prioritize.. And as always, any
> help is greatly appreciated.
>
> Thanks much,
> Kristof
>
>
> On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic <
> cheetah@monkeysintown.com> wrote:
>
>> ... well done Kristof! Thanks for taking the time and giving this in
>> depth view on the performance bottlenecks!
>>
>> Cheers
>>
>> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
>> wrote:
>>
>>> Today I had a very lengthy discussion in person with a core Gradle
>>> engineer. We looked at our Fineract build, and the various issues we
>>> gathered, and also looked for reasons and possible ways to get around them.
>>> Let me summarize the hints and suggestions we received:
>>>
>>> *- The swagger plugin in fineract-provider module*
>>> This one seems to be the root of one of our primary pain points. This is
>>> wired in the *fineract-provider* module, and it's done in a quite
>>> cumbersome way.. but for a reason. The reason seems to be that this plugin
>>> looks plain incorrect, because of its hard-wired dependencies, as clearly
>>> shown in its source at
>>> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
>>> The problem with this plugin and its wiring is that it forces Gradle to
>>> rebuild this module every time, and it's a very lengthy move..
>>>
>>> The suggested way would be to include swagger's generated resources
>>> using the processResources task, but approaching it that way causes a
>>> circular dependency, because the plugin's incorrectly wired dependencies..
>>> To get this fixed, we need to raise an issue at the plugin and get it fixed
>>> first. An alternative approach would be splitting the swagger resource
>>> generation to its own module inside Fineract, and accept the fact that it
>>> corrupts Gradle's cache mechanism (so that this new module would always get
>>> re-built in every case).
>>>
>>> The cache issue can also be seen in Develocity:
>>> [image: image.png]
>>>
>>> We concluded that I'll try to raise an issue at the Swagger Github about
>>> the plugins incorrect ways of doing things, and they'll also emphasize it
>>> from Gradle's side, hopefully getting it fixed mid-term.
>>>
>>> *- Console log's readabilty*
>>> Gradle team understands these concerns, and are working on improving
>>> these parts already. They are considering adding information to each line
>>> which helps to understand where the piece of information is coming from.
>>> That could help demystifing logs like this (and cleaning up the related
>>> parts in the build):
>>> [image: image.png]
>>> They are also considering gathering all error reports at the end of the
>>> builds (again), so we won't have to hunt the errors from the flood of 897
>>> warning messages. Speaking of which:
>>>
>>> *- Suggestion: remove warning messages for headless builds*
>>> We discussed the importance of warning messages, but their point is that
>>> either those are important (so could be errors as well), or not important,
>>> so we should silence them. At least for the headless builds running on CI
>>> servers, because no one will care about those warnings anyway. That doesn't
>>> perfectly fit my view, but for hundreds of warnings being there for many
>>> years, they have a point...
>>>
>>> *- IntelliJ Build View readability*
>>> They also consider improving the UI on IntelliJ's Build view, this one:
>>> [image: image.png]
>>> (today I learnt that lines nested under bootRun are exceptions gathered
>>> and picked up from the console logs.. Personally I think they are not
>>> useful at all, especially with all of the stacktrace removed from the
>>> detail view on the right side, and not related to the build either...)
>>>
>>> *- Suggestion: switch to the Kotlin DSL and clean up builds*
>>> Gradle team thinks the Kotlin DSL is the way forward, and we should
>>> seriously consider switching our build from Groovy to Kotlin, especially if
>>> we are aiming to clean up our build somewhat meanwhile. There are also
>>> parts which look deprecated and possibly no longer required, like
>>> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
>>> `println` statements while we're at it..
>>>
>>> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
>>> running*
>>> Our Fineract build tries to utilize the --scan feature already, which
>>> provides deep insights of what's happening, why it's happening and how long
>>> it takes. The only issue is that these scans get uploaded to
>>> ge.apache.org by default, and it's down... I got connected to another
>>> Gradle team member, who mentioned that ASF Infra hosts ge.apache.org,
>>> and there are recurring availability issues with it, but they (Gradle team)
>>> are happy to help. They also drafted a plan forward, but I'd prefer to
>>> discuss it personally with the AST team member usually taking care of this
>>> host.. @Aleksandar maybe you know who can touch these parts? Let's discuss
>>> this on Slack..
>>>
>>> That's all I could push these issues forward today. Hope that helps in
>>> the long run...
>>>
>>> Kristof
>>>
>>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
>>> cheetah@monkeysintown.com> wrote:
>>>
>>>> ... the performance issues stem from "dependsOn"... and similar...
>>>> there are also neat features in Gradle where you can precompile pretty much
>>>> all of the build script (and "configuration" cache it) which would boost
>>>> the build process by a lot (disclosure: I am using this technique in
>>>> projects with 3x of Fineract's project modules).
>>>>
>>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>>>> caching... which is really a big deal performance wise... BUT: to fully
>>>> take advantage of that feature all plugins used must support that (and of
>>>> course some don't). Still, configuration caching would be the "cherry on
>>>> top"... our main problem is "dependsOn" (explicit and implicit).
>>>>
>>>> I can see the declarative style of Maven (having used it since before
>>>> it was released 1.0)... but I would still bet my money on Gradle... and you
>>>> are right, it would be a lot of work to create a Maven build from scratch.
>>>>
>>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> I believe many of our issues stem from our current Gradle build being
>>>>> overly complex. One of my main concerns with Gradle is that it allows
>>>>> customization to evolve completely out of control. Following this
>>>>> principle, I would be opposed to adding new elements to our build.
>>>>>
>>>>> Instead, I’d like to see the current build configuration simplified as
>>>>> much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>>>> always preferred Maven’s declarative approach over the programmatic style
>>>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>>>> could dedicate some time to reducing the custom behavior in our build, I
>>>>> believe it would resolve many of the issues we’re currently facing.
>>>>> An alternative would be turning back to a more maintainable,
>>>>> declarative approach, but crafting a new build from scratch covering the
>>>>> complete current functionality would be a hell of a work..
>>>>>
>>>>> Kristof
>>>>>
>>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>>> victor.romero@fintecheando.mx> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> What about adding observability to the building process? In this way
>>>>>> we can have information about the root causes of the issues during the
>>>>>> building time.
>>>>>>
>>>>>> A OSS suggestion:
>>>>>>
>>>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Victor
>>>>>>
>>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<
>>>>>> jamespdailey@gmail.com>) escribió:
>>>>>>
>>>>>>> +1  agree
>>>>>>> How do we tackle this ?
>>>>>>>
>>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build,
>>>>>>>>> it's a horror to find out which ones failed, and our best bet is to
>>>>>>>>> text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> Items like this should be high priority in CI improvements epic.
>>>>>>>> The overall failed test history, flaky test history, log tracing problem
>>>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>>>
>>>>>>>> I had overestimated what Fineract modularization design efforts
>>>>>>>> have achieved. Would be interesting to know more about the design here as
>>>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>>>
>>>>>>>>> Kristof,
>>>>>>>>>
>>>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>>>
>>>>>>>>> Repeatable builds: we should consider this as a major bug and not
>>>>>>>>> just "gloss over" it by executing the build multiple times... stuff like
>>>>>>>>> this should halt all current development on other features and everyone (or
>>>>>>>>> at least one person) should focus on fixing this as soon as possible. I
>>>>>>>>> would consider this also a show stopper for a release.
>>>>>>>>>
>>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too
>>>>>>>>> bad here, but it seems that this is now actually worse than before... my
>>>>>>>>> bet is that we introduced some performance hogs by splitting up the code
>>>>>>>>> base into separate project modules; this didn't really fix the main problem
>>>>>>>>> of the source code (too many interdependencies between packages aka
>>>>>>>>> "modules"); just by splitting up the code as is and moving some classes to
>>>>>>>>> different project modules is not really taking into account the
>>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds),
>>>>>>>>> neither for publishing them on Maven central nor chopping up the code into
>>>>>>>>> smaller pieces and taking full advantage of caching.
>>>>>>>>>
>>>>>>>>> Caching: disabling caching is just not such a good idea... at
>>>>>>>>> all... again: I would consider this a major bug and halt all machines until
>>>>>>>>> it's fixed; also: consider this a show stopper for releases.
>>>>>>>>>
>>>>>>>>> Console readability: we've talked about this already on occasions
>>>>>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>>>>>> enforcements, because the reported style and quality checks literally
>>>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>>>> also avoiding to fix the quality issues.
>>>>>>>>>
>>>>>>>>> Suggestions:
>>>>>>>>>
>>>>>>>>>    - let's make an effort to reduce the interdependencies between
>>>>>>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>>>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>>>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>>>>>>    no added value of having classes in different jar files when it hurts build
>>>>>>>>>    performance
>>>>>>>>>    - let's spend time and fix those style and quality issues, not
>>>>>>>>>    just pile feature on feature on feature... I am pretty sure that everyone
>>>>>>>>>    can grab a handful of those warnings and fix them as part of their PRs
>>>>>>>>>    (even if it's not one that is explicitly dedicated to quality assurance),
>>>>>>>>>    in other words: everyone look a bit left and right to see what ELSE you can
>>>>>>>>>    fix/add in addition to your tasks (chores vs cherry picking)... fixing
>>>>>>>>>    everything in one PR is obviously not practical; we'll have a thousand
>>>>>>>>>    files changed... this will interrupt everyone else quite significantly and
>>>>>>>>>    is a nightmare to review
>>>>>>>>>    - there might be less than a handful of Gradle build
>>>>>>>>>    constraints that are again too paranoid and mess up the build; I think it's
>>>>>>>>>    worth looking at the OpenAPI yaml build... that one had issues in the past
>>>>>>>>>    with proper dirty checks, but there might be other/new constraints that are
>>>>>>>>>    equally unhelpful
>>>>>>>>>
>>>>>>>>> My 2 cents
>>>>>>>>>
>>>>>>>>> Aleks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>>>> build from the web UI.
>>>>>>>>>>
>>>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>>>
>>>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>>>> fails for the first time with:
>>>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>>>
>>>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>>>
>>>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>>>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>>>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>>>> development.
>>>>>>>>>>
>>>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>>>>>>> local builds.
>>>>>>>>>>
>>>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>>>> recommending `--no-build-cache --no-daemon` settings when
>>>>>>>>>> invoking gradle, to avoid issues with our build. This kind of denies even
>>>>>>>>>> any potential adventages of using Gradle over more stable build tools.
>>>>>>>>>> Note, that I do not use these settings (so my issues are definetely not
>>>>>>>>>> caused by this), but I keep getting this recommendation for a reason.
>>>>>>>>>>
>>>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>>>>>> running gradle with `--console=plain` to get some chance to read
>>>>>>>>>> the logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+
>>>>>>>>>> on a Github build, it's a horror to find out which ones failed, and our
>>>>>>>>>> best bet is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>>
>>>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point,
>>>>>>>>>> check
>>>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>>>> and shows a couple of warnings below, but the actual error is
>>>>>>>>>> shown at line 1393 which appears to be part of
>>>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>>>> which is clearly misleading, and your best bet to find the actual
>>>>>>>>>> error on Github is to search for "error: ", but NOT using
>>>>>>>>>> Ctrl+F, but the search box at the top right corner. That's ridiculous - a
>>>>>>>>>> sensible build tool should communicate build errors in a straightforward
>>>>>>>>>> way.
>>>>>>>>>>
>>>>>>>>>> I'd like to ask you to share your experiences and pain points
>>>>>>>>>> with the current build that we have and use every day. Once we have a
>>>>>>>>>> clearer understanding of the areas that need improvement, we can discuss
>>>>>>>>>> potential strategies to revise our build system and ensure it meets basic
>>>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Kristof
>>>>>>>>>>
>>>>>>>>>>

Re: Discussion on Build System improvements

[Ádám Sághy <ad...@gmail.com>]

Hi Kristof,

  

I really appreciate all the work you put into this.

  

I am happy to say it is working, it's definitely building faster!

  

I was using whole day these changes, built (after gradle clean) and rebuilt
multiple times the backend, deployed locally back and forth and there was no
error, no issues.

  

Good job!

  

Regards

Adam

  

Sent from my iPhone

  

> On 17 Jan 2025, at 22:15, Kristof Jozsa <kr...@gmail.com> wrote:  
>  
>

> 
>
> Hi all,  
>  
> I'd like to revive this thread. While I've been relatively quiet on the
> topic here on the mailing list, those who work with me in person know that
> I've spent countless hours over the past few months trying to address the
> most serious pain points in our ever-growing Gradle build. I won't bore you
> with all the details, but I’ve had several attempts—some that didn’t work,
> some that partially worked, and I had to restart from scratch several times
> also.  
>
>
>  
>
>
> In my most recent attempt, mostly during the holidays and since, I started
> fresh from a recent apache/develop HEAD with a narrowed focus:
>
> \- enable caching for the longest-running, most critical tasks.  
> \- introduce the new :fineract-provider:devRun task  
> \- avoid breaking any existing functionality  
>  
> The second goal probably deserves some explanation. Most of us build and run
> Fineract _several times per hour_ during development, typically using the
> :fineract-provider:bootRun task. I identified several plugins that run every
> time (e.g., Checkstyle, Spotless, ErrorProne, Javadoc, SpotBugs, and
> Modernizer) and decided to exclude them from the new :fineract-
> provider:devRun task. This new task is otherwise a clone of bootRun and
> should achieve the same outcome but with improved efficiency.  
>
>
>  
>
>
> To share some of my latest measurements with this PR, here are the rebuild
> times of the project (right after a successful build) excluding tests on
> both the current apache/develop HEAD and my branch for a direct comparison:
>
>  **Rebuild on`apache/develop`:**  
> BUILD SUCCESSFUL in 4m 20s  
> 524 actionable tasks: 186 executed, 338 up-to-date  
>
>
>  **Rebuild on`#devrun` branch:  
> **BUILD SUCCESSFUL in 27s  
> 606 actionable tasks: 88 executed, 518 up-to-date
>
> You can also check the apache github page to see improved build times, see
> eg. <https://github.com/apache/fineract/actions/workflows/build-docker-
> postgresql.yml> \- the Docker Postgres job's build time decreased from
> 13-14mins to 8m30s.. but there are similar advantages across all build jobs
> triggered by the PR's build.  
>
>
>  
>
>
> This PR is currently in draft form and can be found here:
> <https://github.com/apache/fineract/pull/4272>. At this point, I’d like to
> **ask the community for help** with testing. Please pull the changes, test
> them, and let me know if they help or if they break anything. Experiment
> with using the new devRun task for local development and see if it improves
> your daily workflow. Feel free to share your feedback here on the mailing
> list or with me directly on the Mifos Slack for more interactive
> discussions.
>
>  
>
>
> Please note that this PR is definitely a first step (though I believe a
> highly valuable one), and not the last one related to improving our build.
> There are still countless opportunities to simplify, clean up, remove
> obsolete dependencies/plugins, rewrite the build files to the new KTS format
> (I started, but it got abandoned along the way), etc.. Feel free to suggest
> additional improvements and let's prioritize.. And as always, any help is
> greatly appreciated.
>
>  
> Thanks much,  
>
>
> Kristof
>
>  
>
>
>  
>
>
> On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic
> <[cheetah@monkeysintown.com](mailto:cheetah@monkeysintown.com)> wrote:  
>
>

>> ... well done Kristof! Thanks for taking the time and giving this in depth
view on the performance bottlenecks!

>>

>>  
>
>>

>> Cheers  
>
>>

>>  
>
>>

>> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa
<[kristof.jozsa@gmail.com](mailto:kristof.jozsa@gmail.com)> wrote:  
>
>>

>>> Today I had a very lengthy discussion in person with a core Gradle
engineer. We looked at our Fineract build, and the various issues we gathered,
and also looked for reasons and possible ways to get around them. Let me
summarize the hints and suggestions we received:

>>>

>>>  
>
>>>

>>>  **\- The swagger plugin  in fineract-provider module**

>>>

>>> This one seems to be the root of one of our primary pain points. This is
wired in the  **fineract-provider** module, and it's done in a quite
cumbersome way.. but for a reason. The reason seems to be that this plugin
looks plain incorrect, because of its hard-wired dependencies, as clearly
shown in its source at <https://github.com/swagger-api/swagger-
core/blob/master/modules/swagger-gradle-
plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java>. The
problem with this plugin and its wiring is that it forces Gradle to rebuild
this module every time, and it's a very lengthy move..

>>>

>>>  
>
>>>

>>> The suggested way would be to include swagger's generated resources using
the processResources task, but approaching it that way causes a circular
dependency, because the plugin's incorrectly wired dependencies.. To get this
fixed, we need to raise an issue at the plugin and get it fixed first. An
alternative approach would be splitting the swagger resource generation to its
own module inside Fineract, and accept the fact that it corrupts Gradle's
cache mechanism (so that this new module would always get re-built in every
case).  
>>>

>>>  
>
>>>

>>> The cache issue can also be seen in Develocity:

>>>

>>> ![image.png](cid:ii_m10ts5hi0)

>>>

>>>  
>
>>>

>>> We concluded that I'll try to raise an issue at the Swagger Github about
the plugins incorrect ways of doing things, and they'll also emphasize it from
Gradle's side, hopefully getting it fixed mid-term.

>>>

>>>  
>
>>>

>>>  **\- Console log's  readabilty**

>>>

>>> Gradle team understands these concerns, and are working on improving these
parts already. They are considering adding information to each line which
helps to understand where the piece of information is coming from. That could
help demystifing logs like this (and cleaning up the related parts in the
build):  
> ![image.png](cid:ii_m10u275z1)
>>>

>>> They are also considering gathering all error reports at the end of the
builds (again), so we won't have to hunt the errors from the flood of 897
warning messages. Speaking of which:

>>>

>>>  
>
>>>

>>>  **\- Suggestion: remove warning messages for headless builds**

>>>

>>> We discussed the importance of warning messages, but their point is that
either those are important (so could be errors as well), or not important, so
we should silence them. At least for the headless builds running on CI
servers, because no one will care about those warnings anyway. That doesn't
perfectly fit my view, but for hundreds of warnings being there for many
years, they have a point...

>>>

>>>  
>
>>>

>>>  **\- IntelliJ Build View readability**

>>>

>>> They also consider improving the UI on IntelliJ's Build view, this one:  
> ![image.png](cid:ii_m10u4b7e2)
>>>

>>> (today I learnt that lines nested under bootRun are exceptions gathered
and picked up from the console logs.. Personally I think they are not useful
at all, especially with all of the stacktrace removed from the detail view on
the right side, and not related to the build either...)  
>
>>>

>>>  
>>>

>>>  **\- Suggestion: switch to the Kotlin DSL and clean up builds**

>>>

>>> Gradle team thinks the Kotlin DSL is the way forward, and we should
seriously consider switching our build from Groovy to Kotlin, especially if we
are aiming to clean up our build somewhat meanwhile. There are also parts
which look deprecated and possibly no longer required, like
`org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
`println` statements while we're at it..

>>>

>>>  
>
>>>

>>>  **\- Suggestion: get[ge.apache.org](http://ge.apache.org) back up and
running**

>>>

>>> Our Fineract build tries to utilize the --scan feature already, which
provides deep insights of what's happening, why it's happening and how long it
takes. The only issue is that these scans get uploaded to
[ge.apache.org](http://ge.apache.org) by default, and it's down... I got
connected to another Gradle team member, who mentioned that ASF Infra hosts
[ge.apache.org](http://ge.apache.org), and there are recurring availability
issues with it, but they (Gradle team) are happy to help. They also drafted a
plan forward, but I'd prefer to discuss it personally with the AST team member
usually taking care of this host.. @Aleksandar maybe you know who can touch
these parts? Let's discuss this on Slack..

>>>

>>>  
>
>>>

>>> That's all I could push these issues forward today. Hope that helps in the
long run...

>>>

>>>  
>
>>>

>>> Kristof

>>>

>>>  
>
>>>

>>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic
<[cheetah@monkeysintown.com](mailto:cheetah@monkeysintown.com)> wrote:  
>
>>>

>>>> ... the performance issues stem from "dependsOn"... and similar... there
are also neat features in Gradle where you can precompile pretty much all of
the build script (and "configuration" cache it) which would boost the build
process by a lot (disclosure: I am using this technique in projects with 3x of
Fineract's project modules).

>>>>

>>>>  
>
>>>>

>>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
caching... which is really a big deal performance wise... BUT: to fully take
advantage of that feature all plugins used must support that (and of course
some don't). Still, configuration caching would be the "cherry on top"... our
main problem is "dependsOn" (explicit and implicit).  
>
>>>>

>>>>  
>
>>>>

>>>> I can see the declarative style of Maven (having used it since before it
was released 1.0)... but I would still bet my money on Gradle... and you are
right, it would be a lot of work to create a Maven build from scratch.  
>
>>>>

>>>>  
>
>>>>

>>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa
<[kristof.jozsa@gmail.com](mailto:kristof.jozsa@gmail.com)> wrote:  
>
>>>>

>>>>> I believe many of our issues stem from our current Gradle build being
overly complex. One of my main concerns with Gradle is that it allows
customization to evolve completely out of control. Following this principle, I
would be opposed to adding new elements to our build.

>>>>>

>>>>> Instead, I’d like to see the current build configuration simplified as
much as possible. Unfortunately, I’m not well-versed in Gradle. I have always
preferred Maven’s declarative approach over the programmatic style of Gradle
(or Ant, for that matter). If any Gradle experts on the team could dedicate
some time to reducing the custom behavior in our build, I believe it would
resolve many of the issues we’re currently facing.

>>>>>

>>>>> An alternative would be turning back to a more maintainable, declarative
approach, but crafting a new build from scratch covering the complete current
functionality would be a hell of a work..

>>>>>

>>>>>  
>
>>>>>

>>>>> Kristof

>>>>>

>>>>>  
>
>>>>>

>>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ
<[victor.romero@fintecheando.mx](mailto:victor.romero@fintecheando.mx)> wrote:  
>
>>>>>

>>>>>> Hello,

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> What about adding observability to the building process? In this way we
can have information about the root causes of the issues during the building
time.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> A OSS suggestion:

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> <https://github.com/craigatk/opentelemetry-gradle-plugin>  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Regards

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Victor

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey
(<[jamespdailey@gmail.com](mailto:jamespdailey@gmail.com)>) escribió:  
>
>>>>>>

>>>>>>> +1  agree

>>>>>>>

>>>>>>> How do we tackle this ?  
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly
<[avik@fynarfin.io](mailto:avik@fynarfin.io)> wrote:  
>
>>>>>>>

>>>>>>>> > Still, when eg. 2 tests fail out of the 1100+ on a Github build,
it's a horror to find out which ones failed, and our best bet is to text-
search for "FAILED" in the 17000+ lines of output.

>>>>>>>>

>>>>>>>> +1

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Items like this should be high priority in CI improvements epic. The
overall failed test history, flaky test history, log tracing problem (actual
result is often not sufficient to quickly debug and linking logs to a failed
test case is important) were historically connected to getting PRs merged in
time. CircleCI has helped us with the first two parts of this problem.  
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>>> just by splitting up the code as is and moving some classes to
different project modules is not really taking into account the
interdependencies... if we don't address those then the project modules are
not really helpful (actually it's adds more work to the Gradle builds)

>>>>>>>>

>>>>>>>> I had overestimated what Fineract modularization design efforts have
achieved. Would be interesting to know more about the design here as
"splitting up the code as is" can be helped with more clarity.

>>>>>>>>

>>>>>>>>  
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic
<[cheetah@monkeysintown.com](mailto:cheetah@monkeysintown.com)> wrote:  
>
>>>>>>>>

>>>>>>>>> Kristof,

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> ... great overview, so first: thanks for that. Some notes...

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Repeatable builds: we should consider this as a major bug and not
just "gloss over" it by executing the build multiple times... stuff like this
should halt all current development on other features and everyone (or at
least one person) should focus on fixing this as soon as possible. I would
consider this also a show stopper for a release.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
here, but it seems that this is now actually worse than before... my bet is
that we introduced some performance hogs by splitting up the code base into
separate project modules; this didn't really fix the main problem of the
source code (too many interdependencies between packages aka "modules"); just
by splitting up the code as is and moving some classes to different project
modules is not really taking into account the interdependencies... if we don't
address those then the project modules are not really helpful (actually it's
adds more work to the Gradle builds), neither for publishing them on Maven
central nor chopping up the code into smaller pieces and taking full advantage
of caching.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Caching: disabling caching is just not such a good idea... at all...
again: I would consider this a major bug and halt all machines until it's
fixed; also: consider this a show stopper for releases.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Console readability: we've talked about this already on occasions
(there might be even a mailing list message)... on every upgrade of the
dependencies (including the quality assurance tools like Errorprone etc.) the
screws get tightened (read: rules get stricter); I remember that after one
upgrade (maybe last year, maybe longer ago) I had to disable some rule
enforcements, because the reported style and quality checks literally
exploded; holding back on Errorprone (and similar) upgrades is obviously not a
solution; I tried to be conservative (just disabled those that brought the
build to a halt aka that were reported as errors), but as we can see there are
enough warnings left to drive everyone crazy; btw: disabling the rules with
certain annotations is neither a solution, that's also avoiding to fix the
quality issues.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Suggestions:

>>>>>>>>>

>>>>>>>>>   * let's make an effort to reduce the interdependencies between
packages aka "modules" before we rip the code apart and move it to separate
project modules; if we can't do that (backwards compatibility, not allowed to
"break eggs" etc.) then let's leave the code in one place as is... I see no
added value of having classes in different jar files when it hurts build
performance

>>>>>>>>>   * let's spend time and fix those style and quality issues, not
just pile feature on feature on feature... I am pretty sure that everyone can
grab a handful of those warnings and fix them as part of their PRs (even if
it's not one that is explicitly dedicated to quality assurance), in other
words: everyone look a bit left and right to see what ELSE you can fix/add in
addition to your tasks (chores vs cherry picking)... fixing everything in one
PR is obviously not practical; we'll have a thousand files changed... this
will interrupt everyone else quite significantly and is a nightmare to review

>>>>>>>>>   * there might be less than a handful of Gradle build constraints
that are again too paranoid and mess up the build; I think it's worth looking
at the OpenAPI yaml build... that one had issues in the past with proper dirty
checks, but there might be other/new constraints that are equally unhelpful  
>
>>>>>>>>>

>>>>>>>>>

>>>>>>>>> My 2 cents

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Aleks  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa
<[kristof.jozsa@gmail.com](mailto:kristof.jozsa@gmail.com)> wrote:  
>
>>>>>>>>>

>>>>>>>>>> Hi all,  
>  
> After working with Fineract for some months now, I collected several issues
> related to the Gradle build system we have in place. I categorized my pain
> points, and would like to initiate a discussion on the topic with the
> community. My usage scenarios include building locally with IntelliJ+Gradle
> wrapper, building locally from CLI and using the Github build from the web
> UI.  
>  
> The problems I gathered fall into the following categories:  
>  
> \- **REPEATABLE BUILDS:** starting Fineract from IntelliJ always fails for
> the first time with:  
> Caused by: java.lang.IllegalArgumentException: Couldn't bind
> FineractProperties to the 'fineract' property  
> and simply trying it again always makes it wor0k.  
>  
> \- **DIRTY CHECKS** : modifying a single file (eg. put a newline character
> into the Loan class) and restarting Fineract makes it "build" for 17-18
> seconds (checking all modules to be up-to-date) before starting it again.
> This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating up to
> 800% CPU and spinning up all the vents in my rig while doing so.  
>  
> \- **INTEGRATION TESTS** : the same dirty check problem happens when working
> with tests as well. Adding a blank line to an existing test class and re-
> running the test again happens after a ~20 seconds "rebuild" process. This
> leads to painfully slow code-test cycles at local development.  
>  
> \- **RESOURCE REQUIREMENTS** : local team reports that in cases the Gradle
> daemon repeatedly grows up to 8Gb memory usage. That causes issues with dev
> boxes having 16Gb or even less RAM.. Even using my high-spec dev box, I
> needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
> -Dorg.gradle.priority=low" to keep my desktop responsible during local
> builds.  
>  
> \- **CACHING** : I see many team members and mailing list members
> recommending `\--no-build-cache --no-daemon` settings when invoking gradle,
> to avoid issues with our build. This kind of denies even any potential
> adventages of using Gradle over more stable build tools. Note, that I do not
> use these settings (so my issues are definetely not caused by this), but I
> keep getting this recommendation for a reason.  
>  
> \- **CONSOLE READABILITY** : I also see many fellow devs suggesting running
> gradle with `\--console=plain` to get some chance to read the logs in a
> sensible way. Still, when eg. 2 tests fail out of the 1100+ on a Github
> build, it's a horror to find out which ones failed, and our best bet is to
> text-search for "FAILED" in the 17000+ lines of output.  
>  
> \- **REPORTING COMPILE ERRORS** : related to the previous point, check
> <https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682>
> for a perfect example. Here at line 1489 Gradle shows  
> > Task :fineract-loan:compileJava FAILED  
> and shows a couple of warnings below, but the actual error is shown at line
> 1393 which appears to be part of  
> > Task :fineract-document:modernizer  
> which is clearly misleading, and your best bet to find the actual error on
> Github is to search for "error: ", but NOT using Ctrl+F, but the search box
> at the top right corner. That's ridiculous - a sensible build tool should
> communicate build errors in a straightforward way.  
>  
> I'd like to ask you to share your experiences and pain points with the
> current build that we have and use every day. Once we have a clearer
> understanding of the areas that need improvement, we can discuss potential
> strategies to revise our build system and ensure it meets basic
> requirements.To be fair, please also make sure you share any recent positive
> experiences related to Fineract builds, if you have any.  
>  
> Thanks,  
> Kristof  
>
>>>>>>>>>>

>>>>>>>>>>  
>

Re: Discussion on Build System improvements

[Kristof Jozsa <kr...@gmail.com>]

Hi all,

I'd like to revive this thread. While I've been relatively quiet on the
topic here on the mailing list, those who work with me in person know that
I've spent countless hours over the past few months trying to address the
most serious pain points in our ever-growing Gradle build. I won't bore you
with all the details, but I’ve had several attempts—some that didn’t work,
some that partially worked, and I had to restart from scratch several times
also.

In my most recent attempt, mostly during the holidays and since, I started
fresh from a recent apache/develop HEAD with a narrowed focus:
- enable caching for the longest-running, most critical tasks.
- introduce the new :fineract-provider:devRun task
- avoid breaking any existing functionality

The second goal probably deserves some explanation. Most of us build and
run Fineract *several times per hour* during development, typically using
the :fineract-provider:bootRun task. I identified several plugins that run
every time (e.g., Checkstyle, Spotless, ErrorProne, Javadoc, SpotBugs, and
Modernizer) and decided to exclude them from the new
:fineract-provider:devRun task. This new task is otherwise a clone of
bootRun and should achieve the same outcome but with improved efficiency.

To share some of my latest measurements with this PR, here are the rebuild
times of the project (right after a successful build) excluding tests on
both the current apache/develop HEAD and my branch for a direct comparison:
*Rebuild on apache/develop:*
BUILD SUCCESSFUL in 4m 20s
524 actionable tasks: 186 executed, 338 up-to-date


*Rebuild on #devrun branch:*BUILD SUCCESSFUL in 27s
606 actionable tasks: 88 executed, 518 up-to-date
You can also check the apache github page to see improved build times, see
eg.
https://github.com/apache/fineract/actions/workflows/build-docker-postgresql.yml
- the Docker Postgres job's build time decreased from 13-14mins to 8m30s..
but there are similar advantages across all build jobs triggered by the
PR's build.

This PR is currently in draft form and can be found here:
https://github.com/apache/fineract/pull/4272. At this point, I’d like to *ask
the community for help* with testing. Please pull the changes, test them,
and let me know if they help or if they break anything. Experiment with
using the new devRun task for local development and see if it improves your
daily workflow. Feel free to share your feedback here on the mailing list
or with me directly on the Mifos Slack for more interactive discussions.

Please note that this PR is definitely a first step (though I believe a
highly valuable one), and not the last one related to improving our build.
There are still countless opportunities to simplify, clean up, remove
obsolete dependencies/plugins, rewrite the build files to the new KTS
format (I started, but it got abandoned along the way), etc.. Feel free to
suggest additional improvements and let's prioritize.. And as always, any
help is greatly appreciated.

Thanks much,
Kristof


On Fri, Sep 13, 2024 at 5:37 PM Aleksandar Vidakovic <
cheetah@monkeysintown.com> wrote:

> ... well done Kristof! Thanks for taking the time and giving this in depth
> view on the performance bottlenecks!
>
> Cheers
>
> On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
> wrote:
>
>> Today I had a very lengthy discussion in person with a core Gradle
>> engineer. We looked at our Fineract build, and the various issues we
>> gathered, and also looked for reasons and possible ways to get around them.
>> Let me summarize the hints and suggestions we received:
>>
>> *- The swagger plugin in fineract-provider module*
>> This one seems to be the root of one of our primary pain points. This is
>> wired in the *fineract-provider* module, and it's done in a quite
>> cumbersome way.. but for a reason. The reason seems to be that this plugin
>> looks plain incorrect, because of its hard-wired dependencies, as clearly
>> shown in its source at
>> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
>> The problem with this plugin and its wiring is that it forces Gradle to
>> rebuild this module every time, and it's a very lengthy move..
>>
>> The suggested way would be to include swagger's generated resources using
>> the processResources task, but approaching it that way causes a circular
>> dependency, because the plugin's incorrectly wired dependencies.. To get
>> this fixed, we need to raise an issue at the plugin and get it fixed first.
>> An alternative approach would be splitting the swagger resource generation
>> to its own module inside Fineract, and accept the fact that it corrupts
>> Gradle's cache mechanism (so that this new module would always get re-built
>> in every case).
>>
>> The cache issue can also be seen in Develocity:
>> [image: image.png]
>>
>> We concluded that I'll try to raise an issue at the Swagger Github about
>> the plugins incorrect ways of doing things, and they'll also emphasize it
>> from Gradle's side, hopefully getting it fixed mid-term.
>>
>> *- Console log's readabilty*
>> Gradle team understands these concerns, and are working on improving
>> these parts already. They are considering adding information to each line
>> which helps to understand where the piece of information is coming from.
>> That could help demystifing logs like this (and cleaning up the related
>> parts in the build):
>> [image: image.png]
>> They are also considering gathering all error reports at the end of the
>> builds (again), so we won't have to hunt the errors from the flood of 897
>> warning messages. Speaking of which:
>>
>> *- Suggestion: remove warning messages for headless builds*
>> We discussed the importance of warning messages, but their point is that
>> either those are important (so could be errors as well), or not important,
>> so we should silence them. At least for the headless builds running on CI
>> servers, because no one will care about those warnings anyway. That doesn't
>> perfectly fit my view, but for hundreds of warnings being there for many
>> years, they have a point...
>>
>> *- IntelliJ Build View readability*
>> They also consider improving the UI on IntelliJ's Build view, this one:
>> [image: image.png]
>> (today I learnt that lines nested under bootRun are exceptions gathered
>> and picked up from the console logs.. Personally I think they are not
>> useful at all, especially with all of the stacktrace removed from the
>> detail view on the right side, and not related to the build either...)
>>
>> *- Suggestion: switch to the Kotlin DSL and clean up builds*
>> Gradle team thinks the Kotlin DSL is the way forward, and we should
>> seriously consider switching our build from Groovy to Kotlin, especially if
>> we are aiming to clean up our build somewhat meanwhile. There are also
>> parts which look deprecated and possibly no longer required, like
>> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
>> `println` statements while we're at it..
>>
>> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
>> running*
>> Our Fineract build tries to utilize the --scan feature already, which
>> provides deep insights of what's happening, why it's happening and how long
>> it takes. The only issue is that these scans get uploaded to
>> ge.apache.org by default, and it's down... I got connected to another
>> Gradle team member, who mentioned that ASF Infra hosts ge.apache.org,
>> and there are recurring availability issues with it, but they (Gradle team)
>> are happy to help. They also drafted a plan forward, but I'd prefer to
>> discuss it personally with the AST team member usually taking care of this
>> host.. @Aleksandar maybe you know who can touch these parts? Let's discuss
>> this on Slack..
>>
>> That's all I could push these issues forward today. Hope that helps in
>> the long run...
>>
>> Kristof
>>
>> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
>> cheetah@monkeysintown.com> wrote:
>>
>>> ... the performance issues stem from "dependsOn"... and similar... there
>>> are also neat features in Gradle where you can precompile pretty much all
>>> of the build script (and "configuration" cache it) which would boost the
>>> build process by a lot (disclosure: I am using this technique in projects
>>> with 3x of Fineract's project modules).
>>>
>>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>>> caching... which is really a big deal performance wise... BUT: to fully
>>> take advantage of that feature all plugins used must support that (and of
>>> course some don't). Still, configuration caching would be the "cherry on
>>> top"... our main problem is "dependsOn" (explicit and implicit).
>>>
>>> I can see the declarative style of Maven (having used it since before it
>>> was released 1.0)... but I would still bet my money on Gradle... and you
>>> are right, it would be a lot of work to create a Maven build from scratch.
>>>
>>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
>>> wrote:
>>>
>>>> I believe many of our issues stem from our current Gradle build being
>>>> overly complex. One of my main concerns with Gradle is that it allows
>>>> customization to evolve completely out of control. Following this
>>>> principle, I would be opposed to adding new elements to our build.
>>>>
>>>> Instead, I’d like to see the current build configuration simplified as
>>>> much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>>> always preferred Maven’s declarative approach over the programmatic style
>>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>>> could dedicate some time to reducing the custom behavior in our build, I
>>>> believe it would resolve many of the issues we’re currently facing.
>>>> An alternative would be turning back to a more maintainable,
>>>> declarative approach, but crafting a new build from scratch covering the
>>>> complete current functionality would be a hell of a work..
>>>>
>>>> Kristof
>>>>
>>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>>> victor.romero@fintecheando.mx> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> What about adding observability to the building process? In this way
>>>>> we can have information about the root causes of the issues during the
>>>>> building time.
>>>>>
>>>>> A OSS suggestion:
>>>>>
>>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>>
>>>>> Regards
>>>>>
>>>>> Victor
>>>>>
>>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> +1  agree
>>>>>> How do we tackle this ?
>>>>>>
>>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>>>>>>
>>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build,
>>>>>>>> it's a horror to find out which ones failed, and our best bet is to
>>>>>>>> text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Items like this should be high priority in CI improvements epic. The
>>>>>>> overall failed test history, flaky test history, log tracing problem
>>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>>> problem.
>>>>>>>
>>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>>> different project modules is not really taking into account the
>>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>>
>>>>>>> I had overestimated what Fineract modularization design efforts have
>>>>>>> achieved. Would be interesting to know more about the design here as
>>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>>
>>>>>>>> Kristof,
>>>>>>>>
>>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>>
>>>>>>>> Repeatable builds: we should consider this as a major bug and not
>>>>>>>> just "gloss over" it by executing the build multiple times... stuff like
>>>>>>>> this should halt all current development on other features and everyone (or
>>>>>>>> at least one person) should focus on fixing this as soon as possible. I
>>>>>>>> would consider this also a show stopper for a release.
>>>>>>>>
>>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>>>>>>> here, but it seems that this is now actually worse than before... my bet is
>>>>>>>> that we introduced some performance hogs by splitting up the code base into
>>>>>>>> separate project modules; this didn't really fix the main problem of the
>>>>>>>> source code (too many interdependencies between packages aka "modules");
>>>>>>>> just by splitting up the code as is and moving some classes to different
>>>>>>>> project modules is not really taking into account the interdependencies...
>>>>>>>> if we don't address those then the project modules are not really helpful
>>>>>>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>>>>>>> them on Maven central nor chopping up the code into smaller pieces and
>>>>>>>> taking full advantage of caching.
>>>>>>>>
>>>>>>>> Caching: disabling caching is just not such a good idea... at
>>>>>>>> all... again: I would consider this a major bug and halt all machines until
>>>>>>>> it's fixed; also: consider this a show stopper for releases.
>>>>>>>>
>>>>>>>> Console readability: we've talked about this already on occasions
>>>>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>>>>> enforcements, because the reported style and quality checks literally
>>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>>> also avoiding to fix the quality issues.
>>>>>>>>
>>>>>>>> Suggestions:
>>>>>>>>
>>>>>>>>    - let's make an effort to reduce the interdependencies between
>>>>>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>>>>>    no added value of having classes in different jar files when it hurts build
>>>>>>>>    performance
>>>>>>>>    - let's spend time and fix those style and quality issues, not
>>>>>>>>    just pile feature on feature on feature... I am pretty sure that everyone
>>>>>>>>    can grab a handful of those warnings and fix them as part of their PRs
>>>>>>>>    (even if it's not one that is explicitly dedicated to quality assurance),
>>>>>>>>    in other words: everyone look a bit left and right to see what ELSE you can
>>>>>>>>    fix/add in addition to your tasks (chores vs cherry picking)... fixing
>>>>>>>>    everything in one PR is obviously not practical; we'll have a thousand
>>>>>>>>    files changed... this will interrupt everyone else quite significantly and
>>>>>>>>    is a nightmare to review
>>>>>>>>    - there might be less than a handful of Gradle build
>>>>>>>>    constraints that are again too paranoid and mess up the build; I think it's
>>>>>>>>    worth looking at the OpenAPI yaml build... that one had issues in the past
>>>>>>>>    with proper dirty checks, but there might be other/new constraints that are
>>>>>>>>    equally unhelpful
>>>>>>>>
>>>>>>>> My 2 cents
>>>>>>>>
>>>>>>>> Aleks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>>> build from the web UI.
>>>>>>>>>
>>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>>
>>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>>> fails for the first time with:
>>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>>
>>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>>
>>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>>> development.
>>>>>>>>>
>>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>>>>>> local builds.
>>>>>>>>>
>>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>>> recommending `--no-build-cache --no-daemon` settings when
>>>>>>>>> invoking gradle, to avoid issues with our build. This kind of denies even
>>>>>>>>> any potential adventages of using Gradle over more stable build tools.
>>>>>>>>> Note, that I do not use these settings (so my issues are definetely not
>>>>>>>>> caused by this), but I keep getting this recommendation for a reason.
>>>>>>>>>
>>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>>>>> running gradle with `--console=plain` to get some chance to read
>>>>>>>>> the logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+
>>>>>>>>> on a Github build, it's a horror to find out which ones failed, and our
>>>>>>>>> best bet is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>>
>>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point,
>>>>>>>>> check
>>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>>> and shows a couple of warnings below, but the actual error is
>>>>>>>>> shown at line 1393 which appears to be part of
>>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>>> which is clearly misleading, and your best bet to find the actual
>>>>>>>>> error on Github is to search for "error: ", but NOT using Ctrl+F,
>>>>>>>>> but the search box at the top right corner. That's ridiculous - a sensible
>>>>>>>>> build tool should communicate build errors in a straightforward way.
>>>>>>>>>
>>>>>>>>> I'd like to ask you to share your experiences and pain points with
>>>>>>>>> the current build that we have and use every day. Once we have a clearer
>>>>>>>>> understanding of the areas that need improvement, we can discuss potential
>>>>>>>>> strategies to revise our build system and ensure it meets basic
>>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kristof
>>>>>>>>>
>>>>>>>>>

Re: Discussion on Build System improvements

[Aleksandar Vidakovic <ch...@monkeysintown.com>]

... well done Kristof! Thanks for taking the time and giving this in depth
view on the performance bottlenecks!

Cheers

On Fri, Sep 13, 2024 at 5:31 PM Kristof Jozsa <kr...@gmail.com>
wrote:

> Today I had a very lengthy discussion in person with a core Gradle
> engineer. We looked at our Fineract build, and the various issues we
> gathered, and also looked for reasons and possible ways to get around them.
> Let me summarize the hints and suggestions we received:
>
> *- The swagger plugin in fineract-provider module*
> This one seems to be the root of one of our primary pain points. This is
> wired in the *fineract-provider* module, and it's done in a quite
> cumbersome way.. but for a reason. The reason seems to be that this plugin
> looks plain incorrect, because of its hard-wired dependencies, as clearly
> shown in its source at
> https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
> The problem with this plugin and its wiring is that it forces Gradle to
> rebuild this module every time, and it's a very lengthy move..
>
> The suggested way would be to include swagger's generated resources using
> the processResources task, but approaching it that way causes a circular
> dependency, because the plugin's incorrectly wired dependencies.. To get
> this fixed, we need to raise an issue at the plugin and get it fixed first.
> An alternative approach would be splitting the swagger resource generation
> to its own module inside Fineract, and accept the fact that it corrupts
> Gradle's cache mechanism (so that this new module would always get re-built
> in every case).
>
> The cache issue can also be seen in Develocity:
> [image: image.png]
>
> We concluded that I'll try to raise an issue at the Swagger Github about
> the plugins incorrect ways of doing things, and they'll also emphasize it
> from Gradle's side, hopefully getting it fixed mid-term.
>
> *- Console log's readabilty*
> Gradle team understands these concerns, and are working on improving these
> parts already. They are considering adding information to each line which
> helps to understand where the piece of information is coming from. That
> could help demystifing logs like this (and cleaning up the related parts in
> the build):
> [image: image.png]
> They are also considering gathering all error reports at the end of the
> builds (again), so we won't have to hunt the errors from the flood of 897
> warning messages. Speaking of which:
>
> *- Suggestion: remove warning messages for headless builds*
> We discussed the importance of warning messages, but their point is that
> either those are important (so could be errors as well), or not important,
> so we should silence them. At least for the headless builds running on CI
> servers, because no one will care about those warnings anyway. That doesn't
> perfectly fit my view, but for hundreds of warnings being there for many
> years, they have a point...
>
> *- IntelliJ Build View readability*
> They also consider improving the UI on IntelliJ's Build view, this one:
> [image: image.png]
> (today I learnt that lines nested under bootRun are exceptions gathered
> and picked up from the console logs.. Personally I think they are not
> useful at all, especially with all of the stacktrace removed from the
> detail view on the right side, and not related to the build either...)
>
> *- Suggestion: switch to the Kotlin DSL and clean up builds*
> Gradle team thinks the Kotlin DSL is the way forward, and we should
> seriously consider switching our build from Groovy to Kotlin, especially if
> we are aiming to clean up our build somewhat meanwhile. There are also
> parts which look deprecated and possibly no longer required, like
> `org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
> `println` statements while we're at it..
>
> *- Suggestion: get ge.apache.org <http://ge.apache.org> back up and
> running*
> Our Fineract build tries to utilize the --scan feature already, which
> provides deep insights of what's happening, why it's happening and how long
> it takes. The only issue is that these scans get uploaded to ge.apache.org
> by default, and it's down... I got connected to another Gradle team member,
> who mentioned that ASF Infra hosts ge.apache.org, and there are recurring
> availability issues with it, but they (Gradle team) are happy to help. They
> also drafted a plan forward, but I'd prefer to discuss it personally with
> the AST team member usually taking care of this host.. @Aleksandar maybe
> you know who can touch these parts? Let's discuss this on Slack..
>
> That's all I could push these issues forward today. Hope that helps in the
> long run...
>
> Kristof
>
> On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
> cheetah@monkeysintown.com> wrote:
>
>> ... the performance issues stem from "dependsOn"... and similar... there
>> are also neat features in Gradle where you can precompile pretty much all
>> of the build script (and "configuration" cache it) which would boost the
>> build process by a lot (disclosure: I am using this technique in projects
>> with 3x of Fineract's project modules).
>>
>> Note: in version(s) 7.x (I believe) Gradle introduced configuration
>> caching... which is really a big deal performance wise... BUT: to fully
>> take advantage of that feature all plugins used must support that (and of
>> course some don't). Still, configuration caching would be the "cherry on
>> top"... our main problem is "dependsOn" (explicit and implicit).
>>
>> I can see the declarative style of Maven (having used it since before it
>> was released 1.0)... but I would still bet my money on Gradle... and you
>> are right, it would be a lot of work to create a Maven build from scratch.
>>
>> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
>> wrote:
>>
>>> I believe many of our issues stem from our current Gradle build being
>>> overly complex. One of my main concerns with Gradle is that it allows
>>> customization to evolve completely out of control. Following this
>>> principle, I would be opposed to adding new elements to our build.
>>>
>>> Instead, I’d like to see the current build configuration simplified as
>>> much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>>> always preferred Maven’s declarative approach over the programmatic style
>>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>>> could dedicate some time to reducing the custom behavior in our build, I
>>> believe it would resolve many of the issues we’re currently facing.
>>> An alternative would be turning back to a more maintainable, declarative
>>> approach, but crafting a new build from scratch covering the complete
>>> current functionality would be a hell of a work..
>>>
>>> Kristof
>>>
>>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>> victor.romero@fintecheando.mx> wrote:
>>>
>>>> Hello,
>>>>
>>>> What about adding observability to the building process? In this way
>>>> we can have information about the root causes of the issues during the
>>>> building time.
>>>>
>>>> A OSS suggestion:
>>>>
>>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>>
>>>> Regards
>>>>
>>>> Victor
>>>>
>>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
>>>> escribió:
>>>>
>>>>> +1  agree
>>>>> How do we tackle this ?
>>>>>
>>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>>>>>
>>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's
>>>>>>> a horror to find out which ones failed, and our best bet is to text-search
>>>>>>> for "FAILED" in the 17000+ lines of output.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> Items like this should be high priority in CI improvements epic. The
>>>>>> overall failed test history, flaky test history, log tracing problem
>>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>>> a failed test case is important) were historically connected to getting PRs
>>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>>> problem.
>>>>>>
>>>>>> just by splitting up the code as is and moving some classes to
>>>>>>> different project modules is not really taking into account the
>>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>>
>>>>>> I had overestimated what Fineract modularization design efforts have
>>>>>> achieved. Would be interesting to know more about the design here as
>>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>>> cheetah@monkeysintown.com> wrote:
>>>>>>
>>>>>>> Kristof,
>>>>>>>
>>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>>
>>>>>>> Repeatable builds: we should consider this as a major bug and not
>>>>>>> just "gloss over" it by executing the build multiple times... stuff like
>>>>>>> this should halt all current development on other features and everyone (or
>>>>>>> at least one person) should focus on fixing this as soon as possible. I
>>>>>>> would consider this also a show stopper for a release.
>>>>>>>
>>>>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>>>>>> here, but it seems that this is now actually worse than before... my bet is
>>>>>>> that we introduced some performance hogs by splitting up the code base into
>>>>>>> separate project modules; this didn't really fix the main problem of the
>>>>>>> source code (too many interdependencies between packages aka "modules");
>>>>>>> just by splitting up the code as is and moving some classes to different
>>>>>>> project modules is not really taking into account the interdependencies...
>>>>>>> if we don't address those then the project modules are not really helpful
>>>>>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>>>>>> them on Maven central nor chopping up the code into smaller pieces and
>>>>>>> taking full advantage of caching.
>>>>>>>
>>>>>>> Caching: disabling caching is just not such a good idea... at all...
>>>>>>> again: I would consider this a major bug and halt all machines until it's
>>>>>>> fixed; also: consider this a show stopper for releases.
>>>>>>>
>>>>>>> Console readability: we've talked about this already on occasions
>>>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>>>> enforcements, because the reported style and quality checks literally
>>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>>> also avoiding to fix the quality issues.
>>>>>>>
>>>>>>> Suggestions:
>>>>>>>
>>>>>>>    - let's make an effort to reduce the interdependencies between
>>>>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>>>>    no added value of having classes in different jar files when it hurts build
>>>>>>>    performance
>>>>>>>    - let's spend time and fix those style and quality issues, not
>>>>>>>    just pile feature on feature on feature... I am pretty sure that everyone
>>>>>>>    can grab a handful of those warnings and fix them as part of their PRs
>>>>>>>    (even if it's not one that is explicitly dedicated to quality assurance),
>>>>>>>    in other words: everyone look a bit left and right to see what ELSE you can
>>>>>>>    fix/add in addition to your tasks (chores vs cherry picking)... fixing
>>>>>>>    everything in one PR is obviously not practical; we'll have a thousand
>>>>>>>    files changed... this will interrupt everyone else quite significantly and
>>>>>>>    is a nightmare to review
>>>>>>>    - there might be less than a handful of Gradle build constraints
>>>>>>>    that are again too paranoid and mess up the build; I think it's worth
>>>>>>>    looking at the OpenAPI yaml build... that one had issues in the past with
>>>>>>>    proper dirty checks, but there might be other/new constraints that are
>>>>>>>    equally unhelpful
>>>>>>>
>>>>>>> My 2 cents
>>>>>>>
>>>>>>> Aleks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <
>>>>>>> kristof.jozsa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> After working with Fineract for some months now, I collected
>>>>>>>> several issues related to the Gradle build system we have in place. I
>>>>>>>> categorized my pain points, and would like to initiate a discussion on the
>>>>>>>> topic with the community. My usage scenarios include building locally with
>>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>>> build from the web UI.
>>>>>>>>
>>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>>
>>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always
>>>>>>>> fails for the first time with:
>>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>>> FineractProperties to the 'fineract' property
>>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>>
>>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>>
>>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>>> development.
>>>>>>>>
>>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>>>>> local builds.
>>>>>>>>
>>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>>>>>>> gradle, to avoid issues with our build. This kind of denies even any
>>>>>>>> potential adventages of using Gradle over more stable build tools. Note,
>>>>>>>> that I do not use these settings (so my issues are definetely not caused by
>>>>>>>> this), but I keep getting this recommendation for a reason.
>>>>>>>>
>>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>>>> running gradle with `--console=plain` to get some chance to read
>>>>>>>> the logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+
>>>>>>>> on a Github build, it's a horror to find out which ones failed, and our
>>>>>>>> best bet is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>>
>>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>>> and shows a couple of warnings below, but the actual error is shown
>>>>>>>> at line 1393 which appears to be part of
>>>>>>>> > Task :fineract-document:modernizer
>>>>>>>> which is clearly misleading, and your best bet to find the actual
>>>>>>>> error on Github is to search for "error: ", but NOT using Ctrl+F,
>>>>>>>> but the search box at the top right corner. That's ridiculous - a sensible
>>>>>>>> build tool should communicate build errors in a straightforward way.
>>>>>>>>
>>>>>>>> I'd like to ask you to share your experiences and pain points with
>>>>>>>> the current build that we have and use every day. Once we have a clearer
>>>>>>>> understanding of the areas that need improvement, we can discuss potential
>>>>>>>> strategies to revise our build system and ensure it meets basic
>>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kristof
>>>>>>>>
>>>>>>>>

Re: Discussion on Build System improvements

[Kristof Jozsa <kr...@gmail.com>]

Today I had a very lengthy discussion in person with a core Gradle
engineer. We looked at our Fineract build, and the various issues we
gathered, and also looked for reasons and possible ways to get around them.
Let me summarize the hints and suggestions we received:

*- The swagger plugin in fineract-provider module*
This one seems to be the root of one of our primary pain points. This is
wired in the *fineract-provider* module, and it's done in a quite
cumbersome way.. but for a reason. The reason seems to be that this plugin
looks plain incorrect, because of its hard-wired dependencies, as clearly
shown in its source at
https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-gradle-plugin/src/main/java/io/swagger/v3/plugins/gradle/SwaggerPlugin.java.
The problem with this plugin and its wiring is that it forces Gradle to
rebuild this module every time, and it's a very lengthy move..

The suggested way would be to include swagger's generated resources using
the processResources task, but approaching it that way causes a circular
dependency, because the plugin's incorrectly wired dependencies.. To get
this fixed, we need to raise an issue at the plugin and get it fixed first.
An alternative approach would be splitting the swagger resource generation
to its own module inside Fineract, and accept the fact that it corrupts
Gradle's cache mechanism (so that this new module would always get re-built
in every case).

The cache issue can also be seen in Develocity:
[image: image.png]

We concluded that I'll try to raise an issue at the Swagger Github about
the plugins incorrect ways of doing things, and they'll also emphasize it
from Gradle's side, hopefully getting it fixed mid-term.

*- Console log's readabilty*
Gradle team understands these concerns, and are working on improving these
parts already. They are considering adding information to each line which
helps to understand where the piece of information is coming from. That
could help demystifing logs like this (and cleaning up the related parts in
the build):
[image: image.png]
They are also considering gathering all error reports at the end of the
builds (again), so we won't have to hunt the errors from the flood of 897
warning messages. Speaking of which:

*- Suggestion: remove warning messages for headless builds*
We discussed the importance of warning messages, but their point is that
either those are important (so could be errors as well), or not important,
so we should silence them. At least for the headless builds running on CI
servers, because no one will care about those warnings anyway. That doesn't
perfectly fit my view, but for hundreds of warnings being there for many
years, they have a point...

*- IntelliJ Build View readability*
They also consider improving the UI on IntelliJ's Build view, this one:
[image: image.png]
(today I learnt that lines nested under bootRun are exceptions gathered and
picked up from the console logs.. Personally I think they are not useful at
all, especially with all of the stacktrace removed from the detail view on
the right side, and not related to the build either...)

*- Suggestion: switch to the Kotlin DSL and clean up builds*
Gradle team thinks the Kotlin DSL is the way forward, and we should
seriously consider switching our build from Groovy to Kotlin, especially if
we are aiming to clean up our build somewhat meanwhile. There are also
parts which look deprecated and possibly no longer required, like
`org.gradle.jvmargs`, use of `buildDir`, etc. We shall also remove the
`println` statements while we're at it..

*- Suggestion: get ge.apache.org <http://ge.apache.org> back up and running*
Our Fineract build tries to utilize the --scan feature already, which
provides deep insights of what's happening, why it's happening and how long
it takes. The only issue is that these scans get uploaded to ge.apache.org
by default, and it's down... I got connected to another Gradle team member,
who mentioned that ASF Infra hosts ge.apache.org, and there are recurring
availability issues with it, but they (Gradle team) are happy to help. They
also drafted a plan forward, but I'd prefer to discuss it personally with
the AST team member usually taking care of this host.. @Aleksandar maybe
you know who can touch these parts? Let's discuss this on Slack..

That's all I could push these issues forward today. Hope that helps in the
long run...

Kristof

On Tue, Sep 3, 2024 at 2:50 PM Aleksandar Vidakovic <
cheetah@monkeysintown.com> wrote:

> ... the performance issues stem from "dependsOn"... and similar... there
> are also neat features in Gradle where you can precompile pretty much all
> of the build script (and "configuration" cache it) which would boost the
> build process by a lot (disclosure: I am using this technique in projects
> with 3x of Fineract's project modules).
>
> Note: in version(s) 7.x (I believe) Gradle introduced configuration
> caching... which is really a big deal performance wise... BUT: to fully
> take advantage of that feature all plugins used must support that (and of
> course some don't). Still, configuration caching would be the "cherry on
> top"... our main problem is "dependsOn" (explicit and implicit).
>
> I can see the declarative style of Maven (having used it since before it
> was released 1.0)... but I would still bet my money on Gradle... and you
> are right, it would be a lot of work to create a Maven build from scratch.
>
> On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
> wrote:
>
>> I believe many of our issues stem from our current Gradle build being
>> overly complex. One of my main concerns with Gradle is that it allows
>> customization to evolve completely out of control. Following this
>> principle, I would be opposed to adding new elements to our build.
>>
>> Instead, I’d like to see the current build configuration simplified as
>> much as possible. Unfortunately, I’m not well-versed in Gradle. I have
>> always preferred Maven’s declarative approach over the programmatic style
>> of Gradle (or Ant, for that matter). If any Gradle experts on the team
>> could dedicate some time to reducing the custom behavior in our build, I
>> believe it would resolve many of the issues we’re currently facing.
>> An alternative would be turning back to a more maintainable, declarative
>> approach, but crafting a new build from scratch covering the complete
>> current functionality would be a hell of a work..
>>
>> Kristof
>>
>> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>> victor.romero@fintecheando.mx> wrote:
>>
>>> Hello,
>>>
>>> What about adding observability to the building process? In this way we
>>> can have information about the root causes of the issues during the
>>> building time.
>>>
>>> A OSS suggestion:
>>>
>>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>>
>>> Regards
>>>
>>> Victor
>>>
>>> El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
>>> escribió:
>>>
>>>> +1  agree
>>>> How do we tackle this ?
>>>>
>>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>>>>
>>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's
>>>>>> a horror to find out which ones failed, and our best bet is to text-search
>>>>>> for "FAILED" in the 17000+ lines of output.
>>>>>
>>>>> +1
>>>>>
>>>>> Items like this should be high priority in CI improvements epic. The
>>>>> overall failed test history, flaky test history, log tracing problem
>>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>>> a failed test case is important) were historically connected to getting PRs
>>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>>> problem.
>>>>>
>>>>> just by splitting up the code as is and moving some classes to
>>>>>> different project modules is not really taking into account the
>>>>>> interdependencies... if we don't address those then the project modules are
>>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>>
>>>>> I had overestimated what Fineract modularization design efforts have
>>>>> achieved. Would be interesting to know more about the design here as
>>>>> "splitting up the code as is" can be helped with more clarity.
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>>> cheetah@monkeysintown.com> wrote:
>>>>>
>>>>>> Kristof,
>>>>>>
>>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>>
>>>>>> Repeatable builds: we should consider this as a major bug and not
>>>>>> just "gloss over" it by executing the build multiple times... stuff like
>>>>>> this should halt all current development on other features and everyone (or
>>>>>> at least one person) should focus on fixing this as soon as possible. I
>>>>>> would consider this also a show stopper for a release.
>>>>>>
>>>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>>>>> here, but it seems that this is now actually worse than before... my bet is
>>>>>> that we introduced some performance hogs by splitting up the code base into
>>>>>> separate project modules; this didn't really fix the main problem of the
>>>>>> source code (too many interdependencies between packages aka "modules");
>>>>>> just by splitting up the code as is and moving some classes to different
>>>>>> project modules is not really taking into account the interdependencies...
>>>>>> if we don't address those then the project modules are not really helpful
>>>>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>>>>> them on Maven central nor chopping up the code into smaller pieces and
>>>>>> taking full advantage of caching.
>>>>>>
>>>>>> Caching: disabling caching is just not such a good idea... at all...
>>>>>> again: I would consider this a major bug and halt all machines until it's
>>>>>> fixed; also: consider this a show stopper for releases.
>>>>>>
>>>>>> Console readability: we've talked about this already on occasions
>>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>>> enforcements, because the reported style and quality checks literally
>>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>>> also avoiding to fix the quality issues.
>>>>>>
>>>>>> Suggestions:
>>>>>>
>>>>>>    - let's make an effort to reduce the interdependencies between
>>>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>>>    no added value of having classes in different jar files when it hurts build
>>>>>>    performance
>>>>>>    - let's spend time and fix those style and quality issues, not
>>>>>>    just pile feature on feature on feature... I am pretty sure that everyone
>>>>>>    can grab a handful of those warnings and fix them as part of their PRs
>>>>>>    (even if it's not one that is explicitly dedicated to quality assurance),
>>>>>>    in other words: everyone look a bit left and right to see what ELSE you can
>>>>>>    fix/add in addition to your tasks (chores vs cherry picking)... fixing
>>>>>>    everything in one PR is obviously not practical; we'll have a thousand
>>>>>>    files changed... this will interrupt everyone else quite significantly and
>>>>>>    is a nightmare to review
>>>>>>    - there might be less than a handful of Gradle build constraints
>>>>>>    that are again too paranoid and mess up the build; I think it's worth
>>>>>>    looking at the OpenAPI yaml build... that one had issues in the past with
>>>>>>    proper dirty checks, but there might be other/new constraints that are
>>>>>>    equally unhelpful
>>>>>>
>>>>>> My 2 cents
>>>>>>
>>>>>> Aleks
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> After working with Fineract for some months now, I collected several
>>>>>>> issues related to the Gradle build system we have in place. I categorized
>>>>>>> my pain points, and would like to initiate a discussion on the topic with
>>>>>>> the community. My usage scenarios include building locally with
>>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>>> build from the web UI.
>>>>>>>
>>>>>>> The problems I gathered fall into the following categories:
>>>>>>>
>>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails
>>>>>>> for the first time with:
>>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>>> FineractProperties to the 'fineract' property
>>>>>>> and simply trying it again always makes it wor0k.
>>>>>>>
>>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>>
>>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>>> development.
>>>>>>>
>>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>>>> local builds.
>>>>>>>
>>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>>>>>> gradle, to avoid issues with our build. This kind of denies even any
>>>>>>> potential adventages of using Gradle over more stable build tools. Note,
>>>>>>> that I do not use these settings (so my issues are definetely not caused by
>>>>>>> this), but I keep getting this recommendation for a reason.
>>>>>>>
>>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>>> running gradle with `--console=plain` to get some chance to read
>>>>>>> the logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+
>>>>>>> on a Github build, it's a horror to find out which ones failed, and our
>>>>>>> best bet is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>>
>>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>>> and shows a couple of warnings below, but the actual error is shown
>>>>>>> at line 1393 which appears to be part of
>>>>>>> > Task :fineract-document:modernizer
>>>>>>> which is clearly misleading, and your best bet to find the actual
>>>>>>> error on Github is to search for "error: ", but NOT using Ctrl+F,
>>>>>>> but the search box at the top right corner. That's ridiculous - a sensible
>>>>>>> build tool should communicate build errors in a straightforward way.
>>>>>>>
>>>>>>> I'd like to ask you to share your experiences and pain points with
>>>>>>> the current build that we have and use every day. Once we have a clearer
>>>>>>> understanding of the areas that need improvement, we can discuss potential
>>>>>>> strategies to revise our build system and ensure it meets basic
>>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kristof
>>>>>>>
>>>>>>>

Re: Discussion on Build System improvements

[Aleksandar Vidakovic <ch...@monkeysintown.com>]

... the performance issues stem from "dependsOn"... and similar... there
are also neat features in Gradle where you can precompile pretty much all
of the build script (and "configuration" cache it) which would boost the
build process by a lot (disclosure: I am using this technique in projects
with 3x of Fineract's project modules).

Note: in version(s) 7.x (I believe) Gradle introduced configuration
caching... which is really a big deal performance wise... BUT: to fully
take advantage of that feature all plugins used must support that (and of
course some don't). Still, configuration caching would be the "cherry on
top"... our main problem is "dependsOn" (explicit and implicit).

I can see the declarative style of Maven (having used it since before it
was released 1.0)... but I would still bet my money on Gradle... and you
are right, it would be a lot of work to create a Maven build from scratch.

On Tue, Sep 3, 2024 at 1:31 PM Kristof Jozsa <kr...@gmail.com>
wrote:

> I believe many of our issues stem from our current Gradle build being
> overly complex. One of my main concerns with Gradle is that it allows
> customization to evolve completely out of control. Following this
> principle, I would be opposed to adding new elements to our build.
>
> Instead, I’d like to see the current build configuration simplified as
> much as possible. Unfortunately, I’m not well-versed in Gradle. I have
> always preferred Maven’s declarative approach over the programmatic style
> of Gradle (or Ant, for that matter). If any Gradle experts on the team
> could dedicate some time to reducing the custom behavior in our build, I
> believe it would resolve many of the issues we’re currently facing.
> An alternative would be turning back to a more maintainable, declarative
> approach, but crafting a new build from scratch covering the complete
> current functionality would be a hell of a work..
>
> Kristof
>
> On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
> victor.romero@fintecheando.mx> wrote:
>
>> Hello,
>>
>> What about adding observability to the building process? In this way we
>> can have information about the root causes of the issues during the
>> building time.
>>
>> A OSS suggestion:
>>
>> https://github.com/craigatk/opentelemetry-gradle-plugin
>>
>> Regards
>>
>> Victor
>>
>> El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
>> escribió:
>>
>>> +1  agree
>>> How do we tackle this ?
>>>
>>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>>>
>>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a
>>>>> horror to find out which ones failed, and our best bet is to text-search
>>>>> for "FAILED" in the 17000+ lines of output.
>>>>
>>>> +1
>>>>
>>>> Items like this should be high priority in CI improvements epic. The
>>>> overall failed test history, flaky test history, log tracing problem
>>>> (actual result is often not sufficient to quickly debug and linking logs to
>>>> a failed test case is important) were historically connected to getting PRs
>>>> merged in time. CircleCI has helped us with the first two parts of this
>>>> problem.
>>>>
>>>> just by splitting up the code as is and moving some classes to
>>>>> different project modules is not really taking into account the
>>>>> interdependencies... if we don't address those then the project modules are
>>>>> not really helpful (actually it's adds more work to the Gradle builds)
>>>>
>>>> I had overestimated what Fineract modularization design efforts have
>>>> achieved. Would be interesting to know more about the design here as
>>>> "splitting up the code as is" can be helped with more clarity.
>>>>
>>>>
>>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>>> cheetah@monkeysintown.com> wrote:
>>>>
>>>>> Kristof,
>>>>>
>>>>> ... great overview, so first: thanks for that. Some notes...
>>>>>
>>>>> Repeatable builds: we should consider this as a major bug and not just
>>>>> "gloss over" it by executing the build multiple times... stuff like this
>>>>> should halt all current development on other features and everyone (or at
>>>>> least one person) should focus on fixing this as soon as possible. I would
>>>>> consider this also a show stopper for a release.
>>>>>
>>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>>>> here, but it seems that this is now actually worse than before... my bet is
>>>>> that we introduced some performance hogs by splitting up the code base into
>>>>> separate project modules; this didn't really fix the main problem of the
>>>>> source code (too many interdependencies between packages aka "modules");
>>>>> just by splitting up the code as is and moving some classes to different
>>>>> project modules is not really taking into account the interdependencies...
>>>>> if we don't address those then the project modules are not really helpful
>>>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>>>> them on Maven central nor chopping up the code into smaller pieces and
>>>>> taking full advantage of caching.
>>>>>
>>>>> Caching: disabling caching is just not such a good idea... at all...
>>>>> again: I would consider this a major bug and halt all machines until it's
>>>>> fixed; also: consider this a show stopper for releases.
>>>>>
>>>>> Console readability: we've talked about this already on occasions
>>>>> (there might be even a mailing list message)... on every upgrade of the
>>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>>> enforcements, because the reported style and quality checks literally
>>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>>> not a solution; I tried to be conservative (just disabled those that
>>>>> brought the build to a halt aka that were reported as errors), but as we
>>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>>> disabling the rules with certain annotations is neither a solution, that's
>>>>> also avoiding to fix the quality issues.
>>>>>
>>>>> Suggestions:
>>>>>
>>>>>    - let's make an effort to reduce the interdependencies between
>>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>>    no added value of having classes in different jar files when it hurts build
>>>>>    performance
>>>>>    - let's spend time and fix those style and quality issues, not
>>>>>    just pile feature on feature on feature... I am pretty sure that everyone
>>>>>    can grab a handful of those warnings and fix them as part of their PRs
>>>>>    (even if it's not one that is explicitly dedicated to quality assurance),
>>>>>    in other words: everyone look a bit left and right to see what ELSE you can
>>>>>    fix/add in addition to your tasks (chores vs cherry picking)... fixing
>>>>>    everything in one PR is obviously not practical; we'll have a thousand
>>>>>    files changed... this will interrupt everyone else quite significantly and
>>>>>    is a nightmare to review
>>>>>    - there might be less than a handful of Gradle build constraints
>>>>>    that are again too paranoid and mess up the build; I think it's worth
>>>>>    looking at the OpenAPI yaml build... that one had issues in the past with
>>>>>    proper dirty checks, but there might be other/new constraints that are
>>>>>    equally unhelpful
>>>>>
>>>>> My 2 cents
>>>>>
>>>>> Aleks
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> After working with Fineract for some months now, I collected several
>>>>>> issues related to the Gradle build system we have in place. I categorized
>>>>>> my pain points, and would like to initiate a discussion on the topic with
>>>>>> the community. My usage scenarios include building locally with
>>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>>> build from the web UI.
>>>>>>
>>>>>> The problems I gathered fall into the following categories:
>>>>>>
>>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails
>>>>>> for the first time with:
>>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>>> FineractProperties to the 'fineract' property
>>>>>> and simply trying it again always makes it wor0k.
>>>>>>
>>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>>
>>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>>> process. This leads to painfully slow code-test cycles at local
>>>>>> development.
>>>>>>
>>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>>> local builds.
>>>>>>
>>>>>> - *CACHING*: I see many team members and mailing list members
>>>>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>>>>> gradle, to avoid issues with our build. This kind of denies even any
>>>>>> potential adventages of using Gradle over more stable build tools. Note,
>>>>>> that I do not use these settings (so my issues are definetely not caused by
>>>>>> this), but I keep getting this recommendation for a reason.
>>>>>>
>>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>>> running gradle with `--console=plain` to get some chance to read the
>>>>>> logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+ on a
>>>>>> Github build, it's a horror to find out which ones failed, and our best bet
>>>>>> is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>>
>>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>>> > Task :fineract-loan:compileJava FAILED
>>>>>> and shows a couple of warnings below, but the actual error is shown
>>>>>> at line 1393 which appears to be part of
>>>>>> > Task :fineract-document:modernizer
>>>>>> which is clearly misleading, and your best bet to find the actual
>>>>>> error on Github is to search for "error: ", but NOT using Ctrl+F,
>>>>>> but the search box at the top right corner. That's ridiculous - a sensible
>>>>>> build tool should communicate build errors in a straightforward way.
>>>>>>
>>>>>> I'd like to ask you to share your experiences and pain points with
>>>>>> the current build that we have and use every day. Once we have a clearer
>>>>>> understanding of the areas that need improvement, we can discuss potential
>>>>>> strategies to revise our build system and ensure it meets basic
>>>>>> requirements.To be fair, please also make sure you share any recent
>>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>>
>>>>>> Thanks,
>>>>>> Kristof
>>>>>>
>>>>>>

Re: Discussion on Build System improvements

[Kristof Jozsa <kr...@gmail.com>]

I believe many of our issues stem from our current Gradle build being
overly complex. One of my main concerns with Gradle is that it allows
customization to evolve completely out of control. Following this
principle, I would be opposed to adding new elements to our build.

Instead, I’d like to see the current build configuration simplified as much
as possible. Unfortunately, I’m not well-versed in Gradle. I have always
preferred Maven’s declarative approach over the programmatic style of
Gradle (or Ant, for that matter). If any Gradle experts on the team could
dedicate some time to reducing the custom behavior in our build, I believe
it would resolve many of the issues we’re currently facing.
An alternative would be turning back to a more maintainable, declarative
approach, but crafting a new build from scratch covering the complete
current functionality would be a hell of a work..

Kristof

On Tue, Sep 3, 2024 at 6:10 AM VICTOR MANUEL ROMERO RODRIGUEZ <
victor.romero@fintecheando.mx> wrote:

> Hello,
>
> What about adding observability to the building process? In this way we
> can have information about the root causes of the issues during the
> building time.
>
> A OSS suggestion:
>
> https://github.com/craigatk/opentelemetry-gradle-plugin
>
> Regards
>
> Victor
>
> El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
> escribió:
>
>> +1  agree
>> How do we tackle this ?
>>
>> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>>
>>> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a
>>>> horror to find out which ones failed, and our best bet is to text-search
>>>> for "FAILED" in the 17000+ lines of output.
>>>
>>> +1
>>>
>>> Items like this should be high priority in CI improvements epic. The
>>> overall failed test history, flaky test history, log tracing problem
>>> (actual result is often not sufficient to quickly debug and linking logs to
>>> a failed test case is important) were historically connected to getting PRs
>>> merged in time. CircleCI has helped us with the first two parts of this
>>> problem.
>>>
>>> just by splitting up the code as is and moving some classes to different
>>>> project modules is not really taking into account the interdependencies...
>>>> if we don't address those then the project modules are not really helpful
>>>> (actually it's adds more work to the Gradle builds)
>>>
>>> I had overestimated what Fineract modularization design efforts have
>>> achieved. Would be interesting to know more about the design here as
>>> "splitting up the code as is" can be helped with more clarity.
>>>
>>>
>>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>>> cheetah@monkeysintown.com> wrote:
>>>
>>>> Kristof,
>>>>
>>>> ... great overview, so first: thanks for that. Some notes...
>>>>
>>>> Repeatable builds: we should consider this as a major bug and not just
>>>> "gloss over" it by executing the build multiple times... stuff like this
>>>> should halt all current development on other features and everyone (or at
>>>> least one person) should focus on fixing this as soon as possible. I would
>>>> consider this also a show stopper for a release.
>>>>
>>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>>> here, but it seems that this is now actually worse than before... my bet is
>>>> that we introduced some performance hogs by splitting up the code base into
>>>> separate project modules; this didn't really fix the main problem of the
>>>> source code (too many interdependencies between packages aka "modules");
>>>> just by splitting up the code as is and moving some classes to different
>>>> project modules is not really taking into account the interdependencies...
>>>> if we don't address those then the project modules are not really helpful
>>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>>> them on Maven central nor chopping up the code into smaller pieces and
>>>> taking full advantage of caching.
>>>>
>>>> Caching: disabling caching is just not such a good idea... at all...
>>>> again: I would consider this a major bug and halt all machines until it's
>>>> fixed; also: consider this a show stopper for releases.
>>>>
>>>> Console readability: we've talked about this already on occasions
>>>> (there might be even a mailing list message)... on every upgrade of the
>>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>>> the screws get tightened (read: rules get stricter); I remember that after
>>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>>> enforcements, because the reported style and quality checks literally
>>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>>> not a solution; I tried to be conservative (just disabled those that
>>>> brought the build to a halt aka that were reported as errors), but as we
>>>> can see there are enough warnings left to drive everyone crazy; btw:
>>>> disabling the rules with certain annotations is neither a solution, that's
>>>> also avoiding to fix the quality issues.
>>>>
>>>> Suggestions:
>>>>
>>>>    - let's make an effort to reduce the interdependencies between
>>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>>    no added value of having classes in different jar files when it hurts build
>>>>    performance
>>>>    - let's spend time and fix those style and quality issues, not just
>>>>    pile feature on feature on feature... I am pretty sure that everyone can
>>>>    grab a handful of those warnings and fix them as part of their PRs (even if
>>>>    it's not one that is explicitly dedicated to quality assurance), in other
>>>>    words: everyone look a bit left and right to see what ELSE you can fix/add
>>>>    in addition to your tasks (chores vs cherry picking)... fixing everything
>>>>    in one PR is obviously not practical; we'll have a thousand files
>>>>    changed... this will interrupt everyone else quite significantly and is a
>>>>    nightmare to review
>>>>    - there might be less than a handful of Gradle build constraints
>>>>    that are again too paranoid and mess up the build; I think it's worth
>>>>    looking at the OpenAPI yaml build... that one had issues in the past with
>>>>    proper dirty checks, but there might be other/new constraints that are
>>>>    equally unhelpful
>>>>
>>>> My 2 cents
>>>>
>>>> Aleks
>>>>
>>>>
>>>>
>>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> After working with Fineract for some months now, I collected several
>>>>> issues related to the Gradle build system we have in place. I categorized
>>>>> my pain points, and would like to initiate a discussion on the topic with
>>>>> the community. My usage scenarios include building locally with
>>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>>> build from the web UI.
>>>>>
>>>>> The problems I gathered fall into the following categories:
>>>>>
>>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails
>>>>> for the first time with:
>>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>>> FineractProperties to the 'fineract' property
>>>>> and simply trying it again always makes it wor0k.
>>>>>
>>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline
>>>>> character into the Loan class) and restarting Fineract makes it "build" for
>>>>> 17-18 seconds (checking all modules to be up-to-date) before starting it
>>>>> again. This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating
>>>>> up to 800% CPU and spinning up all the vents in my rig while doing so.
>>>>>
>>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>>> working with tests as well. Adding a blank line to an existing test class
>>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>>> process. This leads to painfully slow code-test cycles at local
>>>>> development.
>>>>>
>>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the
>>>>> Gradle daemon repeatedly grows up to 8Gb memory usage. That causes issues
>>>>> with dev boxes having 16Gb or even less RAM.. Even using my high-spec dev
>>>>> box, I needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during
>>>>> local builds.
>>>>>
>>>>> - *CACHING*: I see many team members and mailing list members
>>>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>>>> gradle, to avoid issues with our build. This kind of denies even any
>>>>> potential adventages of using Gradle over more stable build tools. Note,
>>>>> that I do not use these settings (so my issues are definetely not caused by
>>>>> this), but I keep getting this recommendation for a reason.
>>>>>
>>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>>> running gradle with `--console=plain` to get some chance to read the
>>>>> logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+ on a
>>>>> Github build, it's a horror to find out which ones failed, and our best bet
>>>>> is to text-search for "FAILED" in the 17000+ lines of output.
>>>>>
>>>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>>> for a perfect example. Here at line 1489 Gradle shows
>>>>> > Task :fineract-loan:compileJava FAILED
>>>>> and shows a couple of warnings below, but the actual error is shown at
>>>>> line 1393 which appears to be part of
>>>>> > Task :fineract-document:modernizer
>>>>> which is clearly misleading, and your best bet to find the actual
>>>>> error on Github is to search for "error: ", but NOT using Ctrl+F, but
>>>>> the search box at the top right corner. That's ridiculous - a sensible
>>>>> build tool should communicate build errors in a straightforward way.
>>>>>
>>>>> I'd like to ask you to share your experiences and pain points with the
>>>>> current build that we have and use every day. Once we have a clearer
>>>>> understanding of the areas that need improvement, we can discuss potential
>>>>> strategies to revise our build system and ensure it meets basic
>>>>> requirements.To be fair, please also make sure you share any recent
>>>>> positive experiences related to Fineract builds, if you have any.
>>>>>
>>>>> Thanks,
>>>>> Kristof
>>>>>
>>>>>

Re: Discussion on Build System improvements

[VICTOR MANUEL ROMERO RODRIGUEZ <vi...@fintecheando.mx>]

Hello,

What about adding observability to the building process? In this way we can
have information about the root causes of the issues during the building
time.

A OSS suggestion:

https://github.com/craigatk/opentelemetry-gradle-plugin

Regards

Victor

El dom, 1 sept 2024 a las 10:06, James Dailey (<ja...@gmail.com>)
escribió:

> +1  agree
> How do we tackle this ?
>
> On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:
>
>> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a
>>> horror to find out which ones failed, and our best bet is to text-search
>>> for "FAILED" in the 17000+ lines of output.
>>
>> +1
>>
>> Items like this should be high priority in CI improvements epic. The
>> overall failed test history, flaky test history, log tracing problem
>> (actual result is often not sufficient to quickly debug and linking logs to
>> a failed test case is important) were historically connected to getting PRs
>> merged in time. CircleCI has helped us with the first two parts of this
>> problem.
>>
>> just by splitting up the code as is and moving some classes to different
>>> project modules is not really taking into account the interdependencies...
>>> if we don't address those then the project modules are not really helpful
>>> (actually it's adds more work to the Gradle builds)
>>
>> I had overestimated what Fineract modularization design efforts have
>> achieved. Would be interesting to know more about the design here as
>> "splitting up the code as is" can be helped with more clarity.
>>
>>
>> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
>> cheetah@monkeysintown.com> wrote:
>>
>>> Kristof,
>>>
>>> ... great overview, so first: thanks for that. Some notes...
>>>
>>> Repeatable builds: we should consider this as a major bug and not just
>>> "gloss over" it by executing the build multiple times... stuff like this
>>> should halt all current development on other features and everyone (or at
>>> least one person) should focus on fixing this as soon as possible. I would
>>> consider this also a show stopper for a release.
>>>
>>> Dirty checks: we were actually at one point (2+ years?) not too bad
>>> here, but it seems that this is now actually worse than before... my bet is
>>> that we introduced some performance hogs by splitting up the code base into
>>> separate project modules; this didn't really fix the main problem of the
>>> source code (too many interdependencies between packages aka "modules");
>>> just by splitting up the code as is and moving some classes to different
>>> project modules is not really taking into account the interdependencies...
>>> if we don't address those then the project modules are not really helpful
>>> (actually it's adds more work to the Gradle builds), neither for publishing
>>> them on Maven central nor chopping up the code into smaller pieces and
>>> taking full advantage of caching.
>>>
>>> Caching: disabling caching is just not such a good idea... at all...
>>> again: I would consider this a major bug and halt all machines until it's
>>> fixed; also: consider this a show stopper for releases.
>>>
>>> Console readability: we've talked about this already on occasions (there
>>> might be even a mailing list message)... on every upgrade of the
>>> dependencies (including the quality assurance tools like Errorprone etc.)
>>> the screws get tightened (read: rules get stricter); I remember that after
>>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>>> enforcements, because the reported style and quality checks literally
>>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>>> not a solution; I tried to be conservative (just disabled those that
>>> brought the build to a halt aka that were reported as errors), but as we
>>> can see there are enough warnings left to drive everyone crazy; btw:
>>> disabling the rules with certain annotations is neither a solution, that's
>>> also avoiding to fix the quality issues.
>>>
>>> Suggestions:
>>>
>>>    - let's make an effort to reduce the interdependencies between
>>>    packages aka "modules" before we rip the code apart and move it to separate
>>>    project modules; if we can't do that (backwards compatibility, not allowed
>>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>>    no added value of having classes in different jar files when it hurts build
>>>    performance
>>>    - let's spend time and fix those style and quality issues, not just
>>>    pile feature on feature on feature... I am pretty sure that everyone can
>>>    grab a handful of those warnings and fix them as part of their PRs (even if
>>>    it's not one that is explicitly dedicated to quality assurance), in other
>>>    words: everyone look a bit left and right to see what ELSE you can fix/add
>>>    in addition to your tasks (chores vs cherry picking)... fixing everything
>>>    in one PR is obviously not practical; we'll have a thousand files
>>>    changed... this will interrupt everyone else quite significantly and is a
>>>    nightmare to review
>>>    - there might be less than a handful of Gradle build constraints
>>>    that are again too paranoid and mess up the build; I think it's worth
>>>    looking at the OpenAPI yaml build... that one had issues in the past with
>>>    proper dirty checks, but there might be other/new constraints that are
>>>    equally unhelpful
>>>
>>> My 2 cents
>>>
>>> Aleks
>>>
>>>
>>>
>>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> After working with Fineract for some months now, I collected several
>>>> issues related to the Gradle build system we have in place. I categorized
>>>> my pain points, and would like to initiate a discussion on the topic with
>>>> the community. My usage scenarios include building locally with
>>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>>> build from the web UI.
>>>>
>>>> The problems I gathered fall into the following categories:
>>>>
>>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails
>>>> for the first time with:
>>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>>> FineractProperties to the 'fineract' property
>>>> and simply trying it again always makes it wor0k.
>>>>
>>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline character
>>>> into the Loan class) and restarting Fineract makes it "build" for 17-18
>>>> seconds (checking all modules to be up-to-date) before starting it again.
>>>> This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating up to
>>>> 800% CPU and spinning up all the vents in my rig while doing so.
>>>>
>>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>>> working with tests as well. Adding a blank line to an existing test class
>>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>>> process. This leads to painfully slow code-test cycles at local
>>>> development.
>>>>
>>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the Gradle
>>>> daemon repeatedly grows up to 8Gb memory usage. That causes issues with dev
>>>> boxes having 16Gb or even less RAM.. Even using my high-spec dev box, I
>>>> needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>>> -Dorg.gradle.priority=low" to keep my desktop responsible during local
>>>> builds.
>>>>
>>>> - *CACHING*: I see many team members and mailing list members
>>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>>> gradle, to avoid issues with our build. This kind of denies even any
>>>> potential adventages of using Gradle over more stable build tools. Note,
>>>> that I do not use these settings (so my issues are definetely not caused by
>>>> this), but I keep getting this recommendation for a reason.
>>>>
>>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting
>>>> running gradle with `--console=plain` to get some chance to read the
>>>> logs in a sensible way. Still, when eg. 2 tests fail out of the 1100+ on a
>>>> Github build, it's a horror to find out which ones failed, and our best bet
>>>> is to text-search for "FAILED" in the 17000+ lines of output.
>>>>
>>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>>> for a perfect example. Here at line 1489 Gradle shows
>>>> > Task :fineract-loan:compileJava FAILED
>>>> and shows a couple of warnings below, but the actual error is shown at
>>>> line 1393 which appears to be part of
>>>> > Task :fineract-document:modernizer
>>>> which is clearly misleading, and your best bet to find the actual error
>>>> on Github is to search for "error: ", but NOT using Ctrl+F, but the
>>>> search box at the top right corner. That's ridiculous - a sensible build
>>>> tool should communicate build errors in a straightforward way.
>>>>
>>>> I'd like to ask you to share your experiences and pain points with the
>>>> current build that we have and use every day. Once we have a clearer
>>>> understanding of the areas that need improvement, we can discuss potential
>>>> strategies to revise our build system and ensure it meets basic
>>>> requirements.To be fair, please also make sure you share any recent
>>>> positive experiences related to Fineract builds, if you have any.
>>>>
>>>> Thanks,
>>>> Kristof
>>>>
>>>>

Re: Discussion on Build System improvements

[James Dailey <ja...@gmail.com>]

+1  agree
How do we tackle this ?

On Sun, Sep 1, 2024 at 7:49 AM Avik Ganguly <av...@fynarfin.io> wrote:

> Still, when eg. 2 tests fail out of the 1100+ on a Github build, it's a
>> horror to find out which ones failed, and our best bet is to text-search
>> for "FAILED" in the 17000+ lines of output.
>
> +1
>
> Items like this should be high priority in CI improvements epic. The
> overall failed test history, flaky test history, log tracing problem
> (actual result is often not sufficient to quickly debug and linking logs to
> a failed test case is important) were historically connected to getting PRs
> merged in time. CircleCI has helped us with the first two parts of this
> problem.
>
> just by splitting up the code as is and moving some classes to different
>> project modules is not really taking into account the interdependencies...
>> if we don't address those then the project modules are not really helpful
>> (actually it's adds more work to the Gradle builds)
>
> I had overestimated what Fineract modularization design efforts have
> achieved. Would be interesting to know more about the design here as
> "splitting up the code as is" can be helped with more clarity.
>
>
> On Thu, Aug 8, 2024 at 8:02 PM Aleksandar Vidakovic <
> cheetah@monkeysintown.com> wrote:
>
>> Kristof,
>>
>> ... great overview, so first: thanks for that. Some notes...
>>
>> Repeatable builds: we should consider this as a major bug and not just
>> "gloss over" it by executing the build multiple times... stuff like this
>> should halt all current development on other features and everyone (or at
>> least one person) should focus on fixing this as soon as possible. I would
>> consider this also a show stopper for a release.
>>
>> Dirty checks: we were actually at one point (2+ years?) not too bad here,
>> but it seems that this is now actually worse than before... my bet is that
>> we introduced some performance hogs by splitting up the code base into
>> separate project modules; this didn't really fix the main problem of the
>> source code (too many interdependencies between packages aka "modules");
>> just by splitting up the code as is and moving some classes to different
>> project modules is not really taking into account the interdependencies...
>> if we don't address those then the project modules are not really helpful
>> (actually it's adds more work to the Gradle builds), neither for publishing
>> them on Maven central nor chopping up the code into smaller pieces and
>> taking full advantage of caching.
>>
>> Caching: disabling caching is just not such a good idea... at all...
>> again: I would consider this a major bug and halt all machines until it's
>> fixed; also: consider this a show stopper for releases.
>>
>> Console readability: we've talked about this already on occasions (there
>> might be even a mailing list message)... on every upgrade of the
>> dependencies (including the quality assurance tools like Errorprone etc.)
>> the screws get tightened (read: rules get stricter); I remember that after
>> one upgrade (maybe last year, maybe longer ago) I had to disable some rule
>> enforcements, because the reported style and quality checks literally
>> exploded; holding back on Errorprone (and similar) upgrades is obviously
>> not a solution; I tried to be conservative (just disabled those that
>> brought the build to a halt aka that were reported as errors), but as we
>> can see there are enough warnings left to drive everyone crazy; btw:
>> disabling the rules with certain annotations is neither a solution, that's
>> also avoiding to fix the quality issues.
>>
>> Suggestions:
>>
>>    - let's make an effort to reduce the interdependencies between
>>    packages aka "modules" before we rip the code apart and move it to separate
>>    project modules; if we can't do that (backwards compatibility, not allowed
>>    to "break eggs" etc.) then let's leave the code in one place as is... I see
>>    no added value of having classes in different jar files when it hurts build
>>    performance
>>    - let's spend time and fix those style and quality issues, not just
>>    pile feature on feature on feature... I am pretty sure that everyone can
>>    grab a handful of those warnings and fix them as part of their PRs (even if
>>    it's not one that is explicitly dedicated to quality assurance), in other
>>    words: everyone look a bit left and right to see what ELSE you can fix/add
>>    in addition to your tasks (chores vs cherry picking)... fixing everything
>>    in one PR is obviously not practical; we'll have a thousand files
>>    changed... this will interrupt everyone else quite significantly and is a
>>    nightmare to review
>>    - there might be less than a handful of Gradle build constraints that
>>    are again too paranoid and mess up the build; I think it's worth looking at
>>    the OpenAPI yaml build... that one had issues in the past with proper dirty
>>    checks, but there might be other/new constraints that are equally unhelpful
>>
>> My 2 cents
>>
>> Aleks
>>
>>
>>
>> On Thu, Aug 8, 2024 at 3:47 PM Kristof Jozsa <kr...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> After working with Fineract for some months now, I collected several
>>> issues related to the Gradle build system we have in place. I categorized
>>> my pain points, and would like to initiate a discussion on the topic with
>>> the community. My usage scenarios include building locally with
>>> IntelliJ+Gradle wrapper, building locally from CLI and using the Github
>>> build from the web UI.
>>>
>>> The problems I gathered fall into the following categories:
>>>
>>> - *REPEATABLE BUILDS:* starting Fineract from IntelliJ always fails for
>>> the first time with:
>>> Caused by: java.lang.IllegalArgumentException: Couldn't bind
>>> FineractProperties to the 'fineract' property
>>> and simply trying it again always makes it wor0k.
>>>
>>> - *DIRTY CHECKS*: modifying a single file (eg. put a newline character
>>> into the Loan class) and restarting Fineract makes it "build" for 17-18
>>> seconds (checking all modules to be up-to-date) before starting it again.
>>> This happens on my 16 core Intel i7 HQ CPU with 64Gb of RAM, eating up to
>>> 800% CPU and spinning up all the vents in my rig while doing so.
>>>
>>> - *INTEGRATION TESTS*: the same dirty check problem happens when
>>> working with tests as well. Adding a blank line to an existing test class
>>> and re-running the test again happens after a ~20 seconds "rebuild"
>>> process. This leads to painfully slow code-test cycles at local
>>> development.
>>>
>>> - *RESOURCE REQUIREMENTS*: local team reports that in cases the Gradle
>>> daemon repeatedly grows up to 8Gb memory usage. That causes issues with dev
>>> boxes having 16Gb or even less RAM.. Even using my high-spec dev box, I
>>> needed to add GRADLE_OPTS="-Dorg.gradle.workers.max=12
>>> -Dorg.gradle.priority=low" to keep my desktop responsible during local
>>> builds.
>>>
>>> - *CACHING*: I see many team members and mailing list members
>>> recommending `--no-build-cache --no-daemon` settings when invoking
>>> gradle, to avoid issues with our build. This kind of denies even any
>>> potential adventages of using Gradle over more stable build tools. Note,
>>> that I do not use these settings (so my issues are definetely not caused by
>>> this), but I keep getting this recommendation for a reason.
>>>
>>> - *CONSOLE READABILITY*: I also see many fellow devs suggesting running
>>> gradle with `--console=plain` to get some chance to read the logs in a
>>> sensible way. Still, when eg. 2 tests fail out of the 1100+ on a Github
>>> build, it's a horror to find out which ones failed, and our best bet is to
>>> text-search for "FAILED" in the 17000+ lines of output.
>>>
>>> - *REPORTING COMPILE ERRORS*: related to the previous point, check
>>> https://github.com/kjozsa/fineract/actions/runs/10302318345/job/28515733682
>>> for a perfect example. Here at line 1489 Gradle shows
>>> > Task :fineract-loan:compileJava FAILED
>>> and shows a couple of warnings below, but the actual error is shown at
>>> line 1393 which appears to be part of
>>> > Task :fineract-document:modernizer
>>> which is clearly misleading, and your best bet to find the actual error
>>> on Github is to search for "error: ", but NOT using Ctrl+F, but the
>>> search box at the top right corner. That's ridiculous - a sensible build
>>> tool should communicate build errors in a straightforward way.
>>>
>>> I'd like to ask you to share your experiences and pain points with the
>>> current build that we have and use every day. Once we have a clearer
>>> understanding of the areas that need improvement, we can discuss potential
>>> strategies to revise our build system and ensure it meets basic
>>> requirements.To be fair, please also make sure you share any recent
>>> positive experiences related to Fineract builds, if you have any.
>>>
>>> Thanks,
>>> Kristof
>>>
>>>