Solr, gradle build and working with files compiled into the source tree

[Dawid Weiss <da...@gmail.com>]

David asked this question regarding the gradle build and I think it
may be a longer debate so I start a new thread:

> What gradlew command is the equivalent of "ant server" for Solr?  I wish to use "bin/solr".

Here comes my very subjective opinion: let's get rid of the
possibility to run various commands right on the source tree... Let me
explain why.

I had a *very* hard time even understanding what is where in the Solr
source tree and whether it's versioned, compiled or generated. I ended
up using a clean repository, compiled repository and a distribution
ZIP and diffing against each other. This can't be a sane thing.

Having compiled files in the source tree makes everything lousy - from
complex rules in .gitignore files to extreme hacks inside build files
(ant, gradle or otherwise) to make this copying and syncing work.

What's the alternative? Well, one of such alternatives is a
"packaging" project that puts together the binary project
distribution. With gradle this is lighting fast, incremental and
always correct (files are synced from source folders and dependencies
-- they are not copied). For example, putting a Solr distribution from
scratch takes 2 minutes on my machine (first run) but subsequent run
(with no changes to anything) is nearly instantaneous (5 secs).

If you remove a file or a dependency in the source and rebuild the package with:

gradlew -p solr/packaging assemble

then the assembled output under:

solr/packaging/build/solr-9.0.0-SNAPSHOT

will always have up-to-date state of a true distribution. You can run
this gradle task from an IDE as well as part of a launch configuration
and then start solr with a debugger so that it attaches right to your
sources...

The benefits of this are enormous. For instance you can shuffle source
modules around and give them a more logical structure (the assembly
takes care of where they're placed in the output). You can clearly see
which components go where and how they're put together. Even now it's
fairly clear how the distribution is put together, even if it's still
not ideal:

https://github.com/apache/lucene-solr/blob/gradle-master/solr/packaging/build.gradle#L50-L92

I realize it's hard to get people out of their comfort zone (I'm a
prime example of this) but give this packaging thing a try, I don't
think you'll be disappointed.

This said, I did try to give people some backward compatibility with
"resolve" target which copies over JARs to their previous source-tree
locations...

gradlew resolve

I didn't verify it thoroughly though and the complexity of even trying
to mimic the ant build here is hair-raising:

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ant-compat/resolve.gradle

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

[Dawid Weiss <da...@gmail.com>]

> It's a trade-off; let's acknowledge that the current system has its benefits.
> For example not needing to change to some other directory to run Solr.

I don't see those, to be honest. Switching directory to run the thing
doesn't seem like a big benefit and running in place... look at the
.gitignore files
and you'll see what I mean when I say it's an ugly thing to do.

> And at present the gradle assemble script will use a target path with a version in the name which may make docs/instructions/scripts and habits more susceptible to getting out of date / being wrong.  That could be fixed easily though.

This is intentional, actually. Docs, instructions and scripts should
only point at and describe the internal content of this folder, not to
the folder name. If somebody builds from sources they will know what
they're doing (?).

> I like that you've hid it away in one place and it doesn't look hideous to me.

Trust me, it is terrible in many ways...

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

[David Smiley <da...@gmail.com>]

It's a trade-off; let's acknowledge that the current system has its
benefits.  For example not needing to change to some other directory to run
Solr.  Some in-place changes (e.g. to bin/solr) are possible without
additional sync steps.  And at present the gradle assemble script will use
a target path with a version in the name which may make
docs/instructions/scripts and habits more susceptible to getting out of
date / being wrong.  That could be fixed easily though.

Thanks for doing "gradlew resolve".  "resolve" is not in my muscle memory;
"server" is, but seems similar.  Maybe some small changes could make
"graldew resolve" more straightforward.  Despite you and Erick not liking
it, I like that you've hid it away in one place and it doesn't look hideous
to me.  Any way there is no hurry on this matter; it doesn't block merging
or adopting the gradle build.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Dec 3, 2019 at 12:39 PM Dawid Weiss <da...@gmail.com> wrote:

> > I’d favor taking out the “resolve” task as well. Thanks for considering
> it,
> > but let’s keep things architecturally sound and add back complexity only
> > when necessary.
>
> That entire folder is meant to be discarded at some point. Let's allow
> it to sit there for a bit though,
> doesn't do any harm and we'll see how the transition goes.
>
> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: Solr, gradle build and working with files compiled into the source tree

[Dawid Weiss <da...@gmail.com>]

> I’d favor taking out the “resolve” task as well. Thanks for considering it,
> but let’s keep things architecturally sound and add back complexity only
> when necessary.

That entire folder is meant to be discarded at some point. Let's allow
it to sit there for a bit though,
doesn't do any harm and we'll see how the transition goes.

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

[Erick Erickson <er...@gmail.com>]

AFAIC, there’s no good reason whatsoever to preserve being able to run
from inside the source tree if it makes maintenance of the build system
harder. The current system grew organically over years and has its warts,
Preserving those warts just “so I can do what I’ve always done” is
unnecessary.

I’d favor taking out the “resolve” task as well. Thanks for considering it,
but let’s keep things architecturally sound and add back complexity only
when necessary.

If I can rebuild Solr quickly after a small change, then the inconvenience
of having to change directories after a clean is minor. Very minor.

Unless I’m missing something about where the project gets built, let’s
keep it clean and nuke the “resolve” target and use the package directory.

Erick

> On Dec 3, 2019, at 4:05 AM, Dawid Weiss <da...@gmail.com> wrote:
> 
> David asked this question regarding the gradle build and I think it
> may be a longer debate so I start a new thread:
> 
>> What gradlew command is the equivalent of "ant server" for Solr?  I wish to use "bin/solr".
> 
> Here comes my very subjective opinion: let's get rid of the
> possibility to run various commands right on the source tree... Let me
> explain why.
> 
> I had a *very* hard time even understanding what is where in the Solr
> source tree and whether it's versioned, compiled or generated. I ended
> up using a clean repository, compiled repository and a distribution
> ZIP and diffing against each other. This can't be a sane thing.
> 
> Having compiled files in the source tree makes everything lousy - from
> complex rules in .gitignore files to extreme hacks inside build files
> (ant, gradle or otherwise) to make this copying and syncing work.
> 
> What's the alternative? Well, one of such alternatives is a
> "packaging" project that puts together the binary project
> distribution. With gradle this is lighting fast, incremental and
> always correct (files are synced from source folders and dependencies
> -- they are not copied). For example, putting a Solr distribution from
> scratch takes 2 minutes on my machine (first run) but subsequent run
> (with no changes to anything) is nearly instantaneous (5 secs).
> 
> If you remove a file or a dependency in the source and rebuild the package with:
> 
> gradlew -p solr/packaging assemble
> 
> then the assembled output under:
> 
> solr/packaging/build/solr-9.0.0-SNAPSHOT
> 
> will always have up-to-date state of a true distribution. You can run
> this gradle task from an IDE as well as part of a launch configuration
> and then start solr with a debugger so that it attaches right to your
> sources...
> 
> The benefits of this are enormous. For instance you can shuffle source
> modules around and give them a more logical structure (the assembly
> takes care of where they're placed in the output). You can clearly see
> which components go where and how they're put together. Even now it's
> fairly clear how the distribution is put together, even if it's still
> not ideal:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/solr/packaging/build.gradle#L50-L92
> 
> I realize it's hard to get people out of their comfort zone (I'm a
> prime example of this) but give this packaging thing a try, I don't
> think you'll be disappointed.
> 
> This said, I did try to give people some backward compatibility with
> "resolve" target which copies over JARs to their previous source-tree
> locations...
> 
> gradlew resolve
> 
> I didn't verify it thoroughly though and the complexity of even trying
> to mimic the ant build here is hair-raising:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ant-compat/resolve.gradle
> 
> Dawid
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org