Building a Solr cluster with Maven

[Greg Pendlebury <gr...@gmail.com>]

Are there any developers with a current working maven build for a
downstream Solr installation? ie. Not a build for Solr itself, but a build
that brings in the core Solr server plus local plugins, third party plugins
etc?

I am in the process of updating one of our old builds (it builds both the
application and various shard instances) and have hit a stumbling block in
sourcing the dashboard static assets (everything under /webapp/web in
Solr's source).

Prior to the move away from being a webapp I could get them by exploding
the war from Maven Central.

In our very first foray into 5.x we had a local custom build to patch
SOLR-2649. We avoided solving this problem then by pushing the webapp into
our local Nexus as part of that build... but that wasn't a very good long
term choice.

So now I'm trying to work out the best long term approach to take here.
Ideas so far:

   1. Manually download the required zip and add it into our Nexus
   repository as a 3rd party artifact. Maven can source and extract anything
   it needs from here. This is where I'm currently leaning for simplicity, but
   the manual step required is annoying. It does have the advantage of causing
   a build failure straight away when a version upgrade occurs, prompting the
   developer to look into why.
   2. Move a copy of the static assets for the dashboard into our project
   and deploy them ourselves. This has the advantage of aligning our approach
   with the resources we already maintain in the project (like
   core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
   worried that it is really fragile and developers will miss it during a
   version upgrade, resulting in the dashboard creeping out-of-date and
   (worse) introducing subtle bugs because of a version mismatch between the
   UI and the underlying server code.
   3. I'd like to think a long term approach would be for the core Solr
   build to ship a JAR (or any other assembly) to Maven Central like
   'solr-dashboard'... but I'm not sure how that aligns with the move away
   from Solr being considered a webapp. It seems a shame that all of the Java
   code ends up in Maven central, but the web layer dead-ends in the ant build.

I might be missing something really obvious and there is already a way to
do this. Is there some other distribution of the dashboard statics? Other
than the downloadable zip that is.

Ta,
Greg

Re: Building a Solr cluster with Maven

[Keith L <ke...@gmail.com>]

My team has modified the ant scripts to publish all the jars/poms and the
zip to our local artifactory when we run our build. We have another project
which pulls down all of these dependencies including the zip to build our
actual solr deploy and a maven assembly which unpacks the zip file and
extracts all of the webapp for our real distribution.

I haven't upstreamed the changes for the ant tasks thinking there wouldn't
be too much interest in that, but I could put together a patch if there is.

The changes do the following:

- Packages the zip along with the parent pom if a flag is set
- Allows changing group which the poms are published to. For example
instead of org.apache you can push it as com.xxx to avoid shadowing
conflicts in your local repository.

On Tue, Oct 18, 2016 at 8:42 AM David Smiley <da...@gmail.com>
wrote:

> Thanks for bringing this up, Greg.  I too have felt the pain of this in
> the move away from a WAR file in a project or two.  In one of the projects
> that comes to mind, we built scripts that re-constituted a Solr
> distribution from artifacts in Maven. For anything that wasn't in Maven
> (e.g. the admin UI pages, Jetty configs), we checked it into source
> control.  In hind sight... the simplicity of what you list as (1) -- check
> the distro zip into a Maven repo local to the organization sounds better...
> but I may be forgetting requirements that led us not to do this.  I look
> forward to that zip shrinking once the docs are gone.  Another option,
> depending on one's needs, is to pursue Docker, which I've lately become a
> huge fan of.  I think Docker is particularly great for integration tests.
> Does the scenario you wish to use the assets for relate to testing or some
> other use-case?
>
> ~ David
>
>
>
> On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury <gr...@gmail.com>
> wrote:
>
> Are there any developers with a current working maven build for a
> downstream Solr installation? ie. Not a build for Solr itself, but a build
> that brings in the core Solr server plus local plugins, third party plugins
> etc?
>
> I am in the process of updating one of our old builds (it builds both the
> application and various shard instances) and have hit a stumbling block in
> sourcing the dashboard static assets (everything under /webapp/web in
> Solr's source).
>
> Prior to the move away from being a webapp I could get them by exploding
> the war from Maven Central.
>
> In our very first foray into 5.x we had a local custom build to patch
> SOLR-2649. We avoided solving this problem then by pushing the webapp into
> our local Nexus as part of that build... but that wasn't a very good long
> term choice.
>
> So now I'm trying to work out the best long term approach to take here.
> Ideas so far:
>
>    1. Manually download the required zip and add it into our Nexus
>    repository as a 3rd party artifact. Maven can source and extract anything
>    it needs from here. This is where I'm currently leaning for simplicity, but
>    the manual step required is annoying. It does have the advantage of causing
>    a build failure straight away when a version upgrade occurs, prompting the
>    developer to look into why.
>    2. Move a copy of the static assets for the dashboard into our project
>    and deploy them ourselves. This has the advantage of aligning our approach
>    with the resources we already maintain in the project (like
>    core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
>    worried that it is really fragile and developers will miss it during a
>    version upgrade, resulting in the dashboard creeping out-of-date and
>    (worse) introducing subtle bugs because of a version mismatch between the
>    UI and the underlying server code.
>    3. I'd like to think a long term approach would be for the core Solr
>    build to ship a JAR (or any other assembly) to Maven Central like
>    'solr-dashboard'... but I'm not sure how that aligns with the move away
>    from Solr being considered a webapp. It seems a shame that all of the Java
>    code ends up in Maven central, but the web layer dead-ends in the ant build.
>
> I might be missing something really obvious and there is already a way to
> do this. Is there some other distribution of the dashboard statics? Other
> than the downloadable zip that is.
>
> Ta,
> Greg
>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>

Re: Building a Solr cluster with Maven

[David Smiley <da...@gmail.com>]

Thanks for bringing this up, Greg.  I too have felt the pain of this in the
move away from a WAR file in a project or two.  In one of the projects that
comes to mind, we built scripts that re-constituted a Solr distribution
from artifacts in Maven. For anything that wasn't in Maven (e.g. the admin
UI pages, Jetty configs), we checked it into source control.  In hind
sight... the simplicity of what you list as (1) -- check the distro zip
into a Maven repo local to the organization sounds better... but I may be
forgetting requirements that led us not to do this.  I look forward to that
zip shrinking once the docs are gone.  Another option, depending on one's
needs, is to pursue Docker, which I've lately become a huge fan of.  I
think Docker is particularly great for integration tests.  Does the
scenario you wish to use the assets for relate to testing or some other
use-case?

~ David



On Mon, Oct 17, 2016 at 7:58 PM Greg Pendlebury <gr...@gmail.com>
wrote:

> Are there any developers with a current working maven build for a
> downstream Solr installation? ie. Not a build for Solr itself, but a build
> that brings in the core Solr server plus local plugins, third party plugins
> etc?
>
> I am in the process of updating one of our old builds (it builds both the
> application and various shard instances) and have hit a stumbling block in
> sourcing the dashboard static assets (everything under /webapp/web in
> Solr's source).
>
> Prior to the move away from being a webapp I could get them by exploding
> the war from Maven Central.
>
> In our very first foray into 5.x we had a local custom build to patch
> SOLR-2649. We avoided solving this problem then by pushing the webapp into
> our local Nexus as part of that build... but that wasn't a very good long
> term choice.
>
> So now I'm trying to work out the best long term approach to take here.
> Ideas so far:
>
>    1. Manually download the required zip and add it into our Nexus
>    repository as a 3rd party artifact. Maven can source and extract anything
>    it needs from here. This is where I'm currently leaning for simplicity, but
>    the manual step required is annoying. It does have the advantage of causing
>    a build failure straight away when a version upgrade occurs, prompting the
>    developer to look into why.
>    2. Move a copy of the static assets for the dashboard into our project
>    and deploy them ourselves. This has the advantage of aligning our approach
>    with the resources we already maintain in the project (like
>    core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
>    worried that it is really fragile and developers will miss it during a
>    version upgrade, resulting in the dashboard creeping out-of-date and
>    (worse) introducing subtle bugs because of a version mismatch between the
>    UI and the underlying server code.
>    3. I'd like to think a long term approach would be for the core Solr
>    build to ship a JAR (or any other assembly) to Maven Central like
>    'solr-dashboard'... but I'm not sure how that aligns with the move away
>    from Solr being considered a webapp. It seems a shame that all of the Java
>    code ends up in Maven central, but the web layer dead-ends in the ant build.
>
> I might be missing something really obvious and there is already a way to
> do this. Is there some other distribution of the dashboard statics? Other
> than the downloadable zip that is.
>
> Ta,
> Greg
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com