You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by David Smiley <ds...@apache.org> on 2021/06/11 11:29:43 UTC
Re: Propose changing the "dist" layout
Bumping this conversation up, based on recent communication. I have yet to
take action but really any of us can.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org> wrote:
> I'll proceed on this with lazy consensus. I suspect most of us don't
> care, unsurprisingly since I doubt anyone has any fondness for the "dist"
> folder.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <er...@gmail.com>
> wrote:
>
>> Well, Solr has grown “organically” so some things just _are_, like
>> sunrises and plagues ;)
>>
>> On a serious note, AFAIC rearrange as you see fit. I wonder how much of
>> this is left over from the war days? Anything that’s lasted through all the
>> transformations Solr has is bound to need cleaning up betimes.
>>
>> How would it relate to splitting Solr off into its own TLP? On the
>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>> in case rearranging the layout would make that task easier or harder...
>>
>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org> wrote:
>> >
>> > I've been doing a bit of dependency work in one of our contribs, and
>> observing more closely than usual exactly what we produce in the
>> distribution layout (result of gradlew assemble). There are some tricks
>> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
>> things as they have been for many years. The distribution layout is
>> awkward, I think. We produce this "dist" folder at the top level that has
>> every JAR this project produces, *even contribs*. But why? I think
>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>> subdirectory there mixed with its dependencies (LTR has none but others
>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>> I think SolrJ is important enough that it deserves its very own top-level
>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>> could be configured to add the SolrJ libs so that we don't need to
>> redundantly put any of them in the distribution. There might be some
>> duplicated jars overall, but not many. Logging libs might be explicitly
>> excluded so that they are only in one spot. (Logging in Java is a mess)
>> >
>> > WDYT?
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Propose changing the "dist" layout
Posted by David Smiley <ds...@apache.org>.
I think your proposal is basically a competitive/alternative proposal to my
proposal for SOLR_VAR in
https://lists.apache.org/thread/3vvld3xnndtthtl7sfgdbsgkbtpm55b0
which you completely agreed with.
In addition to my proposal, we could move SOLR_HOME from
$SOLR_TIP/server/solr to $SOLR_TIP/home since that's what Solr knows this
as and there's no reason, I think, for it to be under Jetty's "server". I
know "solr home" / "home" is maybe ambiguous as to its purpose to a new
user but It's been a Solr concept since forever and it is possible for it
to not just have configuration but data as well depending on how someone
configures/uses it.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Jan 13, 2022 at 11:27 AM Houston Putman <ho...@apache.org> wrote:
> So would make sense to move configsets out to $SOLR_TIP/configsets and
>>> have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>>> something?
>>>
>>
>> Regardless of the rest of this discussion I think this would be a big
>> win. Currently it's a pain (especially for new users/contributors) to find
>> the default configsets that solr ships with. This would make it a lot
>> easier.
>>
>
> Looking at this again, maybe we just move solr/server/solr to
> solr/defaults or something else. This directory can be copied to $SOLR_HOME
> when starting solr, but I think it's nice to have all of those defaults
> (configsets, solr.xml, zoo.cfg) together.
>
> On Thu, Jan 13, 2022 at 11:08 AM Houston Putman <ho...@apache.org>
> wrote:
>
>> So would make sense to move configsets out to $SOLR_TIP/configsets and
>>> have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>>> something?
>>>
>>
>> Regardless of the rest of this discussion I think this would be a big
>> win. Currently it's a pain (especially for new users/contributors) to find
>> the default configsets that solr ships with. This would make it a lot
>> easier.
>>
>> That would also have the benefit of never polluting the git area with
>>> test data?
>>>
>>
>> Spent a few hours chasing down an issue that turned out to be this, so I
>> am very much +1 here
>>
>> +1 renaming for contribs -> *modules*
>> +1 for re-structuring dist
>>
>> On Thu, Jan 13, 2022 at 8:12 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>
>>> See https://issues.apache.org/jira/browse/SOLR-15917 for renaming of
>>> "contribs" discussion.
>>>
>>> Wrt server/ folder that is in reality the jetty distribution with an
>>> added "solr" folder for historic reasons. Yea, it is confusing. The "solr"
>>> folder will become $SOLR_HOME if you start Solr without an explicit
>>> $SOLR_HOME var. But it is also the authoritative location of our
>>> config-sets, even if you have $SOLR_HOME somewhere else (I believe?). So
>>> would make sense to move configsets out to $SOLR_TIP/configsets and have
>>> the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>>> something? That would also have the benefit of never polluting the git area
>>> with test data?
>>>
>>> Jan
>>>
>>> 13. jan. 2022 kl. 12:00 skrev Alessandro Benedetti <
>>> abenedetti@apache.org>:
>>>
>>> +1 renaming for contribs -> plugins
>>> +1 for re-structuring dist
>>>
>>> I also would like to raise some concern over the 'server' directory
>>> structure:
>>> README.txt lib resources solr-webapp
>>> contexts logs scripts start.jar
>>> etc modules solr
>>>
>>> I have been using it for years, and still, sometimes I get lost, some of
>>> them are not really clear:
>>> *resources* -> very generic, resources for what? currently, it contains
>>> logging configurations
>>> *etc* -> even more generic, currently it contains a lot of jetty stuff?
>>> *contexts* -> also in this case, not sure what to expect in here (web
>>> app context I suppose?)
>>> *solr* -> we are in the solr binary directory, so 'solr' is again very
>>> generic, this feels like the solr-home to me?
>>> *modules* -> also in this case, not sure what to expect
>>> *solr-webapp* -> is it just the UI? also in this case, it sounds a bit
>>> generic
>>> *start.jar* -> It's clear it's the entrypoint jar, but what does it
>>> contain?
>>>
>>> Cheers
>>>
>>> On Thu, 13 Jan 2022 at 02:42, David Smiley <ds...@apache.org> wrote:
>>>
>>>> (now to dev@solr.apache.org not lucene)
>>>> I'm just re-surfacing an older conversation where it is time for us to
>>>> take action.
>>>> Some JIRAs were recently created, and a separate thread about the
>>>> prometheus-exporter
>>>>
>>>> Full thread in ponymail:
>>>> https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <ja...@cominvent.com>
>>>> wrote:
>>>>
>>>>> When we collapse the solr/solr structure I hope we can keep the number
>>>>> of top-level folders in git to a minimum. There are too many already, so
>>>>> adding all contribs don’t help.
>>>>>
>>>>> I hope the cobtribs will be converted to 1st party packages soon, so
>>>>> perhaps “plugins” or “packages” is a good top level folder name?
>>>>>
>>>>> Those that are not yet converted can stay in “contribs”?
>>>>>
>>>>> Can we make solr-exporter a separate git repo? With separate artifacts
>>>>> and separate docker image. Don’t know if that means it must also be a full
>>>>> sub project?
>>>>>
>>>>> Jan Høydahl
>>>>>
>>>>> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>>>>>
>>>>>
>>>>> We (all?) agree to do away with "contrib" :-).
>>>>> I think a folder grouping the modules (that which can plug inside
>>>>> Solr) is useful as there are a number of them -- as such this is a nice
>>>>> organization IMO. There's a bunch of other stuff at the top level and I'd
>>>>> rather not intermix all our modules at this layer.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>
>>>>>
>>>>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>>>>>
>>>>>> We can have modules, but why do they need to be in an additional
>>>>>> folder deep? Why not just have langid next to solrj and core? Contrib to me
>>>>>> connotes experimental or unsupported, which these things are decidedly not.
>>>>>>
>>>>>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> The contrib folder is just a folder of modules -- optional plugins
>>>>>>> for solr-core. IMO we should simply rename "contrib" to "modules". I
>>>>>>> think the only non-module in there is the prometheus exporter which could
>>>>>>> move out. Mike, I'm not sure if you have a different notion of what
>>>>>>> "module" is? I believe most of us would be happy to move away from
>>>>>>> "contrib" wording, anyway.
>>>>>>>
>>>>>>> ~ David Smiley
>>>>>>> Apache Lucene/Solr Search Developer
>>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>>>>>
>>>>>>>> I think related to this, I would like to see some "contibs" moved
>>>>>>>> out
>>>>>>>> from the contrib folder and into proper modules. Right now the
>>>>>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>>>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>>>>>> hdfs all under it. LangId is already mentioned in our ref guide but
>>>>>>>> we
>>>>>>>> pretend like it is always present and don't think of it as a
>>>>>>>> contrib.
>>>>>>>> We kind of think of contrib as optional extra stuff, so maybe we
>>>>>>>> call
>>>>>>>> the things what they are - plugins and extensions? Then we don't
>>>>>>>> have
>>>>>>>> to think as hard about why certain things are showing up in which
>>>>>>>> lib
>>>>>>>> folders.
>>>>>>>>
>>>>>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>>>>>> having to type cor<tab> to disambiguate from con<tab> in my
>>>>>>>> terminal.
>>>>>>>>
>>>>>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > I believe we can do a fair amount of re-organization pertaining
>>>>>>>> to Jetty without losing the Jetty configuration that I think is valuable to
>>>>>>>> users who want to tweak something.
>>>>>>>> >
>>>>>>>> > ~ David Smiley
>>>>>>>> > Apache Lucene/Solr Search Developer
>>>>>>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <
>>>>>>>> jan.asf@cominvent.com> wrote:
>>>>>>>> >>
>>>>>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>>>>>>> possible. Perhaps rename "dist" -> "lib"?
>>>>>>>> >>
>>>>>>>> >> I wish we could get rid of the server (jetty) folder altogether,
>>>>>>>> and move everything from server/solr-webapp/webapp/WEB-INF/lib to
>>>>>>>> "lib/deps/". But that ties into custom boot-class, getting rid of web.xml
>>>>>>>> and building Jetty context in Java code.. I'm willing to help here if
>>>>>>>> others also want to go this direction. This would further hide Jetty as an
>>>>>>>> impl detail and let us organize stuff more freely.
>>>>>>>> >>
>>>>>>>> >> Jan
>>>>>>>> >>
>>>>>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>>>>>> >>
>>>>>>>> >> Bumping this conversation up, based on recent communication. I
>>>>>>>> have yet to take action but really any of us can.
>>>>>>>> >>
>>>>>>>> >> ~ David Smiley
>>>>>>>> >> Apache Lucene/Solr Search Developer
>>>>>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>>>>>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>>>>>>> "dist" folder.
>>>>>>>> >>>
>>>>>>>> >>> ~ David Smiley
>>>>>>>> >>> Apache Lucene/Solr Search Developer
>>>>>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>>>>>>> erickerickson@gmail.com> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Well, Solr has grown “organically” so some things just _are_,
>>>>>>>> like sunrises and plagues ;)
>>>>>>>> >>>>
>>>>>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder
>>>>>>>> how much of this is left over from the war days? Anything that’s lasted
>>>>>>>> through all the transformations Solr has is bound to need cleaning up
>>>>>>>> betimes.
>>>>>>>> >>>>
>>>>>>>> >>>> How would it relate to splitting Solr off into its own TLP? On
>>>>>>>> the surface, I’d guess the two efforts would be orthogonal, I mention it
>>>>>>>> just in case rearranging the layout would make that task easier or harder...
>>>>>>>> >>>>
>>>>>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <
>>>>>>>> dsmiley@apache.org> wrote:
>>>>>>>> >>>> >
>>>>>>>> >>>> > I've been doing a bit of dependency work in one of our
>>>>>>>> contribs, and observing more closely than usual exactly what we produce in
>>>>>>>> the distribution layout (result of gradlew assemble). There are some
>>>>>>>> tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to
>>>>>>>> keep things as they have been for many years. The distribution layout is
>>>>>>>> awkward, I think. We produce this "dist" folder at the top level that has
>>>>>>>> every JAR this project produces, *even contribs*. But why? I think
>>>>>>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>>>>>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>>>>>>> subdirectory there mixed with its dependencies (LTR has none but others
>>>>>>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>>>>>>> I think SolrJ is important enough that it deserves its very own top-level
>>>>>>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>>>>>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>>>>>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>>>>>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>>>>>>> could be configured to add the SolrJ libs so that we don't need to
>>>>>>>> redundantly put any of them in the distribution. There might be some
>>>>>>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>>>>>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>>>>>>> >>>> >
>>>>>>>> >>>> > WDYT?
>>>>>>>> >>>> >
>>>>>>>> >>>> > ~ David Smiley
>>>>>>>> >>>> > Apache Lucene/Solr Search Developer
>>>>>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>>> >>>>
>>>>>>>> >>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>
Re: Propose changing the "dist" layout
Posted by Houston Putman <ho...@apache.org>.
>
> So would make sense to move configsets out to $SOLR_TIP/configsets and
>> have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>> something?
>>
>
> Regardless of the rest of this discussion I think this would be a big win.
> Currently it's a pain (especially for new users/contributors) to find the
> default configsets that solr ships with. This would make it a lot easier.
>
Looking at this again, maybe we just move solr/server/solr to solr/defaults
or something else. This directory can be copied to $SOLR_HOME when starting
solr, but I think it's nice to have all of those defaults (configsets,
solr.xml, zoo.cfg) together.
On Thu, Jan 13, 2022 at 11:08 AM Houston Putman <ho...@apache.org> wrote:
> So would make sense to move configsets out to $SOLR_TIP/configsets and
>> have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>> something?
>>
>
> Regardless of the rest of this discussion I think this would be a big win.
> Currently it's a pain (especially for new users/contributors) to find the
> default configsets that solr ships with. This would make it a lot easier.
>
> That would also have the benefit of never polluting the git area with test
>> data?
>>
>
> Spent a few hours chasing down an issue that turned out to be this, so I
> am very much +1 here
>
> +1 renaming for contribs -> *modules*
> +1 for re-structuring dist
>
> On Thu, Jan 13, 2022 at 8:12 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> See https://issues.apache.org/jira/browse/SOLR-15917 for renaming of
>> "contribs" discussion.
>>
>> Wrt server/ folder that is in reality the jetty distribution with an
>> added "solr" folder for historic reasons. Yea, it is confusing. The "solr"
>> folder will become $SOLR_HOME if you start Solr without an explicit
>> $SOLR_HOME var. But it is also the authoritative location of our
>> config-sets, even if you have $SOLR_HOME somewhere else (I believe?). So
>> would make sense to move configsets out to $SOLR_TIP/configsets and have
>> the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
>> something? That would also have the benefit of never polluting the git area
>> with test data?
>>
>> Jan
>>
>> 13. jan. 2022 kl. 12:00 skrev Alessandro Benedetti <abenedetti@apache.org
>> >:
>>
>> +1 renaming for contribs -> plugins
>> +1 for re-structuring dist
>>
>> I also would like to raise some concern over the 'server' directory
>> structure:
>> README.txt lib resources solr-webapp
>> contexts logs scripts start.jar
>> etc modules solr
>>
>> I have been using it for years, and still, sometimes I get lost, some of
>> them are not really clear:
>> *resources* -> very generic, resources for what? currently, it contains
>> logging configurations
>> *etc* -> even more generic, currently it contains a lot of jetty stuff?
>> *contexts* -> also in this case, not sure what to expect in here (web
>> app context I suppose?)
>> *solr* -> we are in the solr binary directory, so 'solr' is again very
>> generic, this feels like the solr-home to me?
>> *modules* -> also in this case, not sure what to expect
>> *solr-webapp* -> is it just the UI? also in this case, it sounds a bit
>> generic
>> *start.jar* -> It's clear it's the entrypoint jar, but what does it
>> contain?
>>
>> Cheers
>>
>> On Thu, 13 Jan 2022 at 02:42, David Smiley <ds...@apache.org> wrote:
>>
>>> (now to dev@solr.apache.org not lucene)
>>> I'm just re-surfacing an older conversation where it is time for us to
>>> take action.
>>> Some JIRAs were recently created, and a separate thread about the
>>> prometheus-exporter
>>>
>>> Full thread in ponymail:
>>> https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <ja...@cominvent.com>
>>> wrote:
>>>
>>>> When we collapse the solr/solr structure I hope we can keep the number
>>>> of top-level folders in git to a minimum. There are too many already, so
>>>> adding all contribs don’t help.
>>>>
>>>> I hope the cobtribs will be converted to 1st party packages soon, so
>>>> perhaps “plugins” or “packages” is a good top level folder name?
>>>>
>>>> Those that are not yet converted can stay in “contribs”?
>>>>
>>>> Can we make solr-exporter a separate git repo? With separate artifacts
>>>> and separate docker image. Don’t know if that means it must also be a full
>>>> sub project?
>>>>
>>>> Jan Høydahl
>>>>
>>>> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>>>>
>>>>
>>>> We (all?) agree to do away with "contrib" :-).
>>>> I think a folder grouping the modules (that which can plug inside Solr)
>>>> is useful as there are a number of them -- as such this is a nice
>>>> organization IMO. There's a bunch of other stuff at the top level and I'd
>>>> rather not intermix all our modules at this layer.
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>>>>
>>>>> We can have modules, but why do they need to be in an additional
>>>>> folder deep? Why not just have langid next to solrj and core? Contrib to me
>>>>> connotes experimental or unsupported, which these things are decidedly not.
>>>>>
>>>>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> The contrib folder is just a folder of modules -- optional plugins
>>>>>> for solr-core. IMO we should simply rename "contrib" to "modules". I
>>>>>> think the only non-module in there is the prometheus exporter which could
>>>>>> move out. Mike, I'm not sure if you have a different notion of what
>>>>>> "module" is? I believe most of us would be happy to move away from
>>>>>> "contrib" wording, anyway.
>>>>>>
>>>>>> ~ David Smiley
>>>>>> Apache Lucene/Solr Search Developer
>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>>>>
>>>>>>> I think related to this, I would like to see some "contibs" moved out
>>>>>>> from the contrib folder and into proper modules. Right now the
>>>>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>>>>> hdfs all under it. LangId is already mentioned in our ref guide but
>>>>>>> we
>>>>>>> pretend like it is always present and don't think of it as a contrib.
>>>>>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>>>>>> the things what they are - plugins and extensions? Then we don't have
>>>>>>> to think as hard about why certain things are showing up in which lib
>>>>>>> folders.
>>>>>>>
>>>>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>>>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>>>>>
>>>>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > I believe we can do a fair amount of re-organization pertaining to
>>>>>>> Jetty without losing the Jetty configuration that I think is valuable to
>>>>>>> users who want to tweak something.
>>>>>>> >
>>>>>>> > ~ David Smiley
>>>>>>> > Apache Lucene/Solr Search Developer
>>>>>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>>>>>> possible. Perhaps rename "dist" -> "lib"?
>>>>>>> >>
>>>>>>> >> I wish we could get rid of the server (jetty) folder altogether,
>>>>>>> and move everything from server/solr-webapp/webapp/WEB-INF/lib to
>>>>>>> "lib/deps/". But that ties into custom boot-class, getting rid of web.xml
>>>>>>> and building Jetty context in Java code.. I'm willing to help here if
>>>>>>> others also want to go this direction. This would further hide Jetty as an
>>>>>>> impl detail and let us organize stuff more freely.
>>>>>>> >>
>>>>>>> >> Jan
>>>>>>> >>
>>>>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>>>>> >>
>>>>>>> >> Bumping this conversation up, based on recent communication. I
>>>>>>> have yet to take action but really any of us can.
>>>>>>> >>
>>>>>>> >> ~ David Smiley
>>>>>>> >> Apache Lucene/Solr Search Developer
>>>>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>>>>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>>>>>> "dist" folder.
>>>>>>> >>>
>>>>>>> >>> ~ David Smiley
>>>>>>> >>> Apache Lucene/Solr Search Developer
>>>>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>>>>>> erickerickson@gmail.com> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Well, Solr has grown “organically” so some things just _are_,
>>>>>>> like sunrises and plagues ;)
>>>>>>> >>>>
>>>>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how
>>>>>>> much of this is left over from the war days? Anything that’s lasted through
>>>>>>> all the transformations Solr has is bound to need cleaning up betimes.
>>>>>>> >>>>
>>>>>>> >>>> How would it relate to splitting Solr off into its own TLP? On
>>>>>>> the surface, I’d guess the two efforts would be orthogonal, I mention it
>>>>>>> just in case rearranging the layout would make that task easier or harder...
>>>>>>> >>>>
>>>>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <
>>>>>>> dsmiley@apache.org> wrote:
>>>>>>> >>>> >
>>>>>>> >>>> > I've been doing a bit of dependency work in one of our
>>>>>>> contribs, and observing more closely than usual exactly what we produce in
>>>>>>> the distribution layout (result of gradlew assemble). There are some
>>>>>>> tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to
>>>>>>> keep things as they have been for many years. The distribution layout is
>>>>>>> awkward, I think. We produce this "dist" folder at the top level that has
>>>>>>> every JAR this project produces, *even contribs*. But why? I think
>>>>>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>>>>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>>>>>> subdirectory there mixed with its dependencies (LTR has none but others
>>>>>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>>>>>> I think SolrJ is important enough that it deserves its very own top-level
>>>>>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>>>>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>>>>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>>>>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>>>>>> could be configured to add the SolrJ libs so that we don't need to
>>>>>>> redundantly put any of them in the distribution. There might be some
>>>>>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>>>>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>>>>>> >>>> >
>>>>>>> >>>> > WDYT?
>>>>>>> >>>> >
>>>>>>> >>>> > ~ David Smiley
>>>>>>> >>>> > Apache Lucene/Solr Search Developer
>>>>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>> >>>>
>>>>>>> >>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>
Re: Propose changing the "dist" layout
Posted by Houston Putman <ho...@apache.org>.
>
> So would make sense to move configsets out to $SOLR_TIP/configsets and
> have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or
> something?
>
Regardless of the rest of this discussion I think this would be a big win.
Currently it's a pain (especially for new users/contributors) to find the
default configsets that solr ships with. This would make it a lot easier.
That would also have the benefit of never polluting the git area with test
> data?
>
Spent a few hours chasing down an issue that turned out to be this, so I am
very much +1 here
+1 renaming for contribs -> *modules*
+1 for re-structuring dist
On Thu, Jan 13, 2022 at 8:12 AM Jan Høydahl <ja...@cominvent.com> wrote:
> See https://issues.apache.org/jira/browse/SOLR-15917 for renaming of
> "contribs" discussion.
>
> Wrt server/ folder that is in reality the jetty distribution with an added
> "solr" folder for historic reasons. Yea, it is confusing. The "solr" folder
> will become $SOLR_HOME if you start Solr without an explicit $SOLR_HOME
> var. But it is also the authoritative location of our config-sets, even if
> you have $SOLR_HOME somewhere else (I believe?). So would make sense to
> move configsets out to $SOLR_TIP/configsets and have the default location
> of SOLR_HOME be not "./solr" but $TEMP/solr or something? That would also
> have the benefit of never polluting the git area with test data?
>
> Jan
>
> 13. jan. 2022 kl. 12:00 skrev Alessandro Benedetti <abenedetti@apache.org
> >:
>
> +1 renaming for contribs -> plugins
> +1 for re-structuring dist
>
> I also would like to raise some concern over the 'server' directory
> structure:
> README.txt lib resources solr-webapp
> contexts logs scripts start.jar
> etc modules solr
>
> I have been using it for years, and still, sometimes I get lost, some of
> them are not really clear:
> *resources* -> very generic, resources for what? currently, it contains
> logging configurations
> *etc* -> even more generic, currently it contains a lot of jetty stuff?
> *contexts* -> also in this case, not sure what to expect in here (web app
> context I suppose?)
> *solr* -> we are in the solr binary directory, so 'solr' is again very
> generic, this feels like the solr-home to me?
> *modules* -> also in this case, not sure what to expect
> *solr-webapp* -> is it just the UI? also in this case, it sounds a bit
> generic
> *start.jar* -> It's clear it's the entrypoint jar, but what does it
> contain?
>
> Cheers
>
> On Thu, 13 Jan 2022 at 02:42, David Smiley <ds...@apache.org> wrote:
>
>> (now to dev@solr.apache.org not lucene)
>> I'm just re-surfacing an older conversation where it is time for us to
>> take action.
>> Some JIRAs were recently created, and a separate thread about the
>> prometheus-exporter
>>
>> Full thread in ponymail:
>> https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>
>>> When we collapse the solr/solr structure I hope we can keep the number
>>> of top-level folders in git to a minimum. There are too many already, so
>>> adding all contribs don’t help.
>>>
>>> I hope the cobtribs will be converted to 1st party packages soon, so
>>> perhaps “plugins” or “packages” is a good top level folder name?
>>>
>>> Those that are not yet converted can stay in “contribs”?
>>>
>>> Can we make solr-exporter a separate git repo? With separate artifacts
>>> and separate docker image. Don’t know if that means it must also be a full
>>> sub project?
>>>
>>> Jan Høydahl
>>>
>>> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>>>
>>>
>>> We (all?) agree to do away with "contrib" :-).
>>> I think a folder grouping the modules (that which can plug inside Solr)
>>> is useful as there are a number of them -- as such this is a nice
>>> organization IMO. There's a bunch of other stuff at the top level and I'd
>>> rather not intermix all our modules at this layer.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>>>
>>>> We can have modules, but why do they need to be in an additional folder
>>>> deep? Why not just have langid next to solrj and core? Contrib to me
>>>> connotes experimental or unsupported, which these things are decidedly not.
>>>>
>>>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org>
>>>> wrote:
>>>>
>>>>> The contrib folder is just a folder of modules -- optional plugins for
>>>>> solr-core. IMO we should simply rename "contrib" to "modules". I think
>>>>> the only non-module in there is the prometheus exporter which could move
>>>>> out. Mike, I'm not sure if you have a different notion of what "module"
>>>>> is? I believe most of us would be happy to move away from "contrib"
>>>>> wording, anyway.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>
>>>>>
>>>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>>>
>>>>>> I think related to this, I would like to see some "contibs" moved out
>>>>>> from the contrib folder and into proper modules. Right now the
>>>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>>>> hdfs all under it. LangId is already mentioned in our ref guide but we
>>>>>> pretend like it is always present and don't think of it as a contrib.
>>>>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>>>>> the things what they are - plugins and extensions? Then we don't have
>>>>>> to think as hard about why certain things are showing up in which lib
>>>>>> folders.
>>>>>>
>>>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>>>>
>>>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org>
>>>>>> wrote:
>>>>>> >
>>>>>> > I believe we can do a fair amount of re-organization pertaining to
>>>>>> Jetty without losing the Jetty configuration that I think is valuable to
>>>>>> users who want to tweak something.
>>>>>> >
>>>>>> > ~ David Smiley
>>>>>> > Apache Lucene/Solr Search Developer
>>>>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>>>>> possible. Perhaps rename "dist" -> "lib"?
>>>>>> >>
>>>>>> >> I wish we could get rid of the server (jetty) folder altogether,
>>>>>> and move everything from server/solr-webapp/webapp/WEB-INF/lib to
>>>>>> "lib/deps/". But that ties into custom boot-class, getting rid of web.xml
>>>>>> and building Jetty context in Java code.. I'm willing to help here if
>>>>>> others also want to go this direction. This would further hide Jetty as an
>>>>>> impl detail and let us organize stuff more freely.
>>>>>> >>
>>>>>> >> Jan
>>>>>> >>
>>>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>>>> >>
>>>>>> >> Bumping this conversation up, based on recent communication. I
>>>>>> have yet to take action but really any of us can.
>>>>>> >>
>>>>>> >> ~ David Smiley
>>>>>> >> Apache Lucene/Solr Search Developer
>>>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>>>> >>
>>>>>> >>
>>>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>>>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>>>>> "dist" folder.
>>>>>> >>>
>>>>>> >>> ~ David Smiley
>>>>>> >>> Apache Lucene/Solr Search Developer
>>>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>>>>> erickerickson@gmail.com> wrote:
>>>>>> >>>>
>>>>>> >>>> Well, Solr has grown “organically” so some things just _are_,
>>>>>> like sunrises and plagues ;)
>>>>>> >>>>
>>>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how
>>>>>> much of this is left over from the war days? Anything that’s lasted through
>>>>>> all the transformations Solr has is bound to need cleaning up betimes.
>>>>>> >>>>
>>>>>> >>>> How would it relate to splitting Solr off into its own TLP? On
>>>>>> the surface, I’d guess the two efforts would be orthogonal, I mention it
>>>>>> just in case rearranging the layout would make that task easier or harder...
>>>>>> >>>>
>>>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
>>>>>> wrote:
>>>>>> >>>> >
>>>>>> >>>> > I've been doing a bit of dependency work in one of our
>>>>>> contribs, and observing more closely than usual exactly what we produce in
>>>>>> the distribution layout (result of gradlew assemble). There are some
>>>>>> tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to
>>>>>> keep things as they have been for many years. The distribution layout is
>>>>>> awkward, I think. We produce this "dist" folder at the top level that has
>>>>>> every JAR this project produces, *even contribs*. But why? I think
>>>>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>>>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>>>>> subdirectory there mixed with its dependencies (LTR has none but others
>>>>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>>>>> I think SolrJ is important enough that it deserves its very own top-level
>>>>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>>>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>>>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>>>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>>>>> could be configured to add the SolrJ libs so that we don't need to
>>>>>> redundantly put any of them in the distribution. There might be some
>>>>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>>>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>>>>> >>>> >
>>>>>> >>>> > WDYT?
>>>>>> >>>> >
>>>>>> >>>> > ~ David Smiley
>>>>>> >>>> > Apache Lucene/Solr Search Developer
>>>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>> >>>>
>>>>>> >>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>
>>>>>>
>
Re: Propose changing the "dist" layout
Posted by Jan Høydahl <ja...@cominvent.com>.
See https://issues.apache.org/jira/browse/SOLR-15917 for renaming of "contribs" discussion.
Wrt server/ folder that is in reality the jetty distribution with an added "solr" folder for historic reasons. Yea, it is confusing. The "solr" folder will become $SOLR_HOME if you start Solr without an explicit $SOLR_HOME var. But it is also the authoritative location of our config-sets, even if you have $SOLR_HOME somewhere else (I believe?). So would make sense to move configsets out to $SOLR_TIP/configsets and have the default location of SOLR_HOME be not "./solr" but $TEMP/solr or something? That would also have the benefit of never polluting the git area with test data?
Jan
> 13. jan. 2022 kl. 12:00 skrev Alessandro Benedetti <ab...@apache.org>:
>
> +1 renaming for contribs -> plugins
> +1 for re-structuring dist
>
> I also would like to raise some concern over the 'server' directory structure:
> README.txt lib resources solr-webapp
> contexts logs scripts start.jar
> etc modules solr
>
> I have been using it for years, and still, sometimes I get lost, some of them are not really clear:
> resources -> very generic, resources for what? currently, it contains logging configurations
> etc -> even more generic, currently it contains a lot of jetty stuff?
> contexts -> also in this case, not sure what to expect in here (web app context I suppose?)
> solr -> we are in the solr binary directory, so 'solr' is again very generic, this feels like the solr-home to me?
> modules -> also in this case, not sure what to expect
> solr-webapp -> is it just the UI? also in this case, it sounds a bit generic
> start.jar -> It's clear it's the entrypoint jar, but what does it contain?
>
> Cheers
>
> On Thu, 13 Jan 2022 at 02:42, David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
> (now to dev@solr.apache.org <ma...@solr.apache.org> not lucene)
> I'm just re-surfacing an older conversation where it is time for us to take action.
> Some JIRAs were recently created, and a separate thread about the prometheus-exporter
>
> Full thread in ponymail: https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61 <https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61>
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>
> On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <jan.asf@cominvent.com <ma...@cominvent.com>> wrote:
> When we collapse the solr/solr structure I hope we can keep the number of top-level folders in git to a minimum. There are too many already, so adding all contribs don’t help.
>
> I hope the cobtribs will be converted to 1st party packages soon, so perhaps “plugins” or “packages” is a good top level folder name?
>
> Those that are not yet converted can stay in “contribs”?
>
> Can we make solr-exporter a separate git repo? With separate artifacts and separate docker image. Don’t know if that means it must also be a full sub project?
>
> Jan Høydahl
>
>> 11. jun. 2021 kl. 22:46 skrev David Smiley <dsmiley@apache.org <ma...@apache.org>>:
>>
>>
>> We (all?) agree to do away with "contrib" :-).
>> I think a folder grouping the modules (that which can plug inside Solr) is useful as there are a number of them -- as such this is a nice organization IMO. There's a bunch of other stuff at the top level and I'd rather not intermix all our modules at this layer.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>>
>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <mdrob@mdrob.com <ma...@mdrob.com>> wrote:
>> We can have modules, but why do they need to be in an additional folder deep? Why not just have langid next to solrj and core? Contrib to me connotes experimental or unsupported, which these things are decidedly not.
>>
>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
>> The contrib folder is just a folder of modules -- optional plugins for solr-core. IMO we should simply rename "contrib" to "modules". I think the only non-module in there is the prometheus exporter which could move out. Mike, I'm not sure if you have a different notion of what "module" is? I believe most of us would be happy to move away from "contrib" wording, anyway.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>>
>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <mdrob@mdrob.com <ma...@mdrob.com>> wrote:
>> I think related to this, I would like to see some "contibs" moved out
>> from the contrib folder and into proper modules. Right now the
>> definition of contrib seems to be anything that isn't core or solrj,
>> but maybe there is room for a backup module that has gcs and s3 and
>> hdfs all under it. LangId is already mentioned in our ref guide but we
>> pretend like it is always present and don't think of it as a contrib.
>> We kind of think of contrib as optional extra stuff, so maybe we call
>> the things what they are - plugins and extensions? Then we don't have
>> to think as hard about why certain things are showing up in which lib
>> folders.
>>
>> Also, minor benefit, I would then be able to type c<tab> instead of
>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>
>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
>> >
>> > I believe we can do a fair amount of re-organization pertaining to Jetty without losing the Jetty configuration that I think is valuable to users who want to tweak something.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>> >
>> >
>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <jan.asf@cominvent.com <ma...@cominvent.com>> wrote:
>> >>
>> >> +1 to a cleanup here for 9.0. As clean and neat organization as possible. Perhaps rename "dist" -> "lib"?
>> >>
>> >> I wish we could get rid of the server (jetty) folder altogether, and move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But that ties into custom boot-class, getting rid of web.xml and building Jetty context in Java code.. I'm willing to help here if others also want to go this direction. This would further hide Jetty as an impl detail and let us organize stuff more freely.
>> >>
>> >> Jan
>> >>
>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <dsmiley@apache.org <ma...@apache.org>>:
>> >>
>> >> Bumping this conversation up, based on recent communication. I have yet to take action but really any of us can.
>> >>
>> >> ~ David Smiley
>> >> Apache Lucene/Solr Search Developer
>> >> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>> >>
>> >>
>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
>> >>>
>> >>> I'll proceed on this with lazy consensus. I suspect most of us don't care, unsurprisingly since I doubt anyone has any fondness for the "dist" folder.
>> >>>
>> >>> ~ David Smiley
>> >>> Apache Lucene/Solr Search Developer
>> >>> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>> >>>
>> >>>
>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <erickerickson@gmail.com <ma...@gmail.com>> wrote:
>> >>>>
>> >>>> Well, Solr has grown “organically” so some things just _are_, like sunrises and plagues ;)
>> >>>>
>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much of this is left over from the war days? Anything that’s lasted through all the transformations Solr has is bound to need cleaning up betimes.
>> >>>>
>> >>>> How would it relate to splitting Solr off into its own TLP? On the surface, I’d guess the two efforts would be orthogonal, I mention it just in case rearranging the layout would make that task easier or harder...
>> >>>>
>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
>> >>>> >
>> >>>> > I've been doing a bit of dependency work in one of our contribs, and observing more closely than usual exactly what we produce in the distribution layout (result of gradlew assemble). There are some tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep things as they have been for many years. The distribution layout is awkward, I think. We produce this "dist" folder at the top level that has every JAR this project produces, *even contribs*. But why? I think contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is empty except for a README.txt... IMO it ought to have the JAR in a "lib" subdirectory there mixed with its dependencies (LTR has none but others sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ? I think SolrJ is important enough that it deserves its very own top-level directory "solrj", and like the contribs, with a "lib" alongside it. Maybe Solrj's optional dependencies could be in a lib-optional dir next to it or lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains the solr-core JAR but this is redundant. Furthermore, the server webapp could be configured to add the SolrJ libs so that we don't need to redundantly put any of them in the distribution. There might be some duplicated jars overall, but not many. Logging libs might be explicitly excluded so that they are only in one spot. (Logging in Java is a mess)
>> >>>> >
>> >>>> > WDYT?
>> >>>> >
>> >>>> > ~ David Smiley
>> >>>> > Apache Lucene/Solr Search Developer
>> >>>> > http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <ma...@lucene.apache.org>
>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org <ma...@lucene.apache.org>
>> >>>>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <ma...@lucene.apache.org>
>> For additional commands, e-mail: dev-help@lucene.apache.org <ma...@lucene.apache.org>
>>
Re: Propose changing the "dist" layout
Posted by Alessandro Benedetti <ab...@apache.org>.
+1 renaming for contribs -> plugins
+1 for re-structuring dist
I also would like to raise some concern over the 'server' directory
structure:
README.txt lib resources solr-webapp
contexts logs scripts start.jar
etc modules solr
I have been using it for years, and still, sometimes I get lost, some of
them are not really clear:
*resources* -> very generic, resources for what? currently, it contains
logging configurations
*etc* -> even more generic, currently it contains a lot of jetty stuff?
*contexts* -> also in this case, not sure what to expect in here (web app
context I suppose?)
*solr* -> we are in the solr binary directory, so 'solr' is again very
generic, this feels like the solr-home to me?
*modules* -> also in this case, not sure what to expect
*solr-webapp* -> is it just the UI? also in this case, it sounds a bit
generic
*start.jar* -> It's clear it's the entrypoint jar, but what does it contain?
Cheers
On Thu, 13 Jan 2022 at 02:42, David Smiley <ds...@apache.org> wrote:
> (now to dev@solr.apache.org not lucene)
> I'm just re-surfacing an older conversation where it is time for us to
> take action.
> Some JIRAs were recently created, and a separate thread about the
> prometheus-exporter
>
> Full thread in ponymail:
> https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> When we collapse the solr/solr structure I hope we can keep the number of
>> top-level folders in git to a minimum. There are too many already, so
>> adding all contribs don’t help.
>>
>> I hope the cobtribs will be converted to 1st party packages soon, so
>> perhaps “plugins” or “packages” is a good top level folder name?
>>
>> Those that are not yet converted can stay in “contribs”?
>>
>> Can we make solr-exporter a separate git repo? With separate artifacts
>> and separate docker image. Don’t know if that means it must also be a full
>> sub project?
>>
>> Jan Høydahl
>>
>> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>>
>>
>> We (all?) agree to do away with "contrib" :-).
>> I think a folder grouping the modules (that which can plug inside Solr)
>> is useful as there are a number of them -- as such this is a nice
>> organization IMO. There's a bunch of other stuff at the top level and I'd
>> rather not intermix all our modules at this layer.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>>
>>> We can have modules, but why do they need to be in an additional folder
>>> deep? Why not just have langid next to solrj and core? Contrib to me
>>> connotes experimental or unsupported, which these things are decidedly not.
>>>
>>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org> wrote:
>>>
>>>> The contrib folder is just a folder of modules -- optional plugins for
>>>> solr-core. IMO we should simply rename "contrib" to "modules". I think
>>>> the only non-module in there is the prometheus exporter which could move
>>>> out. Mike, I'm not sure if you have a different notion of what "module"
>>>> is? I believe most of us would be happy to move away from "contrib"
>>>> wording, anyway.
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>>
>>>>> I think related to this, I would like to see some "contibs" moved out
>>>>> from the contrib folder and into proper modules. Right now the
>>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>>> hdfs all under it. LangId is already mentioned in our ref guide but we
>>>>> pretend like it is always present and don't think of it as a contrib.
>>>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>>>> the things what they are - plugins and extensions? Then we don't have
>>>>> to think as hard about why certain things are showing up in which lib
>>>>> folders.
>>>>>
>>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>>>
>>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > I believe we can do a fair amount of re-organization pertaining to
>>>>> Jetty without losing the Jetty configuration that I think is valuable to
>>>>> users who want to tweak something.
>>>>> >
>>>>> > ~ David Smiley
>>>>> > Apache Lucene/Solr Search Developer
>>>>> > http://www.linkedin.com/in/davidwsmiley
>>>>> >
>>>>> >
>>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>>>>> wrote:
>>>>> >>
>>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>>>> possible. Perhaps rename "dist" -> "lib"?
>>>>> >>
>>>>> >> I wish we could get rid of the server (jetty) folder altogether,
>>>>> and move everything from server/solr-webapp/webapp/WEB-INF/lib to
>>>>> "lib/deps/". But that ties into custom boot-class, getting rid of web.xml
>>>>> and building Jetty context in Java code.. I'm willing to help here if
>>>>> others also want to go this direction. This would further hide Jetty as an
>>>>> impl detail and let us organize stuff more freely.
>>>>> >>
>>>>> >> Jan
>>>>> >>
>>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>>> >>
>>>>> >> Bumping this conversation up, based on recent communication. I
>>>>> have yet to take action but really any of us can.
>>>>> >>
>>>>> >> ~ David Smiley
>>>>> >> Apache Lucene/Solr Search Developer
>>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>>> >>
>>>>> >>
>>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>>>> wrote:
>>>>> >>>
>>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>>>> "dist" folder.
>>>>> >>>
>>>>> >>> ~ David Smiley
>>>>> >>> Apache Lucene/Solr Search Developer
>>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>>> >>>
>>>>> >>>
>>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>>>> erickerickson@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Well, Solr has grown “organically” so some things just _are_,
>>>>> like sunrises and plagues ;)
>>>>> >>>>
>>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how
>>>>> much of this is left over from the war days? Anything that’s lasted through
>>>>> all the transformations Solr has is bound to need cleaning up betimes.
>>>>> >>>>
>>>>> >>>> How would it relate to splitting Solr off into its own TLP? On
>>>>> the surface, I’d guess the two efforts would be orthogonal, I mention it
>>>>> just in case rearranging the layout would make that task easier or harder...
>>>>> >>>>
>>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
>>>>> wrote:
>>>>> >>>> >
>>>>> >>>> > I've been doing a bit of dependency work in one of our
>>>>> contribs, and observing more closely than usual exactly what we produce in
>>>>> the distribution layout (result of gradlew assemble). There are some
>>>>> tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to
>>>>> keep things as they have been for many years. The distribution layout is
>>>>> awkward, I think. We produce this "dist" folder at the top level that has
>>>>> every JAR this project produces, *even contribs*. But why? I think
>>>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>>>> subdirectory there mixed with its dependencies (LTR has none but others
>>>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>>>> I think SolrJ is important enough that it deserves its very own top-level
>>>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>>>> could be configured to add the SolrJ libs so that we don't need to
>>>>> redundantly put any of them in the distribution. There might be some
>>>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>>>> >>>> >
>>>>> >>>> > WDYT?
>>>>> >>>> >
>>>>> >>>> > ~ David Smiley
>>>>> >>>> > Apache Lucene/Solr Search Developer
>>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> ---------------------------------------------------------------------
>>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>> >>>>
>>>>> >>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>>
Re: Propose changing the "dist" layout
Posted by David Smiley <ds...@apache.org>.
(now to dev@solr.apache.org not lucene)
I'm just re-surfacing an older conversation where it is time for us to take
action.
Some JIRAs were recently created, and a separate thread about the
prometheus-exporter
Full thread in ponymail:
https://lists.apache.org/thread/jbs4ds0w3r3v1hto9rqhs4qq1xfk5z61
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Sun, Jun 13, 2021 at 7:31 AM Jan Høydahl <ja...@cominvent.com> wrote:
> When we collapse the solr/solr structure I hope we can keep the number of
> top-level folders in git to a minimum. There are too many already, so
> adding all contribs don’t help.
>
> I hope the cobtribs will be converted to 1st party packages soon, so
> perhaps “plugins” or “packages” is a good top level folder name?
>
> Those that are not yet converted can stay in “contribs”?
>
> Can we make solr-exporter a separate git repo? With separate artifacts and
> separate docker image. Don’t know if that means it must also be a full sub
> project?
>
> Jan Høydahl
>
> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>
>
> We (all?) agree to do away with "contrib" :-).
> I think a folder grouping the modules (that which can plug inside Solr) is
> useful as there are a number of them -- as such this is a nice organization
> IMO. There's a bunch of other stuff at the top level and I'd rather not
> intermix all our modules at this layer.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>
>> We can have modules, but why do they need to be in an additional folder
>> deep? Why not just have langid next to solrj and core? Contrib to me
>> connotes experimental or unsupported, which these things are decidedly not.
>>
>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org> wrote:
>>
>>> The contrib folder is just a folder of modules -- optional plugins for
>>> solr-core. IMO we should simply rename "contrib" to "modules". I think
>>> the only non-module in there is the prometheus exporter which could move
>>> out. Mike, I'm not sure if you have a different notion of what "module"
>>> is? I believe most of us would be happy to move away from "contrib"
>>> wording, anyway.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>
>>>> I think related to this, I would like to see some "contibs" moved out
>>>> from the contrib folder and into proper modules. Right now the
>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>> hdfs all under it. LangId is already mentioned in our ref guide but we
>>>> pretend like it is always present and don't think of it as a contrib.
>>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>>> the things what they are - plugins and extensions? Then we don't have
>>>> to think as hard about why certain things are showing up in which lib
>>>> folders.
>>>>
>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>>
>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org>
>>>> wrote:
>>>> >
>>>> > I believe we can do a fair amount of re-organization pertaining to
>>>> Jetty without losing the Jetty configuration that I think is valuable to
>>>> users who want to tweak something.
>>>> >
>>>> > ~ David Smiley
>>>> > Apache Lucene/Solr Search Developer
>>>> > http://www.linkedin.com/in/davidwsmiley
>>>> >
>>>> >
>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>>>> wrote:
>>>> >>
>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>>> possible. Perhaps rename "dist" -> "lib"?
>>>> >>
>>>> >> I wish we could get rid of the server (jetty) folder altogether, and
>>>> move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/".
>>>> But that ties into custom boot-class, getting rid of web.xml and building
>>>> Jetty context in Java code.. I'm willing to help here if others also want
>>>> to go this direction. This would further hide Jetty as an impl detail and
>>>> let us organize stuff more freely.
>>>> >>
>>>> >> Jan
>>>> >>
>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>> >>
>>>> >> Bumping this conversation up, based on recent communication. I have
>>>> yet to take action but really any of us can.
>>>> >>
>>>> >> ~ David Smiley
>>>> >> Apache Lucene/Solr Search Developer
>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>> >>
>>>> >>
>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>>> wrote:
>>>> >>>
>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>>> "dist" folder.
>>>> >>>
>>>> >>> ~ David Smiley
>>>> >>> Apache Lucene/Solr Search Developer
>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>> >>>
>>>> >>>
>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>>> erickerickson@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Well, Solr has grown “organically” so some things just _are_, like
>>>> sunrises and plagues ;)
>>>> >>>>
>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how
>>>> much of this is left over from the war days? Anything that’s lasted through
>>>> all the transformations Solr has is bound to need cleaning up betimes.
>>>> >>>>
>>>> >>>> How would it relate to splitting Solr off into its own TLP? On the
>>>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>>>> in case rearranging the layout would make that task easier or harder...
>>>> >>>>
>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
>>>> wrote:
>>>> >>>> >
>>>> >>>> > I've been doing a bit of dependency work in one of our contribs,
>>>> and observing more closely than usual exactly what we produce in the
>>>> distribution layout (result of gradlew assemble). There are some tricks
>>>> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
>>>> things as they have been for many years. The distribution layout is
>>>> awkward, I think. We produce this "dist" folder at the top level that has
>>>> every JAR this project produces, *even contribs*. But why? I think
>>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>>> subdirectory there mixed with its dependencies (LTR has none but others
>>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>>> I think SolrJ is important enough that it deserves its very own top-level
>>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>>> could be configured to add the SolrJ libs so that we don't need to
>>>> redundantly put any of them in the distribution. There might be some
>>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>>> >>>> >
>>>> >>>> > WDYT?
>>>> >>>> >
>>>> >>>> > ~ David Smiley
>>>> >>>> > Apache Lucene/Solr Search Developer
>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> ---------------------------------------------------------------------
>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>> >>>>
>>>> >>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>>
Re: Propose changing the "dist" layout
Posted by Jan Høydahl <ja...@cominvent.com>.
When we collapse the solr/solr structure I hope we can keep the number of top-level folders in git to a minimum. There are too many already, so adding all contribs don’t help.
I hope the cobtribs will be converted to 1st party packages soon, so perhaps “plugins” or “packages” is a good top level folder name?
Those that are not yet converted can stay in “contribs”?
Can we make solr-exporter a separate git repo? With separate artifacts and separate docker image. Don’t know if that means it must also be a full sub project?
Jan Høydahl
> 11. jun. 2021 kl. 22:46 skrev David Smiley <ds...@apache.org>:
>
>
> We (all?) agree to do away with "contrib" :-).
> I think a folder grouping the modules (that which can plug inside Solr) is useful as there are a number of them -- as such this is a nice organization IMO. There's a bunch of other stuff at the top level and I'd rather not intermix all our modules at this layer.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
>> On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
>> We can have modules, but why do they need to be in an additional folder deep? Why not just have langid next to solrj and core? Contrib to me connotes experimental or unsupported, which these things are decidedly not.
>>
>>> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org> wrote:
>>> The contrib folder is just a folder of modules -- optional plugins for solr-core. IMO we should simply rename "contrib" to "modules". I think the only non-module in there is the prometheus exporter which could move out. Mike, I'm not sure if you have a different notion of what "module" is? I believe most of us would be happy to move away from "contrib" wording, anyway.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>>> I think related to this, I would like to see some "contibs" moved out
>>>> from the contrib folder and into proper modules. Right now the
>>>> definition of contrib seems to be anything that isn't core or solrj,
>>>> but maybe there is room for a backup module that has gcs and s3 and
>>>> hdfs all under it. LangId is already mentioned in our ref guide but we
>>>> pretend like it is always present and don't think of it as a contrib.
>>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>>> the things what they are - plugins and extensions? Then we don't have
>>>> to think as hard about why certain things are showing up in which lib
>>>> folders.
>>>>
>>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>>
>>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org> wrote:
>>>> >
>>>> > I believe we can do a fair amount of re-organization pertaining to Jetty without losing the Jetty configuration that I think is valuable to users who want to tweak something.
>>>> >
>>>> > ~ David Smiley
>>>> > Apache Lucene/Solr Search Developer
>>>> > http://www.linkedin.com/in/davidwsmiley
>>>> >
>>>> >
>>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>> >>
>>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as possible. Perhaps rename "dist" -> "lib"?
>>>> >>
>>>> >> I wish we could get rid of the server (jetty) folder altogether, and move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But that ties into custom boot-class, getting rid of web.xml and building Jetty context in Java code.. I'm willing to help here if others also want to go this direction. This would further hide Jetty as an impl detail and let us organize stuff more freely.
>>>> >>
>>>> >> Jan
>>>> >>
>>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>>> >>
>>>> >> Bumping this conversation up, based on recent communication. I have yet to take action but really any of us can.
>>>> >>
>>>> >> ~ David Smiley
>>>> >> Apache Lucene/Solr Search Developer
>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>> >>
>>>> >>
>>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org> wrote:
>>>> >>>
>>>> >>> I'll proceed on this with lazy consensus. I suspect most of us don't care, unsurprisingly since I doubt anyone has any fondness for the "dist" folder.
>>>> >>>
>>>> >>> ~ David Smiley
>>>> >>> Apache Lucene/Solr Search Developer
>>>> >>> http://www.linkedin.com/in/davidwsmiley
>>>> >>>
>>>> >>>
>>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <er...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Well, Solr has grown “organically” so some things just _are_, like sunrises and plagues ;)
>>>> >>>>
>>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much of this is left over from the war days? Anything that’s lasted through all the transformations Solr has is bound to need cleaning up betimes.
>>>> >>>>
>>>> >>>> How would it relate to splitting Solr off into its own TLP? On the surface, I’d guess the two efforts would be orthogonal, I mention it just in case rearranging the layout would make that task easier or harder...
>>>> >>>>
>>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org> wrote:
>>>> >>>> >
>>>> >>>> > I've been doing a bit of dependency work in one of our contribs, and observing more closely than usual exactly what we produce in the distribution layout (result of gradlew assemble). There are some tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep things as they have been for many years. The distribution layout is awkward, I think. We produce this "dist" folder at the top level that has every JAR this project produces, *even contribs*. But why? I think contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is empty except for a README.txt... IMO it ought to have the JAR in a "lib" subdirectory there mixed with its dependencies (LTR has none but others sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ? I think SolrJ is important enough that it deserves its very own top-level directory "solrj", and like the contribs, with a "lib" alongside it. Maybe Solrj's optional dependencies could be in a lib-optional dir next to it or lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains the solr-core JAR but this is redundant. Furthermore, the server webapp could be configured to add the SolrJ libs so that we don't need to redundantly put any of them in the distribution. There might be some duplicated jars overall, but not many. Logging libs might be explicitly excluded so that they are only in one spot. (Logging in Java is a mess)
>>>> >>>> >
>>>> >>>> > WDYT?
>>>> >>>> >
>>>> >>>> > ~ David Smiley
>>>> >>>> > Apache Lucene/Solr Search Developer
>>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>>> >>>>
>>>> >>>>
>>>> >>>> ---------------------------------------------------------------------
>>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>> >>>>
>>>> >>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
Re: Propose changing the "dist" layout
Posted by David Smiley <ds...@apache.org>.
We (all?) agree to do away with "contrib" :-).
I think a folder grouping the modules (that which can plug inside Solr) is
useful as there are a number of them -- as such this is a nice organization
IMO. There's a bunch of other stuff at the top level and I'd rather not
intermix all our modules at this layer.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jun 11, 2021 at 4:41 PM Mike Drob <md...@mdrob.com> wrote:
> We can have modules, but why do they need to be in an additional folder
> deep? Why not just have langid next to solrj and core? Contrib to me
> connotes experimental or unsupported, which these things are decidedly not.
>
> On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org> wrote:
>
>> The contrib folder is just a folder of modules -- optional plugins for
>> solr-core. IMO we should simply rename "contrib" to "modules". I think
>> the only non-module in there is the prometheus exporter which could move
>> out. Mike, I'm not sure if you have a different notion of what "module"
>> is? I believe most of us would be happy to move away from "contrib"
>> wording, anyway.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>>
>>> I think related to this, I would like to see some "contibs" moved out
>>> from the contrib folder and into proper modules. Right now the
>>> definition of contrib seems to be anything that isn't core or solrj,
>>> but maybe there is room for a backup module that has gcs and s3 and
>>> hdfs all under it. LangId is already mentioned in our ref guide but we
>>> pretend like it is always present and don't think of it as a contrib.
>>> We kind of think of contrib as optional extra stuff, so maybe we call
>>> the things what they are - plugins and extensions? Then we don't have
>>> to think as hard about why certain things are showing up in which lib
>>> folders.
>>>
>>> Also, minor benefit, I would then be able to type c<tab> instead of
>>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>>
>>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org> wrote:
>>> >
>>> > I believe we can do a fair amount of re-organization pertaining to
>>> Jetty without losing the Jetty configuration that I think is valuable to
>>> users who want to tweak something.
>>> >
>>> > ~ David Smiley
>>> > Apache Lucene/Solr Search Developer
>>> > http://www.linkedin.com/in/davidwsmiley
>>> >
>>> >
>>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>>> wrote:
>>> >>
>>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>>> possible. Perhaps rename "dist" -> "lib"?
>>> >>
>>> >> I wish we could get rid of the server (jetty) folder altogether, and
>>> move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/".
>>> But that ties into custom boot-class, getting rid of web.xml and building
>>> Jetty context in Java code.. I'm willing to help here if others also want
>>> to go this direction. This would further hide Jetty as an impl detail and
>>> let us organize stuff more freely.
>>> >>
>>> >> Jan
>>> >>
>>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>> >>
>>> >> Bumping this conversation up, based on recent communication. I have
>>> yet to take action but really any of us can.
>>> >>
>>> >> ~ David Smiley
>>> >> Apache Lucene/Solr Search Developer
>>> >> http://www.linkedin.com/in/davidwsmiley
>>> >>
>>> >>
>>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>>> wrote:
>>> >>>
>>> >>> I'll proceed on this with lazy consensus. I suspect most of us
>>> don't care, unsurprisingly since I doubt anyone has any fondness for the
>>> "dist" folder.
>>> >>>
>>> >>> ~ David Smiley
>>> >>> Apache Lucene/Solr Search Developer
>>> >>> http://www.linkedin.com/in/davidwsmiley
>>> >>>
>>> >>>
>>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>>> erickerickson@gmail.com> wrote:
>>> >>>>
>>> >>>> Well, Solr has grown “organically” so some things just _are_, like
>>> sunrises and plagues ;)
>>> >>>>
>>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how
>>> much of this is left over from the war days? Anything that’s lasted through
>>> all the transformations Solr has is bound to need cleaning up betimes.
>>> >>>>
>>> >>>> How would it relate to splitting Solr off into its own TLP? On the
>>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>>> in case rearranging the layout would make that task easier or harder...
>>> >>>>
>>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
>>> wrote:
>>> >>>> >
>>> >>>> > I've been doing a bit of dependency work in one of our contribs,
>>> and observing more closely than usual exactly what we produce in the
>>> distribution layout (result of gradlew assemble). There are some tricks
>>> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
>>> things as they have been for many years. The distribution layout is
>>> awkward, I think. We produce this "dist" folder at the top level that has
>>> every JAR this project produces, *even contribs*. But why? I think
>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>> subdirectory there mixed with its dependencies (LTR has none but others
>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>> I think SolrJ is important enough that it deserves its very own top-level
>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>> could be configured to add the SolrJ libs so that we don't need to
>>> redundantly put any of them in the distribution. There might be some
>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>> >>>> >
>>> >>>> > WDYT?
>>> >>>> >
>>> >>>> > ~ David Smiley
>>> >>>> > Apache Lucene/Solr Search Developer
>>> >>>> > http://www.linkedin.com/in/davidwsmiley
>>> >>>>
>>> >>>>
>>> >>>>
>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>>>
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
Re: Propose changing the "dist" layout
Posted by Mike Drob <md...@mdrob.com>.
We can have modules, but why do they need to be in an additional folder
deep? Why not just have langid next to solrj and core? Contrib to me
connotes experimental or unsupported, which these things are decidedly not.
On Fri, Jun 11, 2021 at 2:59 PM David Smiley <ds...@apache.org> wrote:
> The contrib folder is just a folder of modules -- optional plugins for
> solr-core. IMO we should simply rename "contrib" to "modules". I think
> the only non-module in there is the prometheus exporter which could move
> out. Mike, I'm not sure if you have a different notion of what "module"
> is? I believe most of us would be happy to move away from "contrib"
> wording, anyway.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
>
>> I think related to this, I would like to see some "contibs" moved out
>> from the contrib folder and into proper modules. Right now the
>> definition of contrib seems to be anything that isn't core or solrj,
>> but maybe there is room for a backup module that has gcs and s3 and
>> hdfs all under it. LangId is already mentioned in our ref guide but we
>> pretend like it is always present and don't think of it as a contrib.
>> We kind of think of contrib as optional extra stuff, so maybe we call
>> the things what they are - plugins and extensions? Then we don't have
>> to think as hard about why certain things are showing up in which lib
>> folders.
>>
>> Also, minor benefit, I would then be able to type c<tab> instead of
>> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>>
>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org> wrote:
>> >
>> > I believe we can do a fair amount of re-organization pertaining to
>> Jetty without losing the Jetty configuration that I think is valuable to
>> users who want to tweak something.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>> >>
>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>> possible. Perhaps rename "dist" -> "lib"?
>> >>
>> >> I wish we could get rid of the server (jetty) folder altogether, and
>> move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/".
>> But that ties into custom boot-class, getting rid of web.xml and building
>> Jetty context in Java code.. I'm willing to help here if others also want
>> to go this direction. This would further hide Jetty as an impl detail and
>> let us organize stuff more freely.
>> >>
>> >> Jan
>> >>
>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>> >>
>> >> Bumping this conversation up, based on recent communication. I have
>> yet to take action but really any of us can.
>> >>
>> >> ~ David Smiley
>> >> Apache Lucene/Solr Search Developer
>> >> http://www.linkedin.com/in/davidwsmiley
>> >>
>> >>
>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
>> wrote:
>> >>>
>> >>> I'll proceed on this with lazy consensus. I suspect most of us don't
>> care, unsurprisingly since I doubt anyone has any fondness for the "dist"
>> folder.
>> >>>
>> >>> ~ David Smiley
>> >>> Apache Lucene/Solr Search Developer
>> >>> http://www.linkedin.com/in/davidwsmiley
>> >>>
>> >>>
>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>> erickerickson@gmail.com> wrote:
>> >>>>
>> >>>> Well, Solr has grown “organically” so some things just _are_, like
>> sunrises and plagues ;)
>> >>>>
>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much
>> of this is left over from the war days? Anything that’s lasted through all
>> the transformations Solr has is bound to need cleaning up betimes.
>> >>>>
>> >>>> How would it relate to splitting Solr off into its own TLP? On the
>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>> in case rearranging the layout would make that task easier or harder...
>> >>>>
>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
>> wrote:
>> >>>> >
>> >>>> > I've been doing a bit of dependency work in one of our contribs,
>> and observing more closely than usual exactly what we produce in the
>> distribution layout (result of gradlew assemble). There are some tricks
>> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
>> things as they have been for many years. The distribution layout is
>> awkward, I think. We produce this "dist" folder at the top level that has
>> every JAR this project produces, *even contribs*. But why? I think
>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>> subdirectory there mixed with its dependencies (LTR has none but others
>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>> I think SolrJ is important enough that it deserves its very own top-level
>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>> could be configured to add the SolrJ libs so that we don't need to
>> redundantly put any of them in the distribution. There might be some
>> duplicated jars overall, but not many. Logging libs might be explicitly
>> excluded so that they are only in one spot. (Logging in Java is a mess)
>> >>>> >
>> >>>> > WDYT?
>> >>>> >
>> >>>> > ~ David Smiley
>> >>>> > Apache Lucene/Solr Search Developer
>> >>>> > http://www.linkedin.com/in/davidwsmiley
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Propose changing the "dist" layout
Posted by David Smiley <ds...@apache.org>.
The contrib folder is just a folder of modules -- optional plugins for
solr-core. IMO we should simply rename "contrib" to "modules". I think
the only non-module in there is the prometheus exporter which could move
out. Mike, I'm not sure if you have a different notion of what "module"
is? I believe most of us would be happy to move away from "contrib"
wording, anyway.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jun 11, 2021 at 3:03 PM Mike Drob <md...@mdrob.com> wrote:
> I think related to this, I would like to see some "contibs" moved out
> from the contrib folder and into proper modules. Right now the
> definition of contrib seems to be anything that isn't core or solrj,
> but maybe there is room for a backup module that has gcs and s3 and
> hdfs all under it. LangId is already mentioned in our ref guide but we
> pretend like it is always present and don't think of it as a contrib.
> We kind of think of contrib as optional extra stuff, so maybe we call
> the things what they are - plugins and extensions? Then we don't have
> to think as hard about why certain things are showing up in which lib
> folders.
>
> Also, minor benefit, I would then be able to type c<tab> instead of
> having to type cor<tab> to disambiguate from con<tab> in my terminal.
>
> On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org> wrote:
> >
> > I believe we can do a fair amount of re-organization pertaining to Jetty
> without losing the Jetty configuration that I think is valuable to users
> who want to tweak something.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >>
> >> +1 to a cleanup here for 9.0. As clean and neat organization as
> possible. Perhaps rename "dist" -> "lib"?
> >>
> >> I wish we could get rid of the server (jetty) folder altogether, and
> move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/".
> But that ties into custom boot-class, getting rid of web.xml and building
> Jetty context in Java code.. I'm willing to help here if others also want
> to go this direction. This would further hide Jetty as an impl detail and
> let us organize stuff more freely.
> >>
> >> Jan
> >>
> >> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
> >>
> >> Bumping this conversation up, based on recent communication. I have
> yet to take action but really any of us can.
> >>
> >> ~ David Smiley
> >> Apache Lucene/Solr Search Developer
> >> http://www.linkedin.com/in/davidwsmiley
> >>
> >>
> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org>
> wrote:
> >>>
> >>> I'll proceed on this with lazy consensus. I suspect most of us don't
> care, unsurprisingly since I doubt anyone has any fondness for the "dist"
> folder.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>>
> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
> erickerickson@gmail.com> wrote:
> >>>>
> >>>> Well, Solr has grown “organically” so some things just _are_, like
> sunrises and plagues ;)
> >>>>
> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much
> of this is left over from the war days? Anything that’s lasted through all
> the transformations Solr has is bound to need cleaning up betimes.
> >>>>
> >>>> How would it relate to splitting Solr off into its own TLP? On the
> surface, I’d guess the two efforts would be orthogonal, I mention it just
> in case rearranging the layout would make that task easier or harder...
> >>>>
> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org>
> wrote:
> >>>> >
> >>>> > I've been doing a bit of dependency work in one of our contribs,
> and observing more closely than usual exactly what we produce in the
> distribution layout (result of gradlew assemble). There are some tricks
> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
> things as they have been for many years. The distribution layout is
> awkward, I think. We produce this "dist" folder at the top level that has
> every JAR this project produces, *even contribs*. But why? I think
> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
> subdirectory there mixed with its dependencies (LTR has none but others
> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
> I think SolrJ is important enough that it deserves its very own top-level
> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
> Solrj's optional dependencies could be in a lib-optional dir next to it or
> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
> the solr-core JAR but this is redundant. Furthermore, the server webapp
> could be configured to add the SolrJ libs so that we don't need to
> redundantly put any of them in the distribution. There might be some
> duplicated jars overall, but not many. Logging libs might be explicitly
> excluded so that they are only in one spot. (Logging in Java is a mess)
> >>>> >
> >>>> > WDYT?
> >>>> >
> >>>> > ~ David Smiley
> >>>> > Apache Lucene/Solr Search Developer
> >>>> > http://www.linkedin.com/in/davidwsmiley
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Propose changing the "dist" layout
Posted by Mike Drob <md...@mdrob.com>.
I think related to this, I would like to see some "contibs" moved out
from the contrib folder and into proper modules. Right now the
definition of contrib seems to be anything that isn't core or solrj,
but maybe there is room for a backup module that has gcs and s3 and
hdfs all under it. LangId is already mentioned in our ref guide but we
pretend like it is always present and don't think of it as a contrib.
We kind of think of contrib as optional extra stuff, so maybe we call
the things what they are - plugins and extensions? Then we don't have
to think as hard about why certain things are showing up in which lib
folders.
Also, minor benefit, I would then be able to type c<tab> instead of
having to type cor<tab> to disambiguate from con<tab> in my terminal.
On Fri, Jun 11, 2021 at 8:09 AM David Smiley <ds...@apache.org> wrote:
>
> I believe we can do a fair amount of re-organization pertaining to Jetty without losing the Jetty configuration that I think is valuable to users who want to tweak something.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> +1 to a cleanup here for 9.0. As clean and neat organization as possible. Perhaps rename "dist" -> "lib"?
>>
>> I wish we could get rid of the server (jetty) folder altogether, and move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But that ties into custom boot-class, getting rid of web.xml and building Jetty context in Java code.. I'm willing to help here if others also want to go this direction. This would further hide Jetty as an impl detail and let us organize stuff more freely.
>>
>> Jan
>>
>> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>>
>> Bumping this conversation up, based on recent communication. I have yet to take action but really any of us can.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org> wrote:
>>>
>>> I'll proceed on this with lazy consensus. I suspect most of us don't care, unsurprisingly since I doubt anyone has any fondness for the "dist" folder.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <er...@gmail.com> wrote:
>>>>
>>>> Well, Solr has grown “organically” so some things just _are_, like sunrises and plagues ;)
>>>>
>>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much of this is left over from the war days? Anything that’s lasted through all the transformations Solr has is bound to need cleaning up betimes.
>>>>
>>>> How would it relate to splitting Solr off into its own TLP? On the surface, I’d guess the two efforts would be orthogonal, I mention it just in case rearranging the layout would make that task easier or harder...
>>>>
>>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org> wrote:
>>>> >
>>>> > I've been doing a bit of dependency work in one of our contribs, and observing more closely than usual exactly what we produce in the distribution layout (result of gradlew assemble). There are some tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep things as they have been for many years. The distribution layout is awkward, I think. We produce this "dist" folder at the top level that has every JAR this project produces, *even contribs*. But why? I think contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is empty except for a README.txt... IMO it ought to have the JAR in a "lib" subdirectory there mixed with its dependencies (LTR has none but others sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ? I think SolrJ is important enough that it deserves its very own top-level directory "solrj", and like the contribs, with a "lib" alongside it. Maybe Solrj's optional dependencies could be in a lib-optional dir next to it or lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains the solr-core JAR but this is redundant. Furthermore, the server webapp could be configured to add the SolrJ libs so that we don't need to redundantly put any of them in the distribution. There might be some duplicated jars overall, but not many. Logging libs might be explicitly excluded so that they are only in one spot. (Logging in Java is a mess)
>>>> >
>>>> > WDYT?
>>>> >
>>>> > ~ David Smiley
>>>> > Apache Lucene/Solr Search Developer
>>>> > http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Propose changing the "dist" layout
Posted by David Smiley <ds...@apache.org>.
I believe we can do a fair amount of re-organization pertaining to Jetty
without losing the Jetty configuration that I think is valuable to users
who want to tweak something.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl <ja...@cominvent.com> wrote:
> +1 to a cleanup here for 9.0. As clean and neat organization as possible.
> Perhaps rename "dist" -> "lib"?
>
> I wish we could get rid of the server (jetty) folder altogether, and move
> everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But
> that ties into custom boot-class, getting rid of web.xml and building Jetty
> context in Java code.. I'm willing to help here if others also want to go
> this direction. This would further hide Jetty as an impl detail and let us
> organize stuff more freely.
>
> Jan
>
> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>
> Bumping this conversation up, based on recent communication. I have yet
> to take action but really any of us can.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <ds...@apache.org> wrote:
>
>> I'll proceed on this with lazy consensus. I suspect most of us don't
>> care, unsurprisingly since I doubt anyone has any fondness for the "dist"
>> folder.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <er...@gmail.com>
>> wrote:
>>
>>> Well, Solr has grown “organically” so some things just _are_, like
>>> sunrises and plagues ;)
>>>
>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much of
>>> this is left over from the war days? Anything that’s lasted through all the
>>> transformations Solr has is bound to need cleaning up betimes.
>>>
>>> How would it relate to splitting Solr off into its own TLP? On the
>>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>>> in case rearranging the layout would make that task easier or harder...
>>>
>>> > On Nov 15, 2020, at 12:18 AM, David Smiley <ds...@apache.org> wrote:
>>> >
>>> > I've been doing a bit of dependency work in one of our contribs, and
>>> observing more closely than usual exactly what we produce in the
>>> distribution layout (result of gradlew assemble). There are some tricks
>>> Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep
>>> things as they have been for many years. The distribution layout is
>>> awkward, I think. We produce this "dist" folder at the top level that has
>>> every JAR this project produces, *even contribs*. But why? I think
>>> contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is
>>> empty except for a README.txt... IMO it ought to have the JAR in a "lib"
>>> subdirectory there mixed with its dependencies (LTR has none but others
>>> sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ?
>>> I think SolrJ is important enough that it deserves its very own top-level
>>> directory "solrj", and like the contribs, with a "lib" alongside it. Maybe
>>> Solrj's optional dependencies could be in a lib-optional dir next to it or
>>> lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains
>>> the solr-core JAR but this is redundant. Furthermore, the server webapp
>>> could be configured to add the SolrJ libs so that we don't need to
>>> redundantly put any of them in the distribution. There might be some
>>> duplicated jars overall, but not many. Logging libs might be explicitly
>>> excluded so that they are only in one spot. (Logging in Java is a mess)
>>> >
>>> > WDYT?
>>> >
>>> > ~ David Smiley
>>> > Apache Lucene/Solr Search Developer
>>> > http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>
Re: Propose changing the "dist" layout
Posted by Jan Høydahl <ja...@cominvent.com>.
+1 to a cleanup here for 9.0. As clean and neat organization as possible. Perhaps rename "dist" -> "lib"?
I wish we could get rid of the server (jetty) folder altogether, and move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But that ties into custom boot-class, getting rid of web.xml and building Jetty context in Java code.. I'm willing to help here if others also want to go this direction. This would further hide Jetty as an impl detail and let us organize stuff more freely.
Jan
> 11. jun. 2021 kl. 13:29 skrev David Smiley <ds...@apache.org>:
>
> Bumping this conversation up, based on recent communication. I have yet to take action but really any of us can.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>
> On Mon, Nov 23, 2020 at 8:48 AM David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
> I'll proceed on this with lazy consensus. I suspect most of us don't care, unsurprisingly since I doubt anyone has any fondness for the "dist" folder.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>
> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <erickerickson@gmail.com <ma...@gmail.com>> wrote:
> Well, Solr has grown “organically” so some things just _are_, like sunrises and plagues ;)
>
> On a serious note, AFAIC rearrange as you see fit. I wonder how much of this is left over from the war days? Anything that’s lasted through all the transformations Solr has is bound to need cleaning up betimes.
>
> How would it relate to splitting Solr off into its own TLP? On the surface, I’d guess the two efforts would be orthogonal, I mention it just in case rearranging the layout would make that task easier or harder...
>
> > On Nov 15, 2020, at 12:18 AM, David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
> >
> > I've been doing a bit of dependency work in one of our contribs, and observing more closely than usual exactly what we produce in the distribution layout (result of gradlew assemble). There are some tricks Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep things as they have been for many years. The distribution layout is awkward, I think. We produce this "dist" folder at the top level that has every JAR this project produces, *even contribs*. But why? I think contribs should keep to themselves. It's ridiculous that /contribs/ltr/ is empty except for a README.txt... IMO it ought to have the JAR in a "lib" subdirectory there mixed with its dependencies (LTR has none but others sure do). Today, each contrib's JAR is in "/dist". And what about SolrJ? I think SolrJ is important enough that it deserves its very own top-level directory "solrj", and like the contribs, with a "lib" alongside it. Maybe Solrj's optional dependencies could be in a lib-optional dir next to it or lib/opt/ (beneath it). Then... we don't need "dist" at all. It contains the solr-core JAR but this is redundant. Furthermore, the server webapp could be configured to add the SolrJ libs so that we don't need to redundantly put any of them in the distribution. There might be some duplicated jars overall, but not many. Logging libs might be explicitly excluded so that they are only in one spot. (Logging in Java is a mess)
> >
> > WDYT?
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <ma...@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org <ma...@lucene.apache.org>
>