You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2023/03/02 14:28:14 UTC
Re: Varying dependency versions? Or sacrifice Hadoop-Auth module
How would its location in the repo address the problem? If we release
something, we support that thing, and that means running tests. How do we
run tests for a module that wants version X when Solr wants version Y?
Docker and high level integration tests could be an answer but that would
be a bunch of work. And I suspect if we look at the tests for it (I
haven't) we'll see its testing details that may be impossible at an
integration level.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Feb 23, 2023 at 2:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Is there a chance the module could be moved up a level in git, next to
> solr-exporter, with own build and a package manifest? We could still ship
> it in tarball as a 1st party package and document the bin/solr package
> install command necessary to install it.
>
> I’d vote for a move in 10.0, with deprecation of the module-version in 9.x.
>
> Jan Høydahl
>
> > 23. feb. 2023 kl. 03:41 skrev David Smiley <ds...@apache.org>:
> >
> > I suppose all code can wait till some future big version unless it's
> some
> > sort of bug fix applicable to the present. It's just a shame that a
> module
> > I perceive to be less used, delaying things.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Sun, Feb 19, 2023 at 11:08 PM Ishan Chattopadhyaya <
> > ichattopadhyaya@gmail.com> wrote:
> >
> >>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
> >> think
> >>> Hadoop-Auth is important enough to continue to thwart this progress on
> >>> SolrCloud. Agreed?
> >>
> >> Agreed, in principle. But, if your suggestion is that we do this in 9x,
> >> then I need more clarification as to why this can't wait until 10x.
> >>
> >> On Mon, Feb 20, 2023 at 9:35 AM Ishan Chattopadhyaya <
> >> ichattopadhyaya@gmail.com> wrote:
> >>
> >>>> We could stop shipping this module with 9.3, say, until the versioning
> >>> issue allows it to return.
> >>>
> >>> Not sure if I follow correctly, but do you mean we stop shipping
> >>> hadoop-auth with Solr starting 9.3?
> >>>
> >>>> On Mon, Feb 20, 2023 at 7:37 AM David Smiley <ds...@apache.org>
> wrote:
> >>>
> >>>> FYI there's the beginning of an important conversation in
> >>>> https://issues.apache.org/jira/browse/SOLR-16116 pertaining to how we
> >>>> deal
> >>>> with modules that want version-X of a dependency but Solr-core would
> >> like
> >>>> to have version-Y. Let's further that discussion here.
> >>>>
> >>>> With a minor ClassLoader change, we could easily support this for a
> user
> >>>> with their own custom module, but these first-party modules are
> another
> >>>> matter. These are compiled, tested, and packaged by our build. It's
> >>>> probably too much work for our build to handle varying versions of a
> >>>> dependency?
> >>>>
> >>>> The problem has blocked progress on a strategic direction of SolrCloud
> >> for
> >>>> a year now, and the module in question is hadoop-auth, and the
> >> dependency
> >>>> is Curator. It's expected to be resolved by the Hadoop project
> someday
> >>>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
> >>>> think
> >>>> Hadoop-Auth is important enough to continue to thwart this progress on
> >>>> SolrCloud. Agreed?
> >>>>
> >>>> We could stop shipping this module with 9.3, say, until the versioning
> >>>> issue allows it to return. One way I prefer is to merely deactivate
> the
> >>>> Gradle module (or say, only tests & JAR publishing) thereby leaving
> >> source
> >>>> in-place, which is nicer for source control history when it returns,
> >> which
> >>>> we expect it to. Of course the other way is outright removal.
> >>>>
> >>>> ~ David Smiley
> >>>> Apache Lucene/Solr Search Developer
> >>>> http://www.linkedin.com/in/davidwsmiley
> >>>>
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: Varying dependency versions? Or sacrifice Hadoop-Auth module
Posted by Jan Høydahl <ja...@cominvent.com>.
It feels more natural for the Hadoop-ecosystem (and commercial vendors) to host a Solr HDFS integration, if they are in need for it for their platforms. It would receive more love if maintained in the hadoop ecosystem rather than in main solr.
The question of removal in 9.x depends on whether it poses a security risk on Solr, which it historically has due to the old dependencies. If no pressing security arguments, I feel we could still justify removal if the module gets a new home outside of the Solr project, and it is technically possible to use it with future 9.x releases, even if we keep upgrading our dependencies.
However, it is not "our job" to find a new home for it, strictly speaking. So announcing its deprecation in 9.2 but keeping it for at least a few more releases would be the most community-friendly move.
Jan
> 2. mar. 2023 kl. 15:58 skrev David Smiley <ds...@apache.org>:
>
> Maybe; same for the "sandbox" but at least externally benefits from the
> *possibility* of a user who wants to maintain it from doing so. That is
> darned close to abandonment
> unless such a user steps forward. But maybe that's how it has to be if we
> want to unblock Curator's use in Solr.
>
> The bigger question on my mind is, can this change happen in the 9x
> codeline. With fewer users/maintainers than in the past, I would prefer we
> not rigidly stick to continuing to include a feature for all of 9x if we
> believe it's not getting much use.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Mar 2, 2023 at 9:38 AM Eric Pugh <ep...@opensourceconnections.com>
> wrote:
>
>> Would it be better to take the DIH route and let it be its own project
>> elsewhere? With its own upgrade schedule for Solr versions?
>>
>>> On Mar 2, 2023, at 9:28 AM, David Smiley <ds...@apache.org> wrote:
>>>
>>> How would its location in the repo address the problem? If we release
>>> something, we support that thing, and that means running tests. How do
>> we
>>> run tests for a module that wants version X when Solr wants version Y?
>>> Docker and high level integration tests could be an answer but that would
>>> be a bunch of work. And I suspect if we look at the tests for it (I
>>> haven't) we'll see its testing details that may be impossible at an
>>> integration level.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Thu, Feb 23, 2023 at 2:44 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>>
>>>> Is there a chance the module could be moved up a level in git, next to
>>>> solr-exporter, with own build and a package manifest? We could still
>> ship
>>>> it in tarball as a 1st party package and document the bin/solr package
>>>> install command necessary to install it.
>>>>
>>>> I’d vote for a move in 10.0, with deprecation of the module-version in
>> 9.x.
>>>>
>>>> Jan Høydahl
>>>>
>>>>> 23. feb. 2023 kl. 03:41 skrev David Smiley <ds...@apache.org>:
>>>>>
>>>>> I suppose all code can wait till some future big version unless it's
>>>> some
>>>>> sort of bug fix applicable to the present. It's just a shame that a
>>>> module
>>>>> I perceive to be less used, delaying things.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>
>>>>>
>>>>> On Sun, Feb 19, 2023 at 11:08 PM Ishan Chattopadhyaya <
>>>>> ichattopadhyaya@gmail.com> wrote:
>>>>>
>>>>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
>>>>>> think
>>>>>>> Hadoop-Auth is important enough to continue to thwart this progress
>> on
>>>>>>> SolrCloud. Agreed?
>>>>>>
>>>>>> Agreed, in principle. But, if your suggestion is that we do this in
>> 9x,
>>>>>> then I need more clarification as to why this can't wait until 10x.
>>>>>>
>>>>>> On Mon, Feb 20, 2023 at 9:35 AM Ishan Chattopadhyaya <
>>>>>> ichattopadhyaya@gmail.com> wrote:
>>>>>>
>>>>>>>> We could stop shipping this module with 9.3, say, until the
>> versioning
>>>>>>> issue allows it to return.
>>>>>>>
>>>>>>> Not sure if I follow correctly, but do you mean we stop shipping
>>>>>>> hadoop-auth with Solr starting 9.3?
>>>>>>>
>>>>>>>> On Mon, Feb 20, 2023 at 7:37 AM David Smiley <ds...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> FYI there's the beginning of an important conversation in
>>>>>>>> https://issues.apache.org/jira/browse/SOLR-16116 pertaining to how
>> we
>>>>>>>> deal
>>>>>>>> with modules that want version-X of a dependency but Solr-core would
>>>>>> like
>>>>>>>> to have version-Y. Let's further that discussion here.
>>>>>>>>
>>>>>>>> With a minor ClassLoader change, we could easily support this for a
>>>> user
>>>>>>>> with their own custom module, but these first-party modules are
>>>> another
>>>>>>>> matter. These are compiled, tested, and packaged by our build.
>> It's
>>>>>>>> probably too much work for our build to handle varying versions of a
>>>>>>>> dependency?
>>>>>>>>
>>>>>>>> The problem has blocked progress on a strategic direction of
>> SolrCloud
>>>>>> for
>>>>>>>> a year now, and the module in question is hadoop-auth, and the
>>>>>> dependency
>>>>>>>> is Curator. It's expected to be resolved by the Hadoop project
>>>> someday
>>>>>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I
>> don't
>>>>>>>> think
>>>>>>>> Hadoop-Auth is important enough to continue to thwart this progress
>> on
>>>>>>>> SolrCloud. Agreed?
>>>>>>>>
>>>>>>>> We could stop shipping this module with 9.3, say, until the
>> versioning
>>>>>>>> issue allows it to return. One way I prefer is to merely deactivate
>>>> the
>>>>>>>> Gradle module (or say, only tests & JAR publishing) thereby leaving
>>>>>> source
>>>>>>>> in-place, which is nicer for source control history when it returns,
>>>>>> which
>>>>>>>> we expect it to. Of course the other way is outright removal.
>>>>>>>>
>>>>>>>> ~ David Smiley
>>>>>>>> Apache Lucene/Solr Search Developer
>>>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>>
>>
>> _______________________
>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
>> http://www.opensourceconnections.com <
>> http://www.opensourceconnections.com/> | My Free/Busy <
>> http://tinyurl.com/eric-cal>
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
>> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>>
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless of
>> whether attachments are marked as such.
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Varying dependency versions? Or sacrifice Hadoop-Auth module
Posted by David Smiley <ds...@apache.org>.
Maybe; same for the "sandbox" but at least externally benefits from the
*possibility* of a user who wants to maintain it from doing so. That is
darned close to abandonment
unless such a user steps forward. But maybe that's how it has to be if we
want to unblock Curator's use in Solr.
The bigger question on my mind is, can this change happen in the 9x
codeline. With fewer users/maintainers than in the past, I would prefer we
not rigidly stick to continuing to include a feature for all of 9x if we
believe it's not getting much use.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Mar 2, 2023 at 9:38 AM Eric Pugh <ep...@opensourceconnections.com>
wrote:
> Would it be better to take the DIH route and let it be its own project
> elsewhere? With its own upgrade schedule for Solr versions?
>
> > On Mar 2, 2023, at 9:28 AM, David Smiley <ds...@apache.org> wrote:
> >
> > How would its location in the repo address the problem? If we release
> > something, we support that thing, and that means running tests. How do
> we
> > run tests for a module that wants version X when Solr wants version Y?
> > Docker and high level integration tests could be an answer but that would
> > be a bunch of work. And I suspect if we look at the tests for it (I
> > haven't) we'll see its testing details that may be impossible at an
> > integration level.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Thu, Feb 23, 2023 at 2:44 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >
> >> Is there a chance the module could be moved up a level in git, next to
> >> solr-exporter, with own build and a package manifest? We could still
> ship
> >> it in tarball as a 1st party package and document the bin/solr package
> >> install command necessary to install it.
> >>
> >> I’d vote for a move in 10.0, with deprecation of the module-version in
> 9.x.
> >>
> >> Jan Høydahl
> >>
> >>> 23. feb. 2023 kl. 03:41 skrev David Smiley <ds...@apache.org>:
> >>>
> >>> I suppose all code can wait till some future big version unless it's
> >> some
> >>> sort of bug fix applicable to the present. It's just a shame that a
> >> module
> >>> I perceive to be less used, delaying things.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>>
> >>> On Sun, Feb 19, 2023 at 11:08 PM Ishan Chattopadhyaya <
> >>> ichattopadhyaya@gmail.com> wrote:
> >>>
> >>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
> >>>> think
> >>>>> Hadoop-Auth is important enough to continue to thwart this progress
> on
> >>>>> SolrCloud. Agreed?
> >>>>
> >>>> Agreed, in principle. But, if your suggestion is that we do this in
> 9x,
> >>>> then I need more clarification as to why this can't wait until 10x.
> >>>>
> >>>> On Mon, Feb 20, 2023 at 9:35 AM Ishan Chattopadhyaya <
> >>>> ichattopadhyaya@gmail.com> wrote:
> >>>>
> >>>>>> We could stop shipping this module with 9.3, say, until the
> versioning
> >>>>> issue allows it to return.
> >>>>>
> >>>>> Not sure if I follow correctly, but do you mean we stop shipping
> >>>>> hadoop-auth with Solr starting 9.3?
> >>>>>
> >>>>>> On Mon, Feb 20, 2023 at 7:37 AM David Smiley <ds...@apache.org>
> >> wrote:
> >>>>>
> >>>>>> FYI there's the beginning of an important conversation in
> >>>>>> https://issues.apache.org/jira/browse/SOLR-16116 pertaining to how
> we
> >>>>>> deal
> >>>>>> with modules that want version-X of a dependency but Solr-core would
> >>>> like
> >>>>>> to have version-Y. Let's further that discussion here.
> >>>>>>
> >>>>>> With a minor ClassLoader change, we could easily support this for a
> >> user
> >>>>>> with their own custom module, but these first-party modules are
> >> another
> >>>>>> matter. These are compiled, tested, and packaged by our build.
> It's
> >>>>>> probably too much work for our build to handle varying versions of a
> >>>>>> dependency?
> >>>>>>
> >>>>>> The problem has blocked progress on a strategic direction of
> SolrCloud
> >>>> for
> >>>>>> a year now, and the module in question is hadoop-auth, and the
> >>>> dependency
> >>>>>> is Curator. It's expected to be resolved by the Hadoop project
> >> someday
> >>>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I
> don't
> >>>>>> think
> >>>>>> Hadoop-Auth is important enough to continue to thwart this progress
> on
> >>>>>> SolrCloud. Agreed?
> >>>>>>
> >>>>>> We could stop shipping this module with 9.3, say, until the
> versioning
> >>>>>> issue allows it to return. One way I prefer is to merely deactivate
> >> the
> >>>>>> Gradle module (or say, only tests & JAR publishing) thereby leaving
> >>>> source
> >>>>>> in-place, which is nicer for source control history when it returns,
> >>>> which
> >>>>>> we expect it to. Of course the other way is outright removal.
> >>>>>>
> >>>>>> ~ David Smiley
> >>>>>> Apache Lucene/Solr Search Developer
> >>>>>> http://www.linkedin.com/in/davidwsmiley
> >>>>>>
> >>>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com <
> http://www.opensourceconnections.com/> | My Free/Busy <
> http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>
Re: Varying dependency versions? Or sacrifice Hadoop-Auth module
Posted by Eric Pugh <ep...@opensourceconnections.com>.
Would it be better to take the DIH route and let it be its own project elsewhere? With its own upgrade schedule for Solr versions?
> On Mar 2, 2023, at 9:28 AM, David Smiley <ds...@apache.org> wrote:
>
> How would its location in the repo address the problem? If we release
> something, we support that thing, and that means running tests. How do we
> run tests for a module that wants version X when Solr wants version Y?
> Docker and high level integration tests could be an answer but that would
> be a bunch of work. And I suspect if we look at the tests for it (I
> haven't) we'll see its testing details that may be impossible at an
> integration level.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Feb 23, 2023 at 2:44 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Is there a chance the module could be moved up a level in git, next to
>> solr-exporter, with own build and a package manifest? We could still ship
>> it in tarball as a 1st party package and document the bin/solr package
>> install command necessary to install it.
>>
>> I’d vote for a move in 10.0, with deprecation of the module-version in 9.x.
>>
>> Jan Høydahl
>>
>>> 23. feb. 2023 kl. 03:41 skrev David Smiley <ds...@apache.org>:
>>>
>>> I suppose all code can wait till some future big version unless it's
>> some
>>> sort of bug fix applicable to the present. It's just a shame that a
>> module
>>> I perceive to be less used, delaying things.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sun, Feb 19, 2023 at 11:08 PM Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>>
>>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
>>>> think
>>>>> Hadoop-Auth is important enough to continue to thwart this progress on
>>>>> SolrCloud. Agreed?
>>>>
>>>> Agreed, in principle. But, if your suggestion is that we do this in 9x,
>>>> then I need more clarification as to why this can't wait until 10x.
>>>>
>>>> On Mon, Feb 20, 2023 at 9:35 AM Ishan Chattopadhyaya <
>>>> ichattopadhyaya@gmail.com> wrote:
>>>>
>>>>>> We could stop shipping this module with 9.3, say, until the versioning
>>>>> issue allows it to return.
>>>>>
>>>>> Not sure if I follow correctly, but do you mean we stop shipping
>>>>> hadoop-auth with Solr starting 9.3?
>>>>>
>>>>>> On Mon, Feb 20, 2023 at 7:37 AM David Smiley <ds...@apache.org>
>> wrote:
>>>>>
>>>>>> FYI there's the beginning of an important conversation in
>>>>>> https://issues.apache.org/jira/browse/SOLR-16116 pertaining to how we
>>>>>> deal
>>>>>> with modules that want version-X of a dependency but Solr-core would
>>>> like
>>>>>> to have version-Y. Let's further that discussion here.
>>>>>>
>>>>>> With a minor ClassLoader change, we could easily support this for a
>> user
>>>>>> with their own custom module, but these first-party modules are
>> another
>>>>>> matter. These are compiled, tested, and packaged by our build. It's
>>>>>> probably too much work for our build to handle varying versions of a
>>>>>> dependency?
>>>>>>
>>>>>> The problem has blocked progress on a strategic direction of SolrCloud
>>>> for
>>>>>> a year now, and the module in question is hadoop-auth, and the
>>>> dependency
>>>>>> is Curator. It's expected to be resolved by the Hadoop project
>> someday
>>>>>> (e.g. by shading) it but it hasn't happened yet. Personally, I don't
>>>>>> think
>>>>>> Hadoop-Auth is important enough to continue to thwart this progress on
>>>>>> SolrCloud. Agreed?
>>>>>>
>>>>>> We could stop shipping this module with 9.3, say, until the versioning
>>>>>> issue allows it to return. One way I prefer is to merely deactivate
>> the
>>>>>> Gradle module (or say, only tests & JAR publishing) thereby leaving
>>>> source
>>>>>> in-place, which is nicer for source control history when it returns,
>>>> which
>>>>>> we expect it to. Of course the other way is outright removal.
>>>>>>
>>>>>> ~ David Smiley
>>>>>> Apache Lucene/Solr Search Developer
>>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.