You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@solr.apache.org by Dawid Weiss <da...@gmail.com> on 2021/05/01 06:15:02 UTC

Re: Proposal to pin the Lucene snapshot version on main

> Other than literally adding the git submodule, would we do anything else to modify the gradle build so that or do we (and Jenkinsfile) have to manually do a step to install Lucene before proceeding?

Technically you add a submodule and then make a composite build from
Solr side. I can provide a PR that does it if you wish... This is
simple. What submodules don't solve is releases - if you're on a
particular unreleased Lucene version then releasing Solr would still
mean you need some kind of "public" pinned Lucene release for the
Maven world. If you distribute the entire thing as binaries you can
just publish a snapshot of Lucene code, of course.

> I think adding git submodule means that we have to add back in all the build code for it.

No. The submodule is just a reference to a particular repository (and
commit) - the build code and remains nearly identical as it was with
Lucene. Composite builds in gradle are quite nice in that they're
(almost) transparent compared to dependency references.

> At which point, I'd rather just copy and commit the
> code they have so we don't have to learn another git system.

It's not really a different git system - it's a mechanism of
interacting with multiple repositories at once. Yes, it does add some
additional complexity but I think sooner or later you'll have to
interact with it anyway - many people favor monorepos these days and
this is a common way to assemble them from fragmented
sub-repositories.

> I've
> heard submodules don't play nice with Jenkins, but don't have any
> direct experience with them.

Some tools may require an additional flag to initialize and clone
sub-modules on the initial pull. Or manually.

git submodule init
git submodule update --recursive.

To summarize - I am not pushing for submodules, I do think they're a
viable alternative but pinned Maven references are simpler. The
problem with maven references is that we'd need a long(er) term maven
repository where such pinned references would live (without being
wiped out). Perhaps infra can help here and it'd solve the problem.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org

Re: Proposal to pin the Lucene snapshot version on main

Posted by David Smiley <ds...@apache.org>.

No, these are not "releases".

> If the general public is being instructed to download a package, then
that package has been released

We will not be instructing the general public to download this stuff.  It's
internal, and it's temporary before we get to an actual 9.0 release.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Fri, May 7, 2021 at 9:35 AM Mike Drob <md...@mdrob.com> wrote:

> Is this allowed by ASF policy? Are the Lucene “releases” that we are
> making something that needs to be voted on and approved by the Lucene PMC?
> https://apache.org/legal/release-policy.html#what
>
> > In our case, that means any publication outside the group of people on
> the product dev list.
>
> Solr dev list is not the Lucene dev list.
>
> On Fri, May 7, 2021 at 12:55 AM Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only
>> manually triggered.
>> It builds the maven artifacts into a local file system repo. As post
>> build job we publish the output folder on nighties repo like the ref guide.
>>
>> The version number is passed as -DreleaseVersion system property (e.g.
>> populated by Jenkins from the build number), like we do on artifacts builds
>> on jenkins anyways. It's no snapshot anymore, but like an half official
>> build with some custom version internal to our process.
>>
>> If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait
>> for it to finish, and finally copy-paste the version number from Jenkins
>> build description into gradle dependency version file.
>>
>> Uwe
>>
>> Uwe
>>
>>
>> Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett <
>> casstargett@gmail.com>:
>>>
>>> It could be a manual process if that’s what you think is easiest and it
>>> could still be on nightlies. It’s just a curl command to push something up:
>>> https://nightlies.apache.org/authoring.html. If it is a manual process,
>>> I think it would be better for it to be hosted in a place any of us could
>>> access, in case you’re busy/something happens to you and we need to pin a
>>> different snapshot.
>>>
>>> I’m not sure I see a reason why Solr couldn’t have our own Jenkins job
>>> that built Lucene once a week (or on demand, or whatever cadence that
>>> works) and pushes the snapshot to nightlies - we don’t have to use one of
>>> Lucene’s jobs, do we?
>>>
>>> I feel like we could also cascade a new Lucene snapshot build into a
>>> “Lucene validation” Solr build that verifies everything is OK with a new
>>> snapshot before updating the hosted snapshot on nightlies (although, I
>>> don’t actually know how to do that, it just seems like something that could
>>> be done).
>>>
>>> Cassandra
>>> On May 6, 2021, 10:13 AM -0500, David Smiley <ds...@apache.org>,
>>> wrote:
>>>
>>> I asked on the Lucene dev list about possible use of the Nightlies
>>> server.  We don't want to pollute nightlies on a regular basis (I think);
>>> this would be an on-demand thing.  As such, I'm not sure a CI server is the
>>> best way to approach this vs a manual script publishing to an ASF personal
>>> Home space.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Mon, May 3, 2021 at 10:55 AM Cassandra Targett <ca...@gmail.com>
>>> wrote:
>>>
>>>> I’m sort of surprised no one has mentioned the
>>>> https://nightlies.apache.org/ server, which could be used for this
>>>> purpose. Jenkins can push to it, and nothing gets deleted until we decide
>>>> to delete it (or overwrite it). Solr Operator builds go there as do drafts
>>>> of the Ref Guide pre-publication (in different directories now, but we’ll
>>>> fix that eventually).
>>>>
>>>> Cassandra
>>>> On May 3, 2021, 9:15 AM -0500, David Smiley <ds...@apache.org>,
>>>> wrote:
>>>>
>>>> RE "an external system like Maven" -- we're merely talking about adding
>>>> another JAR repo to the list of repos we already have.  Heck, for this
>>>> limited purpose, we could even use http://home.apache.org/~dsmiley/
>>>>  Note that the Lucene project already uses home dirs of some users for
>>>> benchmark data.  If we were talking about adding a Maven *build* then I
>>>> would totally appreciate your concern.
>>>>
>>>> I volunteer to use my space for this purpose.
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley
>>>>
>>>>
>>>> On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya <
>>>> ichattopadhyaya@gmail.com> wrote:
>>>>
>>>>> > Let’s not complicate things.
>>>>> By bringing in external systems like Maven, I think we're complicating
>>>>> things even though a straight forward way (git submodules) exists.
>>>>>
>>>>> On Sat, May 1, 2021 at 5:00 PM Jan Høydahl <
>>>>> janhoy-apache@cominvent.com> wrote:
>>>>>
>>>>>> Sub modules is for organizing internal repos, not for pulling in
>>>>>> external deps. Let’s not complicate things. And once we switch main to 10.x
>>>>>> we’d need to use pure jar dependencies in branch_9x to depend upon actually
>>>>>> released and voted Lucene binaries.
>>>>>>
>>>>>> We need some tooling to smoothly work with bleeding edge Lucene
>>>>>> including local snapshot builds with not yet pushed changes to Lucene. I’d
>>>>>> rather put in some work to establish such tooling or procedures.
>>>>>>
>>>>>> Jan Høydahl
>>>>>>
>>>>>> 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya <
>>>>>> ichattopadhyaya@gmail.com>:
>>>>>>
>>>>>> 
>>>>>> > What submodules don't solve is releases - if you're
>>>>>> > on a
>>>>>> particular unreleased Lucene version then releasing
>>>>>> > Solr would still mean you need some kind of
>>>>>> > "public" pinned Lucene release for the
>>>>>> > Mavenworld.
>>>>>>
>>>>>>
>>>>>> Can we then update the submodule to point to the release tag or sha
>>>>>> that Lucene got released from?
>>>>>>
>>>>>> On Sat, 1 May, 2021, 11:45 am Dawid Weiss, <da...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> > Other than literally adding the git submodule, would we do
>>>>>>> anything else to modify the gradle build so that or do we (and Jenkinsfile)
>>>>>>> have to manually do a step to install Lucene before proceeding?
>>>>>>>
>>>>>>> Technically you add a submodule and then make a composite build from
>>>>>>> Solr side. I can provide a PR that does it if you wish... This is
>>>>>>> simple. What submodules don't solve is releases - if you're on a
>>>>>>> particular unreleased Lucene version then releasing Solr would still
>>>>>>> mean you need some kind of "public" pinned Lucene release for the
>>>>>>> Maven world. If you distribute the entire thing as binaries you can
>>>>>>> just publish a snapshot of Lucene code, of course.
>>>>>>>
>>>>>>> > I think adding git submodule means that we have to add back in all
>>>>>>> the build code for it.
>>>>>>>
>>>>>>> No. The submodule is just a reference to a particular repository (and
>>>>>>> commit) - the build code and remains nearly identical as it was with
>>>>>>> Lucene. Composite builds in gradle are quite nice in that they're
>>>>>>> (almost) transparent compared to dependency references.
>>>>>>>
>>>>>>> > At which point, I'd rather just copy and commit the
>>>>>>> > code they have so we don't have to learn another git system.
>>>>>>>
>>>>>>> It's not really a different git system - it's a mechanism of
>>>>>>> interacting with multiple repositories at once. Yes, it does add some
>>>>>>> additional complexity but I think sooner or later you'll have to
>>>>>>> interact with it anyway - many people favor monorepos these days and
>>>>>>> this is a common way to assemble them from fragmented
>>>>>>> sub-repositories.
>>>>>>>
>>>>>>> > I've
>>>>>>> > heard submodules don't play nice with Jenkins, but don't have any
>>>>>>> > direct experience with them.
>>>>>>>
>>>>>>> Some tools may require an additional flag to initialize and clone
>>>>>>> sub-modules on the initial pull. Or manually.
>>>>>>>
>>>>>>> git submodule init
>>>>>>> git submodule update --recursive.
>>>>>>>
>>>>>>> To summarize - I am not pushing for submodules, I do think they're a
>>>>>>> viable alternative but pinned Maven references are simpler. The
>>>>>>> problem with maven references is that we'd need a long(er) term maven
>>>>>>> repository where such pinned references would live (without being
>>>>>>> wiped out). Perhaps infra can help here and it'd solve the problem.
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>>
>>>>>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, 28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+28357+Bremen?entry=gmail&source=g>
>> https://www.thetaphi.de
>>
>

Re: Proposal to pin the Lucene snapshot version on main

Posted by Mike Drob <md...@mdrob.com>.

Is this allowed by ASF policy? Are the Lucene “releases” that we are making
something that needs to be voted on and approved by the Lucene PMC?
https://apache.org/legal/release-policy.html#what

> In our case, that means any publication outside the group of people on
the product dev list.

Solr dev list is not the Lucene dev list.

On Fri, May 7, 2021 at 12:55 AM Uwe Schindler <uw...@thetaphi.de> wrote:

> Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only
> manually triggered.
> It builds the maven artifacts into a local file system repo. As post build
> job we publish the output folder on nighties repo like the ref guide.
>
> The version number is passed as -DreleaseVersion system property (e.g.
> populated by Jenkins from the build number), like we do on artifacts builds
> on jenkins anyways. It's no snapshot anymore, but like an half official
> build with some custom version internal to our process.
>
> If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait
> for it to finish, and finally copy-paste the version number from Jenkins
> build description into gradle dependency version file.
>
> Uwe
>
> Uwe
>
>
> Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett <
> casstargett@gmail.com>:
>>
>> It could be a manual process if that’s what you think is easiest and it
>> could still be on nightlies. It’s just a curl command to push something up:
>> https://nightlies.apache.org/authoring.html. If it is a manual process,
>> I think it would be better for it to be hosted in a place any of us could
>> access, in case you’re busy/something happens to you and we need to pin a
>> different snapshot.
>>
>> I’m not sure I see a reason why Solr couldn’t have our own Jenkins job
>> that built Lucene once a week (or on demand, or whatever cadence that
>> works) and pushes the snapshot to nightlies - we don’t have to use one of
>> Lucene’s jobs, do we?
>>
>> I feel like we could also cascade a new Lucene snapshot build into a
>> “Lucene validation” Solr build that verifies everything is OK with a new
>> snapshot before updating the hosted snapshot on nightlies (although, I
>> don’t actually know how to do that, it just seems like something that could
>> be done).
>>
>> Cassandra
>> On May 6, 2021, 10:13 AM -0500, David Smiley <ds...@apache.org>, wrote:
>>
>> I asked on the Lucene dev list about possible use of the Nightlies
>> server.  We don't want to pollute nightlies on a regular basis (I think);
>> this would be an on-demand thing.  As such, I'm not sure a CI server is the
>> best way to approach this vs a manual script publishing to an ASF personal
>> Home space.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Mon, May 3, 2021 at 10:55 AM Cassandra Targett <ca...@gmail.com>
>> wrote:
>>
>>> I’m sort of surprised no one has mentioned the
>>> https://nightlies.apache.org/ server, which could be used for this
>>> purpose. Jenkins can push to it, and nothing gets deleted until we decide
>>> to delete it (or overwrite it). Solr Operator builds go there as do drafts
>>> of the Ref Guide pre-publication (in different directories now, but we’ll
>>> fix that eventually).
>>>
>>> Cassandra
>>> On May 3, 2021, 9:15 AM -0500, David Smiley <ds...@apache.org>, wrote:
>>>
>>> RE "an external system like Maven" -- we're merely talking about adding
>>> another JAR repo to the list of repos we already have.  Heck, for this
>>> limited purpose, we could even use http://home.apache.org/~dsmiley/
>>>  Note that the Lucene project already uses home dirs of some users for
>>> benchmark data.  If we were talking about adding a Maven *build* then I
>>> would totally appreciate your concern.
>>>
>>> I volunteer to use my space for this purpose.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>>
>>>> > Let’s not complicate things.
>>>> By bringing in external systems like Maven, I think we're complicating
>>>> things even though a straight forward way (git submodules) exists.
>>>>
>>>> On Sat, May 1, 2021 at 5:00 PM Jan Høydahl <ja...@cominvent.com>
>>>> wrote:
>>>>
>>>>> Sub modules is for organizing internal repos, not for pulling in
>>>>> external deps. Let’s not complicate things. And once we switch main to 10.x
>>>>> we’d need to use pure jar dependencies in branch_9x to depend upon actually
>>>>> released and voted Lucene binaries.
>>>>>
>>>>> We need some tooling to smoothly work with bleeding edge Lucene
>>>>> including local snapshot builds with not yet pushed changes to Lucene. I’d
>>>>> rather put in some work to establish such tooling or procedures.
>>>>>
>>>>> Jan Høydahl
>>>>>
>>>>> 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya <
>>>>> ichattopadhyaya@gmail.com>:
>>>>>
>>>>> 
>>>>> > What submodules don't solve is releases - if you're
>>>>> > on a
>>>>> particular unreleased Lucene version then releasing
>>>>> > Solr would still mean you need some kind of
>>>>> > "public" pinned Lucene release for the
>>>>> > Mavenworld.
>>>>>
>>>>>
>>>>> Can we then update the submodule to point to the release tag or sha
>>>>> that Lucene got released from?
>>>>>
>>>>> On Sat, 1 May, 2021, 11:45 am Dawid Weiss, <da...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> > Other than literally adding the git submodule, would we do anything
>>>>>> else to modify the gradle build so that or do we (and Jenkinsfile) have to
>>>>>> manually do a step to install Lucene before proceeding?
>>>>>>
>>>>>> Technically you add a submodule and then make a composite build from
>>>>>> Solr side. I can provide a PR that does it if you wish... This is
>>>>>> simple. What submodules don't solve is releases - if you're on a
>>>>>> particular unreleased Lucene version then releasing Solr would still
>>>>>> mean you need some kind of "public" pinned Lucene release for the
>>>>>> Maven world. If you distribute the entire thing as binaries you can
>>>>>> just publish a snapshot of Lucene code, of course.
>>>>>>
>>>>>> > I think adding git submodule means that we have to add back in all
>>>>>> the build code for it.
>>>>>>
>>>>>> No. The submodule is just a reference to a particular repository (and
>>>>>> commit) - the build code and remains nearly identical as it was with
>>>>>> Lucene. Composite builds in gradle are quite nice in that they're
>>>>>> (almost) transparent compared to dependency references.
>>>>>>
>>>>>> > At which point, I'd rather just copy and commit the
>>>>>> > code they have so we don't have to learn another git system.
>>>>>>
>>>>>> It's not really a different git system - it's a mechanism of
>>>>>> interacting with multiple repositories at once. Yes, it does add some
>>>>>> additional complexity but I think sooner or later you'll have to
>>>>>> interact with it anyway - many people favor monorepos these days and
>>>>>> this is a common way to assemble them from fragmented
>>>>>> sub-repositories.
>>>>>>
>>>>>> > I've
>>>>>> > heard submodules don't play nice with Jenkins, but don't have any
>>>>>> > direct experience with them.
>>>>>>
>>>>>> Some tools may require an additional flag to initialize and clone
>>>>>> sub-modules on the initial pull. Or manually.
>>>>>>
>>>>>> git submodule init
>>>>>> git submodule update --recursive.
>>>>>>
>>>>>> To summarize - I am not pushing for submodules, I do think they're a
>>>>>> viable alternative but pinned Maven references are simpler. The
>>>>>> problem with maven references is that we'd need a long(er) term maven
>>>>>> repository where such pinned references would live (without being
>>>>>> wiped out). Perhaps infra can help here and it'd solve the problem.
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>
>>>>>>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> <https://www.google.com/maps/search/Achterdiek+19,+28357+Bremen?entry=gmail&source=g>
> https://www.thetaphi.de
>