You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Modassar Ather <mo...@gmail.com> on 2020/11/03 11:39:11 UTC
Solr migration related issues.
Hi,
I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
upgrade I have the first task to install and configure the solr with the
core and collection. The solr is installed in SolrCloud mode.
In Solr 6.5.1 I was using the following key values in core.properties file.
The configuration files were uploaded to zookeeper using the upconfig
command.
The core and collection was automatically created with the setting in
core.properties files and the configSet uploaded in zookeeper and it used
to display on the Solr 6.5.1 dashboard.
numShards=12
name=mycore
collection=mycore
configSet=mycore
With the latest Solr 8.6.3 the same approach is not working. As per my
understanding the core is identified using the location of core.properties
which is under *<SOLR_HOME>/mycore/core.properties.*
Can you please help me with the following?
- Is there any property I am missing to load the core and collection as
it used to be in Solr 6.5.1 with the help of core.properties and config set
on zookeeper?
- The name of the core and collection should be configurable and not the
dynamically generated names. How can I control that in the latest Solr?
- Is the core and collection API the only way to create core and
collection as I see that the core is also not getting listed even if the
core.properties file is present?
Please note that I will be doing a full indexing once the setup is done.
Kindly help me with your suggestions.
Best,
Modassar
Re: Solr migration related issues.
Posted by Modassar Ather <mo...@gmail.com>.
Thanks Erick.
I am going through the recommendations given in this email chain.
Best,
Modassar
On Thu, Nov 5, 2020 at 7:17 PM Erick Erickson <er...@gmail.com>
wrote:
> Oh dear.
>
> You made me look at the reference guide. Ouch.
>
> We have this nice page “Defining core.properties” that talks about
> defining core.properties. Unfortunately it has _no_ warning about the fact
> that trying to use this in SolrCloud is a really bad idea. As in “don’t do
> it”. To make matters worse, as you have found out painfully, it kinda
> worked in cloud mode in times past.
>
> Then the collections API doc says you can add property.name=value, with
> no mention that the name here should NOT be any of the properties necessary
> for SolrCloud to operate.
>
> The problem here is that adding property.name=value would set that
> property to the _same_ value in all cores. Naming all the replicas for a
> collection to the same thing is not supported officially, if it works in
> older Solrs that was by chance not design. And there’s no special provision
> for just using that property as a prefix. That’s really designed for custom
> properties. And, by the way, “name” is really kind of a no-op, the thing
> displayed in the drop-down is taken from Zookeeper’s node_name. Please
> don’t try to name that.
>
> I very strongly recommend that you stop trying to do this. Whatever you
> are doing that requires a specific name, I’d change _that_ process to use
> the names assigned by Solr. If it’s just for aesthetics, there’s really no
> good way to change what’s in the drop-down.
>
> Best,
> Erick
>
> > On Nov 5, 2020, at 5:25 AM, Modassar Ather <mo...@gmail.com>
> wrote:
> >
> > Hi Shawn,
> >
> > I understand that we do not need to modify the core.properties and use
> the
> > APIs to create core and collection and that is what I am doing now.
> > This question of naming the core as per the choice comes from our older
> > setup where we have 12 shards, a collection and core both named the same
> > and the core were discovered by core.properties with entries as mentioned
> > in my previous mail.
> >
> > Thanks for the responses. I will continue with the new collection and
> core
> > created by the APIs and test our indexing and queries.
> >
> > Best,
> > Modassar
> >
> >
> > On Thu, Nov 5, 2020 at 12:58 PM Shawn Heisey <ap...@elyograg.org>
> wrote:
> >
> >> On 11/4/2020 9:32 PM, Modassar Ather wrote:
> >>> Another thing: how can I control the core naming? I want the core name
> to
> >>> be *mycore* instead of *mycore**_shard1_replica_n1*/*mycore*
> >>> *_shard2_replica_n2*.
> >>> I tried setting it using property.name=*mycore* but it did not work.
> >>> What can I do to achieve this? I am not able to find any config option.
> >>
> >> Why would you need to this or even want to? It sounds to me like an XY
> >> problem.
> >>
> >> http://xyproblem.info/
> >>
> >>> I understand the core.properties file is required for core discovery
> but
> >>> when this file is present under a subdirectory of SOLR_HOME I see it
> not
> >>> getting loaded and not available in Solr dashboard.
> >>
> >> You should not be trying to manipulate core.properties files yourself.
> >> This is especially discouraged when Solr is running in cloud mode.
> >>
> >> When you're in cloud mode, the collection information in zookeeper will
> >> always be consulted during core discovery. If the found core is NOT
> >> described in zookeeper, it will not be loaded. And in any recent Solr
> >> version when running in cloud mode, a core that is not referenced in ZK
> >> will be entirely deleted.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>
Re: Solr migration related issues.
Posted by Erick Erickson <er...@gmail.com>.
Oh dear.
You made me look at the reference guide. Ouch.
We have this nice page “Defining core.properties” that talks about defining core.properties. Unfortunately it has _no_ warning about the fact that trying to use this in SolrCloud is a really bad idea. As in “don’t do it”. To make matters worse, as you have found out painfully, it kinda worked in cloud mode in times past.
Then the collections API doc says you can add property.name=value, with no mention that the name here should NOT be any of the properties necessary for SolrCloud to operate.
The problem here is that adding property.name=value would set that property to the _same_ value in all cores. Naming all the replicas for a collection to the same thing is not supported officially, if it works in older Solrs that was by chance not design. And there’s no special provision for just using that property as a prefix. That’s really designed for custom properties. And, by the way, “name” is really kind of a no-op, the thing displayed in the drop-down is taken from Zookeeper’s node_name. Please don’t try to name that.
I very strongly recommend that you stop trying to do this. Whatever you are doing that requires a specific name, I’d change _that_ process to use the names assigned by Solr. If it’s just for aesthetics, there’s really no good way to change what’s in the drop-down.
Best,
Erick
> On Nov 5, 2020, at 5:25 AM, Modassar Ather <mo...@gmail.com> wrote:
>
> Hi Shawn,
>
> I understand that we do not need to modify the core.properties and use the
> APIs to create core and collection and that is what I am doing now.
> This question of naming the core as per the choice comes from our older
> setup where we have 12 shards, a collection and core both named the same
> and the core were discovered by core.properties with entries as mentioned
> in my previous mail.
>
> Thanks for the responses. I will continue with the new collection and core
> created by the APIs and test our indexing and queries.
>
> Best,
> Modassar
>
>
> On Thu, Nov 5, 2020 at 12:58 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 11/4/2020 9:32 PM, Modassar Ather wrote:
>>> Another thing: how can I control the core naming? I want the core name to
>>> be *mycore* instead of *mycore**_shard1_replica_n1*/*mycore*
>>> *_shard2_replica_n2*.
>>> I tried setting it using property.name=*mycore* but it did not work.
>>> What can I do to achieve this? I am not able to find any config option.
>>
>> Why would you need to this or even want to? It sounds to me like an XY
>> problem.
>>
>> http://xyproblem.info/
>>
>>> I understand the core.properties file is required for core discovery but
>>> when this file is present under a subdirectory of SOLR_HOME I see it not
>>> getting loaded and not available in Solr dashboard.
>>
>> You should not be trying to manipulate core.properties files yourself.
>> This is especially discouraged when Solr is running in cloud mode.
>>
>> When you're in cloud mode, the collection information in zookeeper will
>> always be consulted during core discovery. If the found core is NOT
>> described in zookeeper, it will not be loaded. And in any recent Solr
>> version when running in cloud mode, a core that is not referenced in ZK
>> will be entirely deleted.
>>
>> Thanks,
>> Shawn
>>
Re: Solr migration related issues.
Posted by Modassar Ather <mo...@gmail.com>.
Hi Shawn,
I understand that we do not need to modify the core.properties and use the
APIs to create core and collection and that is what I am doing now.
This question of naming the core as per the choice comes from our older
setup where we have 12 shards, a collection and core both named the same
and the core were discovered by core.properties with entries as mentioned
in my previous mail.
Thanks for the responses. I will continue with the new collection and core
created by the APIs and test our indexing and queries.
Best,
Modassar
On Thu, Nov 5, 2020 at 12:58 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/4/2020 9:32 PM, Modassar Ather wrote:
> > Another thing: how can I control the core naming? I want the core name to
> > be *mycore* instead of *mycore**_shard1_replica_n1*/*mycore*
> > *_shard2_replica_n2*.
> > I tried setting it using property.name=*mycore* but it did not work.
> > What can I do to achieve this? I am not able to find any config option.
>
> Why would you need to this or even want to? It sounds to me like an XY
> problem.
>
> http://xyproblem.info/
>
> > I understand the core.properties file is required for core discovery but
> > when this file is present under a subdirectory of SOLR_HOME I see it not
> > getting loaded and not available in Solr dashboard.
>
> You should not be trying to manipulate core.properties files yourself.
> This is especially discouraged when Solr is running in cloud mode.
>
> When you're in cloud mode, the collection information in zookeeper will
> always be consulted during core discovery. If the found core is NOT
> described in zookeeper, it will not be loaded. And in any recent Solr
> version when running in cloud mode, a core that is not referenced in ZK
> will be entirely deleted.
>
> Thanks,
> Shawn
>
Re: Solr migration related issues.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/4/2020 9:32 PM, Modassar Ather wrote:
> Another thing: how can I control the core naming? I want the core name to
> be *mycore* instead of *mycore**_shard1_replica_n1*/*mycore*
> *_shard2_replica_n2*.
> I tried setting it using property.name=*mycore* but it did not work.
> What can I do to achieve this? I am not able to find any config option.
Why would you need to this or even want to? It sounds to me like an XY
problem.
http://xyproblem.info/
> I understand the core.properties file is required for core discovery but
> when this file is present under a subdirectory of SOLR_HOME I see it not
> getting loaded and not available in Solr dashboard.
You should not be trying to manipulate core.properties files yourself.
This is especially discouraged when Solr is running in cloud mode.
When you're in cloud mode, the collection information in zookeeper will
always be consulted during core discovery. If the found core is NOT
described in zookeeper, it will not be loaded. And in any recent Solr
version when running in cloud mode, a core that is not referenced in ZK
will be entirely deleted.
Thanks,
Shawn
Re: Solr migration related issues.
Posted by Modassar Ather <mo...@gmail.com>.
Hi Erick,
I have put solr configs in Zookeeper. I have created a collection using the
following API.
admin/collections?action=CREATE&name=mycore&numShards=2&replicationFactor=1&collection.configName=mycore&
property.name=mycore
The collection got created and I can see *mycore**_shard1_replica_n1* and
*mycore**_shard2_replica_n2* under core on the dashboard. The
core.properties got created with the following values.
numShards=2
collection.configName=mycore
name=mycore
replicaType=NRT
shard=shard1
collection=mycore
coreNodeName=core_node3
Another thing: how can I control the core naming? I want the core name to
be *mycore* instead of *mycore**_shard1_replica_n1*/*mycore*
*_shard2_replica_n2*.
I tried setting it using property.name=*mycore* but it did not work.
What can I do to achieve this? I am not able to find any config option.
I understand the core.properties file is required for core discovery but
when this file is present under a subdirectory of SOLR_HOME I see it not
getting loaded and not available in Solr dashboard.
Previous core.properties values :
numShards=2
name=mycore
collection=mycore
configSet=mycore
Can you please help me with this?
Best,
Modassar
On Wed, Nov 4, 2020 at 7:37 PM Erick Erickson <er...@gmail.com>
wrote:
> inline
>
> > On Nov 4, 2020, at 2:17 AM, Modassar Ather <mo...@gmail.com>
> wrote:
> >
> > Thanks Erick and Ilan.
> >
> > I am using APIs to create core and collection and have removed all the
> > entries from core.properties. Currently I am facing init failure and
> > debugging it.
> > Will write back if I am facing any issues.
> >
>
> If that means you still _have_ a core.properties file and it’s empty, that
> won’t
> work.
>
> When Solr starts, it goes through “core discovery”. Starting at SOLR_HOME
> it
> recursively descends the directories and whenever it finds a
> “core.properties”
> file says “aha! There’s a replica here. I'll go tell Zookeeper who I am
> and that
> I'm open for business”. It uses the values in core.properties to know what
> collection and shard it belongs to and which replica of that shard it is.
>
> Incidentally, core discovery stops descending and moves to the next sibling
> directory when it hits the first core.properties file so you can’t have a
> replica
> underneath another replica in your directory tree.
>
> You’ll save yourself a lot of grief if you start with an empty SOLR_HOME
> (except
> for solr.xml if you haven’t put it in Zookeeper. BTW, I’d recommend you do
> put
> put solr.xml in Zookeeper!).
>
> Best,
> Erick
>
>
> > Best,
> > Modassar
> >
> > On Wed, Nov 4, 2020 at 3:20 AM Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> Do note, though, that the default value for legacyCloud changed from
> >> true to false so even though you can get it to work by setting
> >> this cluster prop I wouldn’t…
> >>
> >> The change in the default value is why it’s failing for you.
> >>
> >>
> >>> On Nov 3, 2020, at 11:20 AM, Ilan Ginzburg <il...@gmail.com> wrote:
> >>>
> >>> I second Erick's recommendation, but just for the record legacyCloud
> was
> >>> removed in (upcoming) Solr 9 and is still available in Solr 8.x. Most
> >>> likely this explains Modassar why you found it in the documentation.
> >>>
> >>> Ilan
> >>>
> >>>
> >>> On Tue, Nov 3, 2020 at 5:11 PM Erick Erickson <erickerickson@gmail.com
> >
> >>> wrote:
> >>>
> >>>> You absolutely need core.properties files. It’s just that they
> >>>> should be considered an “implementation detail” that you
> >>>> should rarely, if ever need to be aware of.
> >>>>
> >>>> Scripting manual creation of core.properties files in order
> >>>> to define your collections has never been officially supported, it
> >>>> just happened to work.
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>>> On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Thanks Erick for your response.
> >>>>>
> >>>>> I will certainly use the APIs and not rely on the core.properties. I
> >> was
> >>>>> going through the documentation on core.properties and found it to be
> >>>> still
> >>>>> there.
> >>>>> I have all the solr install scripts based on older Solr versions and
> >>>> wanted
> >>>>> to re-use the same as the core.properties way is still available.
> >>>>>
> >>>>> So does this mean that we do not need core.properties anymore?
> >>>>> How can we ensure that the core name is configurable and not
> >> dynamically
> >>>>> set?
> >>>>>
> >>>>> I will try to use the APIs to create the collection as well as the
> >> cores.
> >>>>>
> >>>>> Best,
> >>>>> Modassar
> >>>>>
> >>>>> On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <
> erickerickson@gmail.com
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> You’re relying on legacyMode, which is no longer supported. In
> >>>>>> older versions of Solr, if a core.properties file was found on disk
> >> Solr
> >>>>>> attempted to create the replica (and collection) on the fly. This is
> >> no
> >>>>>> longer true.
> >>>>>>
> >>>>>>
> >>>>>> Why are you doing it this manually instead of using the collections
> >> API?
> >>>>>> You can precisely place each replica with that API in a way that’ll
> >>>>>> be continued to be supported going forward.
> >>>>>>
> >>>>>> This really sounds like an XY problem, what is the use-case you’re
> >>>>>> trying to solve?
> >>>>>>
> >>>>>> Best,
> >>>>>> Erick
> >>>>>>
> >>>>>>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <modather1981@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the
> entire
> >>>>>>> upgrade I have the first task to install and configure the solr
> with
> >>>> the
> >>>>>>> core and collection. The solr is installed in SolrCloud mode.
> >>>>>>>
> >>>>>>> In Solr 6.5.1 I was using the following key values in
> core.properties
> >>>>>> file.
> >>>>>>> The configuration files were uploaded to zookeeper using the
> upconfig
> >>>>>>> command.
> >>>>>>> The core and collection was automatically created with the setting
> in
> >>>>>>> core.properties files and the configSet uploaded in zookeeper and
> it
> >>>> used
> >>>>>>> to display on the Solr 6.5.1 dashboard.
> >>>>>>>
> >>>>>>> numShards=12
> >>>>>>>
> >>>>>>> name=mycore
> >>>>>>>
> >>>>>>> collection=mycore
> >>>>>>>
> >>>>>>> configSet=mycore
> >>>>>>>
> >>>>>>>
> >>>>>>> With the latest Solr 8.6.3 the same approach is not working. As per
> >> my
> >>>>>>> understanding the core is identified using the location of
> >>>>>> core.properties
> >>>>>>> which is under *<SOLR_HOME>/mycore/core.properties.*
> >>>>>>>
> >>>>>>> Can you please help me with the following?
> >>>>>>>
> >>>>>>>
> >>>>>>> - Is there any property I am missing to load the core and
> collection
> >>>> as
> >>>>>>> it used to be in Solr 6.5.1 with the help of core.properties and
> >>>>>> config set
> >>>>>>> on zookeeper?
> >>>>>>> - The name of the core and collection should be configurable and
> not
> >>>>>> the
> >>>>>>> dynamically generated names. How can I control that in the latest
> >>>> Solr?
> >>>>>>> - Is the core and collection API the only way to create core and
> >>>>>>> collection as I see that the core is also not getting listed even
> if
> >>>>>> the
> >>>>>>> core.properties file is present?
> >>>>>>>
> >>>>>>> Please note that I will be doing a full indexing once the setup is
> >>>> done.
> >>>>>>>
> >>>>>>> Kindly help me with your suggestions.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Modassar
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Solr migration related issues.
Posted by Erick Erickson <er...@gmail.com>.
inline
> On Nov 4, 2020, at 2:17 AM, Modassar Ather <mo...@gmail.com> wrote:
>
> Thanks Erick and Ilan.
>
> I am using APIs to create core and collection and have removed all the
> entries from core.properties. Currently I am facing init failure and
> debugging it.
> Will write back if I am facing any issues.
>
If that means you still _have_ a core.properties file and it’s empty, that won’t
work.
When Solr starts, it goes through “core discovery”. Starting at SOLR_HOME it
recursively descends the directories and whenever it finds a “core.properties”
file says “aha! There’s a replica here. I'll go tell Zookeeper who I am and that
I'm open for business”. It uses the values in core.properties to know what
collection and shard it belongs to and which replica of that shard it is.
Incidentally, core discovery stops descending and moves to the next sibling
directory when it hits the first core.properties file so you can’t have a replica
underneath another replica in your directory tree.
You’ll save yourself a lot of grief if you start with an empty SOLR_HOME (except
for solr.xml if you haven’t put it in Zookeeper. BTW, I’d recommend you do put
put solr.xml in Zookeeper!).
Best,
Erick
> Best,
> Modassar
>
> On Wed, Nov 4, 2020 at 3:20 AM Erick Erickson <er...@gmail.com>
> wrote:
>
>> Do note, though, that the default value for legacyCloud changed from
>> true to false so even though you can get it to work by setting
>> this cluster prop I wouldn’t…
>>
>> The change in the default value is why it’s failing for you.
>>
>>
>>> On Nov 3, 2020, at 11:20 AM, Ilan Ginzburg <il...@gmail.com> wrote:
>>>
>>> I second Erick's recommendation, but just for the record legacyCloud was
>>> removed in (upcoming) Solr 9 and is still available in Solr 8.x. Most
>>> likely this explains Modassar why you found it in the documentation.
>>>
>>> Ilan
>>>
>>>
>>> On Tue, Nov 3, 2020 at 5:11 PM Erick Erickson <er...@gmail.com>
>>> wrote:
>>>
>>>> You absolutely need core.properties files. It’s just that they
>>>> should be considered an “implementation detail” that you
>>>> should rarely, if ever need to be aware of.
>>>>
>>>> Scripting manual creation of core.properties files in order
>>>> to define your collections has never been officially supported, it
>>>> just happened to work.
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>>> On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Thanks Erick for your response.
>>>>>
>>>>> I will certainly use the APIs and not rely on the core.properties. I
>> was
>>>>> going through the documentation on core.properties and found it to be
>>>> still
>>>>> there.
>>>>> I have all the solr install scripts based on older Solr versions and
>>>> wanted
>>>>> to re-use the same as the core.properties way is still available.
>>>>>
>>>>> So does this mean that we do not need core.properties anymore?
>>>>> How can we ensure that the core name is configurable and not
>> dynamically
>>>>> set?
>>>>>
>>>>> I will try to use the APIs to create the collection as well as the
>> cores.
>>>>>
>>>>> Best,
>>>>> Modassar
>>>>>
>>>>> On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <erickerickson@gmail.com
>>>
>>>>> wrote:
>>>>>
>>>>>> You’re relying on legacyMode, which is no longer supported. In
>>>>>> older versions of Solr, if a core.properties file was found on disk
>> Solr
>>>>>> attempted to create the replica (and collection) on the fly. This is
>> no
>>>>>> longer true.
>>>>>>
>>>>>>
>>>>>> Why are you doing it this manually instead of using the collections
>> API?
>>>>>> You can precisely place each replica with that API in a way that’ll
>>>>>> be continued to be supported going forward.
>>>>>>
>>>>>> This really sounds like an XY problem, what is the use-case you’re
>>>>>> trying to solve?
>>>>>>
>>>>>> Best,
>>>>>> Erick
>>>>>>
>>>>>>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
>>>>>>> upgrade I have the first task to install and configure the solr with
>>>> the
>>>>>>> core and collection. The solr is installed in SolrCloud mode.
>>>>>>>
>>>>>>> In Solr 6.5.1 I was using the following key values in core.properties
>>>>>> file.
>>>>>>> The configuration files were uploaded to zookeeper using the upconfig
>>>>>>> command.
>>>>>>> The core and collection was automatically created with the setting in
>>>>>>> core.properties files and the configSet uploaded in zookeeper and it
>>>> used
>>>>>>> to display on the Solr 6.5.1 dashboard.
>>>>>>>
>>>>>>> numShards=12
>>>>>>>
>>>>>>> name=mycore
>>>>>>>
>>>>>>> collection=mycore
>>>>>>>
>>>>>>> configSet=mycore
>>>>>>>
>>>>>>>
>>>>>>> With the latest Solr 8.6.3 the same approach is not working. As per
>> my
>>>>>>> understanding the core is identified using the location of
>>>>>> core.properties
>>>>>>> which is under *<SOLR_HOME>/mycore/core.properties.*
>>>>>>>
>>>>>>> Can you please help me with the following?
>>>>>>>
>>>>>>>
>>>>>>> - Is there any property I am missing to load the core and collection
>>>> as
>>>>>>> it used to be in Solr 6.5.1 with the help of core.properties and
>>>>>> config set
>>>>>>> on zookeeper?
>>>>>>> - The name of the core and collection should be configurable and not
>>>>>> the
>>>>>>> dynamically generated names. How can I control that in the latest
>>>> Solr?
>>>>>>> - Is the core and collection API the only way to create core and
>>>>>>> collection as I see that the core is also not getting listed even if
>>>>>> the
>>>>>>> core.properties file is present?
>>>>>>>
>>>>>>> Please note that I will be doing a full indexing once the setup is
>>>> done.
>>>>>>>
>>>>>>> Kindly help me with your suggestions.
>>>>>>>
>>>>>>> Best,
>>>>>>> Modassar
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Solr migration related issues.
Posted by Modassar Ather <mo...@gmail.com>.
Thanks Erick and Ilan.
I am using APIs to create core and collection and have removed all the
entries from core.properties. Currently I am facing init failure and
debugging it.
Will write back if I am facing any issues.
Best,
Modassar
On Wed, Nov 4, 2020 at 3:20 AM Erick Erickson <er...@gmail.com>
wrote:
> Do note, though, that the default value for legacyCloud changed from
> true to false so even though you can get it to work by setting
> this cluster prop I wouldn’t…
>
> The change in the default value is why it’s failing for you.
>
>
> > On Nov 3, 2020, at 11:20 AM, Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > I second Erick's recommendation, but just for the record legacyCloud was
> > removed in (upcoming) Solr 9 and is still available in Solr 8.x. Most
> > likely this explains Modassar why you found it in the documentation.
> >
> > Ilan
> >
> >
> > On Tue, Nov 3, 2020 at 5:11 PM Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> You absolutely need core.properties files. It’s just that they
> >> should be considered an “implementation detail” that you
> >> should rarely, if ever need to be aware of.
> >>
> >> Scripting manual creation of core.properties files in order
> >> to define your collections has never been officially supported, it
> >> just happened to work.
> >>
> >> Best,
> >> Erick
> >>
> >>> On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks Erick for your response.
> >>>
> >>> I will certainly use the APIs and not rely on the core.properties. I
> was
> >>> going through the documentation on core.properties and found it to be
> >> still
> >>> there.
> >>> I have all the solr install scripts based on older Solr versions and
> >> wanted
> >>> to re-use the same as the core.properties way is still available.
> >>>
> >>> So does this mean that we do not need core.properties anymore?
> >>> How can we ensure that the core name is configurable and not
> dynamically
> >>> set?
> >>>
> >>> I will try to use the APIs to create the collection as well as the
> cores.
> >>>
> >>> Best,
> >>> Modassar
> >>>
> >>> On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <erickerickson@gmail.com
> >
> >>> wrote:
> >>>
> >>>> You’re relying on legacyMode, which is no longer supported. In
> >>>> older versions of Solr, if a core.properties file was found on disk
> Solr
> >>>> attempted to create the replica (and collection) on the fly. This is
> no
> >>>> longer true.
> >>>>
> >>>>
> >>>> Why are you doing it this manually instead of using the collections
> API?
> >>>> You can precisely place each replica with that API in a way that’ll
> >>>> be continued to be supported going forward.
> >>>>
> >>>> This really sounds like an XY problem, what is the use-case you’re
> >>>> trying to solve?
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
> >>>>> upgrade I have the first task to install and configure the solr with
> >> the
> >>>>> core and collection. The solr is installed in SolrCloud mode.
> >>>>>
> >>>>> In Solr 6.5.1 I was using the following key values in core.properties
> >>>> file.
> >>>>> The configuration files were uploaded to zookeeper using the upconfig
> >>>>> command.
> >>>>> The core and collection was automatically created with the setting in
> >>>>> core.properties files and the configSet uploaded in zookeeper and it
> >> used
> >>>>> to display on the Solr 6.5.1 dashboard.
> >>>>>
> >>>>> numShards=12
> >>>>>
> >>>>> name=mycore
> >>>>>
> >>>>> collection=mycore
> >>>>>
> >>>>> configSet=mycore
> >>>>>
> >>>>>
> >>>>> With the latest Solr 8.6.3 the same approach is not working. As per
> my
> >>>>> understanding the core is identified using the location of
> >>>> core.properties
> >>>>> which is under *<SOLR_HOME>/mycore/core.properties.*
> >>>>>
> >>>>> Can you please help me with the following?
> >>>>>
> >>>>>
> >>>>> - Is there any property I am missing to load the core and collection
> >> as
> >>>>> it used to be in Solr 6.5.1 with the help of core.properties and
> >>>> config set
> >>>>> on zookeeper?
> >>>>> - The name of the core and collection should be configurable and not
> >>>> the
> >>>>> dynamically generated names. How can I control that in the latest
> >> Solr?
> >>>>> - Is the core and collection API the only way to create core and
> >>>>> collection as I see that the core is also not getting listed even if
> >>>> the
> >>>>> core.properties file is present?
> >>>>>
> >>>>> Please note that I will be doing a full indexing once the setup is
> >> done.
> >>>>>
> >>>>> Kindly help me with your suggestions.
> >>>>>
> >>>>> Best,
> >>>>> Modassar
> >>>>
> >>>>
> >>
> >>
>
>
Re: Solr migration related issues.
Posted by Erick Erickson <er...@gmail.com>.
Do note, though, that the default value for legacyCloud changed from
true to false so even though you can get it to work by setting
this cluster prop I wouldn’t…
The change in the default value is why it’s failing for you.
> On Nov 3, 2020, at 11:20 AM, Ilan Ginzburg <il...@gmail.com> wrote:
>
> I second Erick's recommendation, but just for the record legacyCloud was
> removed in (upcoming) Solr 9 and is still available in Solr 8.x. Most
> likely this explains Modassar why you found it in the documentation.
>
> Ilan
>
>
> On Tue, Nov 3, 2020 at 5:11 PM Erick Erickson <er...@gmail.com>
> wrote:
>
>> You absolutely need core.properties files. It’s just that they
>> should be considered an “implementation detail” that you
>> should rarely, if ever need to be aware of.
>>
>> Scripting manual creation of core.properties files in order
>> to define your collections has never been officially supported, it
>> just happened to work.
>>
>> Best,
>> Erick
>>
>>> On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com>
>> wrote:
>>>
>>> Thanks Erick for your response.
>>>
>>> I will certainly use the APIs and not rely on the core.properties. I was
>>> going through the documentation on core.properties and found it to be
>> still
>>> there.
>>> I have all the solr install scripts based on older Solr versions and
>> wanted
>>> to re-use the same as the core.properties way is still available.
>>>
>>> So does this mean that we do not need core.properties anymore?
>>> How can we ensure that the core name is configurable and not dynamically
>>> set?
>>>
>>> I will try to use the APIs to create the collection as well as the cores.
>>>
>>> Best,
>>> Modassar
>>>
>>> On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <er...@gmail.com>
>>> wrote:
>>>
>>>> You’re relying on legacyMode, which is no longer supported. In
>>>> older versions of Solr, if a core.properties file was found on disk Solr
>>>> attempted to create the replica (and collection) on the fly. This is no
>>>> longer true.
>>>>
>>>>
>>>> Why are you doing it this manually instead of using the collections API?
>>>> You can precisely place each replica with that API in a way that’ll
>>>> be continued to be supported going forward.
>>>>
>>>> This really sounds like an XY problem, what is the use-case you’re
>>>> trying to solve?
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
>>>>> upgrade I have the first task to install and configure the solr with
>> the
>>>>> core and collection. The solr is installed in SolrCloud mode.
>>>>>
>>>>> In Solr 6.5.1 I was using the following key values in core.properties
>>>> file.
>>>>> The configuration files were uploaded to zookeeper using the upconfig
>>>>> command.
>>>>> The core and collection was automatically created with the setting in
>>>>> core.properties files and the configSet uploaded in zookeeper and it
>> used
>>>>> to display on the Solr 6.5.1 dashboard.
>>>>>
>>>>> numShards=12
>>>>>
>>>>> name=mycore
>>>>>
>>>>> collection=mycore
>>>>>
>>>>> configSet=mycore
>>>>>
>>>>>
>>>>> With the latest Solr 8.6.3 the same approach is not working. As per my
>>>>> understanding the core is identified using the location of
>>>> core.properties
>>>>> which is under *<SOLR_HOME>/mycore/core.properties.*
>>>>>
>>>>> Can you please help me with the following?
>>>>>
>>>>>
>>>>> - Is there any property I am missing to load the core and collection
>> as
>>>>> it used to be in Solr 6.5.1 with the help of core.properties and
>>>> config set
>>>>> on zookeeper?
>>>>> - The name of the core and collection should be configurable and not
>>>> the
>>>>> dynamically generated names. How can I control that in the latest
>> Solr?
>>>>> - Is the core and collection API the only way to create core and
>>>>> collection as I see that the core is also not getting listed even if
>>>> the
>>>>> core.properties file is present?
>>>>>
>>>>> Please note that I will be doing a full indexing once the setup is
>> done.
>>>>>
>>>>> Kindly help me with your suggestions.
>>>>>
>>>>> Best,
>>>>> Modassar
>>>>
>>>>
>>
>>
Re: Solr migration related issues.
Posted by Ilan Ginzburg <il...@gmail.com>.
I second Erick's recommendation, but just for the record legacyCloud was
removed in (upcoming) Solr 9 and is still available in Solr 8.x. Most
likely this explains Modassar why you found it in the documentation.
Ilan
On Tue, Nov 3, 2020 at 5:11 PM Erick Erickson <er...@gmail.com>
wrote:
> You absolutely need core.properties files. It’s just that they
> should be considered an “implementation detail” that you
> should rarely, if ever need to be aware of.
>
> Scripting manual creation of core.properties files in order
> to define your collections has never been officially supported, it
> just happened to work.
>
> Best,
> Erick
>
> > On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com>
> wrote:
> >
> > Thanks Erick for your response.
> >
> > I will certainly use the APIs and not rely on the core.properties. I was
> > going through the documentation on core.properties and found it to be
> still
> > there.
> > I have all the solr install scripts based on older Solr versions and
> wanted
> > to re-use the same as the core.properties way is still available.
> >
> > So does this mean that we do not need core.properties anymore?
> > How can we ensure that the core name is configurable and not dynamically
> > set?
> >
> > I will try to use the APIs to create the collection as well as the cores.
> >
> > Best,
> > Modassar
> >
> > On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> You’re relying on legacyMode, which is no longer supported. In
> >> older versions of Solr, if a core.properties file was found on disk Solr
> >> attempted to create the replica (and collection) on the fly. This is no
> >> longer true.
> >>
> >>
> >> Why are you doing it this manually instead of using the collections API?
> >> You can precisely place each replica with that API in a way that’ll
> >> be continued to be supported going forward.
> >>
> >> This really sounds like an XY problem, what is the use-case you’re
> >> trying to solve?
> >>
> >> Best,
> >> Erick
> >>
> >>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
> >>> upgrade I have the first task to install and configure the solr with
> the
> >>> core and collection. The solr is installed in SolrCloud mode.
> >>>
> >>> In Solr 6.5.1 I was using the following key values in core.properties
> >> file.
> >>> The configuration files were uploaded to zookeeper using the upconfig
> >>> command.
> >>> The core and collection was automatically created with the setting in
> >>> core.properties files and the configSet uploaded in zookeeper and it
> used
> >>> to display on the Solr 6.5.1 dashboard.
> >>>
> >>> numShards=12
> >>>
> >>> name=mycore
> >>>
> >>> collection=mycore
> >>>
> >>> configSet=mycore
> >>>
> >>>
> >>> With the latest Solr 8.6.3 the same approach is not working. As per my
> >>> understanding the core is identified using the location of
> >> core.properties
> >>> which is under *<SOLR_HOME>/mycore/core.properties.*
> >>>
> >>> Can you please help me with the following?
> >>>
> >>>
> >>> - Is there any property I am missing to load the core and collection
> as
> >>> it used to be in Solr 6.5.1 with the help of core.properties and
> >> config set
> >>> on zookeeper?
> >>> - The name of the core and collection should be configurable and not
> >> the
> >>> dynamically generated names. How can I control that in the latest
> Solr?
> >>> - Is the core and collection API the only way to create core and
> >>> collection as I see that the core is also not getting listed even if
> >> the
> >>> core.properties file is present?
> >>>
> >>> Please note that I will be doing a full indexing once the setup is
> done.
> >>>
> >>> Kindly help me with your suggestions.
> >>>
> >>> Best,
> >>> Modassar
> >>
> >>
>
>
Re: Solr migration related issues.
Posted by Erick Erickson <er...@gmail.com>.
You absolutely need core.properties files. It’s just that they
should be considered an “implementation detail” that you
should rarely, if ever need to be aware of.
Scripting manual creation of core.properties files in order
to define your collections has never been officially supported, it
just happened to work.
Best,
Erick
> On Nov 3, 2020, at 11:06 AM, Modassar Ather <mo...@gmail.com> wrote:
>
> Thanks Erick for your response.
>
> I will certainly use the APIs and not rely on the core.properties. I was
> going through the documentation on core.properties and found it to be still
> there.
> I have all the solr install scripts based on older Solr versions and wanted
> to re-use the same as the core.properties way is still available.
>
> So does this mean that we do not need core.properties anymore?
> How can we ensure that the core name is configurable and not dynamically
> set?
>
> I will try to use the APIs to create the collection as well as the cores.
>
> Best,
> Modassar
>
> On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <er...@gmail.com>
> wrote:
>
>> You’re relying on legacyMode, which is no longer supported. In
>> older versions of Solr, if a core.properties file was found on disk Solr
>> attempted to create the replica (and collection) on the fly. This is no
>> longer true.
>>
>>
>> Why are you doing it this manually instead of using the collections API?
>> You can precisely place each replica with that API in a way that’ll
>> be continued to be supported going forward.
>>
>> This really sounds like an XY problem, what is the use-case you’re
>> trying to solve?
>>
>> Best,
>> Erick
>>
>>> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
>>> upgrade I have the first task to install and configure the solr with the
>>> core and collection. The solr is installed in SolrCloud mode.
>>>
>>> In Solr 6.5.1 I was using the following key values in core.properties
>> file.
>>> The configuration files were uploaded to zookeeper using the upconfig
>>> command.
>>> The core and collection was automatically created with the setting in
>>> core.properties files and the configSet uploaded in zookeeper and it used
>>> to display on the Solr 6.5.1 dashboard.
>>>
>>> numShards=12
>>>
>>> name=mycore
>>>
>>> collection=mycore
>>>
>>> configSet=mycore
>>>
>>>
>>> With the latest Solr 8.6.3 the same approach is not working. As per my
>>> understanding the core is identified using the location of
>> core.properties
>>> which is under *<SOLR_HOME>/mycore/core.properties.*
>>>
>>> Can you please help me with the following?
>>>
>>>
>>> - Is there any property I am missing to load the core and collection as
>>> it used to be in Solr 6.5.1 with the help of core.properties and
>> config set
>>> on zookeeper?
>>> - The name of the core and collection should be configurable and not
>> the
>>> dynamically generated names. How can I control that in the latest Solr?
>>> - Is the core and collection API the only way to create core and
>>> collection as I see that the core is also not getting listed even if
>> the
>>> core.properties file is present?
>>>
>>> Please note that I will be doing a full indexing once the setup is done.
>>>
>>> Kindly help me with your suggestions.
>>>
>>> Best,
>>> Modassar
>>
>>
Re: Solr migration related issues.
Posted by Modassar Ather <mo...@gmail.com>.
Thanks Erick for your response.
I will certainly use the APIs and not rely on the core.properties. I was
going through the documentation on core.properties and found it to be still
there.
I have all the solr install scripts based on older Solr versions and wanted
to re-use the same as the core.properties way is still available.
So does this mean that we do not need core.properties anymore?
How can we ensure that the core name is configurable and not dynamically
set?
I will try to use the APIs to create the collection as well as the cores.
Best,
Modassar
On Tue, Nov 3, 2020 at 5:55 PM Erick Erickson <er...@gmail.com>
wrote:
> You’re relying on legacyMode, which is no longer supported. In
> older versions of Solr, if a core.properties file was found on disk Solr
> attempted to create the replica (and collection) on the fly. This is no
> longer true.
>
>
> Why are you doing it this manually instead of using the collections API?
> You can precisely place each replica with that API in a way that’ll
> be continued to be supported going forward.
>
> This really sounds like an XY problem, what is the use-case you’re
> trying to solve?
>
> Best,
> Erick
>
> > On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
> > upgrade I have the first task to install and configure the solr with the
> > core and collection. The solr is installed in SolrCloud mode.
> >
> > In Solr 6.5.1 I was using the following key values in core.properties
> file.
> > The configuration files were uploaded to zookeeper using the upconfig
> > command.
> > The core and collection was automatically created with the setting in
> > core.properties files and the configSet uploaded in zookeeper and it used
> > to display on the Solr 6.5.1 dashboard.
> >
> > numShards=12
> >
> > name=mycore
> >
> > collection=mycore
> >
> > configSet=mycore
> >
> >
> > With the latest Solr 8.6.3 the same approach is not working. As per my
> > understanding the core is identified using the location of
> core.properties
> > which is under *<SOLR_HOME>/mycore/core.properties.*
> >
> > Can you please help me with the following?
> >
> >
> > - Is there any property I am missing to load the core and collection as
> > it used to be in Solr 6.5.1 with the help of core.properties and
> config set
> > on zookeeper?
> > - The name of the core and collection should be configurable and not
> the
> > dynamically generated names. How can I control that in the latest Solr?
> > - Is the core and collection API the only way to create core and
> > collection as I see that the core is also not getting listed even if
> the
> > core.properties file is present?
> >
> > Please note that I will be doing a full indexing once the setup is done.
> >
> > Kindly help me with your suggestions.
> >
> > Best,
> > Modassar
>
>
Re: Solr migration related issues.
Posted by Erick Erickson <er...@gmail.com>.
You’re relying on legacyMode, which is no longer supported. In
older versions of Solr, if a core.properties file was found on disk Solr
attempted to create the replica (and collection) on the fly. This is no
longer true.
Why are you doing it this manually instead of using the collections API?
You can precisely place each replica with that API in a way that’ll
be continued to be supported going forward.
This really sounds like an XY problem, what is the use-case you’re
trying to solve?
Best,
Erick
> On Nov 3, 2020, at 6:39 AM, Modassar Ather <mo...@gmail.com> wrote:
>
> Hi,
>
> I am migrating from Solr 6.5.1 to Solr 8.6.3. As a part of the entire
> upgrade I have the first task to install and configure the solr with the
> core and collection. The solr is installed in SolrCloud mode.
>
> In Solr 6.5.1 I was using the following key values in core.properties file.
> The configuration files were uploaded to zookeeper using the upconfig
> command.
> The core and collection was automatically created with the setting in
> core.properties files and the configSet uploaded in zookeeper and it used
> to display on the Solr 6.5.1 dashboard.
>
> numShards=12
>
> name=mycore
>
> collection=mycore
>
> configSet=mycore
>
>
> With the latest Solr 8.6.3 the same approach is not working. As per my
> understanding the core is identified using the location of core.properties
> which is under *<SOLR_HOME>/mycore/core.properties.*
>
> Can you please help me with the following?
>
>
> - Is there any property I am missing to load the core and collection as
> it used to be in Solr 6.5.1 with the help of core.properties and config set
> on zookeeper?
> - The name of the core and collection should be configurable and not the
> dynamically generated names. How can I control that in the latest Solr?
> - Is the core and collection API the only way to create core and
> collection as I see that the core is also not getting listed even if the
> core.properties file is present?
>
> Please note that I will be doing a full indexing once the setup is done.
>
> Kindly help me with your suggestions.
>
> Best,
> Modassar