You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bharath Kumar <bh...@gmail.com> on 2019/01/30 01:51:55 UTC
Creating shard with core.properties
Hi All,
I am trying to create a shard using solr 7.6.0 using just core.properties
file (like auto-discovering the shard) with legacyCloud set to false. But i
am getting an error message like below even though i specify the
coreNodeName in the core.properties file:-
"coreNodeName " + coreNodeName + " does not exist in shard " +
cloudDesc.getShardId() +
", ignore the exception if the replica was deleted");
Please note my zookeeper state is new and does not have any state
registered earlier. Can you please help? The reason i need this is, we are
trying to migrate from 6.1 to 7.6.0 and i have a single shard with 2
replicas created using core.properties and not using the collection api.
--
Thanks & Regards,
Bharath MV Kumar
"Life is short, enjoy every moment of it"
Re: Creating shard with core.properties
Posted by Erick Erickson <er...@gmail.com>.
I think you're making this much more difficult for yourself than necessary.
I'd _strongly_ recommend you abandon this approach and use the
collections AP. Perhaps you'd need to create some kind of script
that handles core creation and the like. If you know where the
core.properties file should be for instance (which you must if
you're trying to create it manually) you can specify instanceDir
and/or dataDir in the ADDREPLICA command to point it to an
index for example.
You can use the EMPTY flag on the collection create command
to create the skeleton in ZK with _no_ replicas defined and use
ADDREPLICA.
Frankly, it sounds like you got started down this road with
legacyCloud quite some time ago and are unwilling to change
even though Solr has changed dramatically. It's always
hard to throw away what's worked in the past, but it's also
sometimes necessary.
Best,
Erick
On Fri, Feb 1, 2019 at 2:41 AM Bharath Kumar <bh...@gmail.com> wrote:
>
> Thanks Shawn for your inputs and the pointer to the documentation. Our
> setup currently has 1 shard and 2 replicas for that shard and we do not
> want a manual step which involves creating a collection since for SOLR
> Cloud at least more than 50% of the shard nodes should be up and running.
> Also if the zookeeper states go bad for some reason, we will need to
> re-create the collection, whereas in the legacy cloud mode with manual
> core.properties creation it has helped us bring up the solr cloud even
> without any known zookeeper states after an upgrade and not do any
> additional step.
>
> On Wed, Jan 30, 2019 at 3:49 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 1/30/2019 3:36 PM, Bharath Kumar wrote:
> > > Thanks Erick. We cleanup the zookeeper state on every installation, so
> > the
> > > zookeeper states are gone. So what should we do in case of a new 7.6
> > > installation where we want to manually create core.properties and use the
> > > non-legacy cloud option? Is it in order to use non-legacy cloud, we
> > should
> > > use the collections api to create a collection first and then use the
> > > manual core.properties for auto-discovery?
> >
> > *ALL* creations and modifications to SolrCloud collections should be
> > done using the Collections API. Creating cores directly (either with
> > core.properties or the CoreAdmin API) is something that will almost
> > certainly bite you hard. Based on what Erick has said, I don't think
> > you can even do it at all when legacy mode is disabled. Even when you
> > can ... don't.
> >
> > > Because in the legacy cloud mode we were just creating the
> > core.properties
> > > manually and that would update the zookeeper state when the solr boots
> > up.
> > > Can you please help me with this?
> >
> > Use the Collections API. This is the recommendation even for experts
> > who really know the code. Creating cores manually in ANY SolrCloud
> > install is a recipe for problems, even in legacy mode.
> >
> > There is a very large warning box (red triangle with an exclamation
> > point) in this section of the documentation:
> >
> >
> > https://lucene.apache.org/solr/guide/7_6/coreadmin-api.html#coreadmin-create
> >
> > One of the first things it says there in that warning box is that the
> > CoreAdmin API should not be used in SolrCloud. Manually creating
> > core.properties files and restarting Solr is effectively the same thing
> > as using the CoreAdmin API.
> >
> > Thanks,
> > Shawn
> >
>
>
> --
> Thanks & Regards,
> Bharath MV Kumar
>
> "Life is short, enjoy every moment of it"
Re: Creating shard with core.properties
Posted by Bharath Kumar <bh...@gmail.com>.
Thanks Shawn for your inputs and the pointer to the documentation. Our
setup currently has 1 shard and 2 replicas for that shard and we do not
want a manual step which involves creating a collection since for SOLR
Cloud at least more than 50% of the shard nodes should be up and running.
Also if the zookeeper states go bad for some reason, we will need to
re-create the collection, whereas in the legacy cloud mode with manual
core.properties creation it has helped us bring up the solr cloud even
without any known zookeeper states after an upgrade and not do any
additional step.
On Wed, Jan 30, 2019 at 3:49 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/30/2019 3:36 PM, Bharath Kumar wrote:
> > Thanks Erick. We cleanup the zookeeper state on every installation, so
> the
> > zookeeper states are gone. So what should we do in case of a new 7.6
> > installation where we want to manually create core.properties and use the
> > non-legacy cloud option? Is it in order to use non-legacy cloud, we
> should
> > use the collections api to create a collection first and then use the
> > manual core.properties for auto-discovery?
>
> *ALL* creations and modifications to SolrCloud collections should be
> done using the Collections API. Creating cores directly (either with
> core.properties or the CoreAdmin API) is something that will almost
> certainly bite you hard. Based on what Erick has said, I don't think
> you can even do it at all when legacy mode is disabled. Even when you
> can ... don't.
>
> > Because in the legacy cloud mode we were just creating the
> core.properties
> > manually and that would update the zookeeper state when the solr boots
> up.
> > Can you please help me with this?
>
> Use the Collections API. This is the recommendation even for experts
> who really know the code. Creating cores manually in ANY SolrCloud
> install is a recipe for problems, even in legacy mode.
>
> There is a very large warning box (red triangle with an exclamation
> point) in this section of the documentation:
>
>
> https://lucene.apache.org/solr/guide/7_6/coreadmin-api.html#coreadmin-create
>
> One of the first things it says there in that warning box is that the
> CoreAdmin API should not be used in SolrCloud. Manually creating
> core.properties files and restarting Solr is effectively the same thing
> as using the CoreAdmin API.
>
> Thanks,
> Shawn
>
--
Thanks & Regards,
Bharath MV Kumar
"Life is short, enjoy every moment of it"
Re: Creating shard with core.properties
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/30/2019 3:36 PM, Bharath Kumar wrote:
> Thanks Erick. We cleanup the zookeeper state on every installation, so the
> zookeeper states are gone. So what should we do in case of a new 7.6
> installation where we want to manually create core.properties and use the
> non-legacy cloud option? Is it in order to use non-legacy cloud, we should
> use the collections api to create a collection first and then use the
> manual core.properties for auto-discovery?
*ALL* creations and modifications to SolrCloud collections should be
done using the Collections API. Creating cores directly (either with
core.properties or the CoreAdmin API) is something that will almost
certainly bite you hard. Based on what Erick has said, I don't think
you can even do it at all when legacy mode is disabled. Even when you
can ... don't.
> Because in the legacy cloud mode we were just creating the core.properties
> manually and that would update the zookeeper state when the solr boots up.
> Can you please help me with this?
Use the Collections API. This is the recommendation even for experts
who really know the code. Creating cores manually in ANY SolrCloud
install is a recipe for problems, even in legacy mode.
There is a very large warning box (red triangle with an exclamation
point) in this section of the documentation:
https://lucene.apache.org/solr/guide/7_6/coreadmin-api.html#coreadmin-create
One of the first things it says there in that warning box is that the
CoreAdmin API should not be used in SolrCloud. Manually creating
core.properties files and restarting Solr is effectively the same thing
as using the CoreAdmin API.
Thanks,
Shawn
Re: Creating shard with core.properties
Posted by Bharath Kumar <bh...@gmail.com>.
Thanks Erick. We cleanup the zookeeper state on every installation, so the
zookeeper states are gone. So what should we do in case of a new 7.6
installation where we want to manually create core.properties and use the
non-legacy cloud option? Is it in order to use non-legacy cloud, we should
use the collections api to create a collection first and then use the
manual core.properties for auto-discovery?
Because in the legacy cloud mode we were just creating the core.properties
manually and that would update the zookeeper state when the solr boots up.
Can you please help me with this?
On Wed, Jan 30, 2019 at 9:08 AM Erick Erickson <er...@gmail.com>
wrote:
> This seems very confused. When you say your zookeeper
> state is new, you mean there's no remnant of your old 6.1
> collection? Then manually creating a core.properties file
> won't do you any good as there's no collection to add it to.
>
> You cannot just create a core.properties file and expect Solr
> to reconstruct the entire collection information with legacyCloud
> false. That means that ZooKeeper is considered "the one
> source of truth" and if something on a local disk isn't reflected
> in the state.json file, it's considered invalid.
>
> This really sounds like an XY problem and your goal is to
> upgrade a cluster from 6.1 to 7.6. You should just be able
> to install 7.6 over 6.1 and fire it up. It should not be necessary
> to do anything else. By "over" here I mean
> > install 7.6
> > shut down 6.1
> > start 7.6 with the SOLR_HOME pointing to the same place
> as your 6.1 for each Solr instance.
>
> If you want to create a new cluster, say for testing purposes or
> whatever what I'd do is just create an identical collection ("Identical"
> here means same number of shards, one replica each) with the
> collections API. Then shut down your new Solr and copy the data
> directory from a replica from your 6.1 install to the corresponding
> replica in your 7.6 install. You should NOT be actively indexing at this
> time and should have issued a commit to the 6.1 or shut 6.1 down.
> "corresponding replica" here is the replica with the same "range", but
> in this case it doesn't matter since you only have one shard.
>
> Then use the collections ADDREPLICA command to add as many
> replicas as you want.
>
> Best,
> Erick
>
> On Tue, Jan 29, 2019 at 8:14 PM Bharath Kumar <bh...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > I am trying to create a shard using solr 7.6.0 using just core.properties
> > file (like auto-discovering the shard) with legacyCloud set to false.
> But i
> > am getting an error message like below even though i specify the
> > coreNodeName in the core.properties file:-
> >
> > "coreNodeName " + coreNodeName + " does not exist in shard " +
> > cloudDesc.getShardId() +
> > ", ignore the exception if the replica was deleted");
> >
> > Please note my zookeeper state is new and does not have any state
> > registered earlier. Can you please help? The reason i need this is, we
> are
> > trying to migrate from 6.1 to 7.6.0 and i have a single shard with 2
> > replicas created using core.properties and not using the collection api.
> > --
> > Thanks & Regards,
> > Bharath MV Kumar
> >
> > "Life is short, enjoy every moment of it"
>
--
Thanks & Regards,
Bharath MV Kumar
"Life is short, enjoy every moment of it"
Re: Creating shard with core.properties
Posted by Erick Erickson <er...@gmail.com>.
This seems very confused. When you say your zookeeper
state is new, you mean there's no remnant of your old 6.1
collection? Then manually creating a core.properties file
won't do you any good as there's no collection to add it to.
You cannot just create a core.properties file and expect Solr
to reconstruct the entire collection information with legacyCloud
false. That means that ZooKeeper is considered "the one
source of truth" and if something on a local disk isn't reflected
in the state.json file, it's considered invalid.
This really sounds like an XY problem and your goal is to
upgrade a cluster from 6.1 to 7.6. You should just be able
to install 7.6 over 6.1 and fire it up. It should not be necessary
to do anything else. By "over" here I mean
> install 7.6
> shut down 6.1
> start 7.6 with the SOLR_HOME pointing to the same place
as your 6.1 for each Solr instance.
If you want to create a new cluster, say for testing purposes or
whatever what I'd do is just create an identical collection ("Identical"
here means same number of shards, one replica each) with the
collections API. Then shut down your new Solr and copy the data
directory from a replica from your 6.1 install to the corresponding
replica in your 7.6 install. You should NOT be actively indexing at this
time and should have issued a commit to the 6.1 or shut 6.1 down.
"corresponding replica" here is the replica with the same "range", but
in this case it doesn't matter since you only have one shard.
Then use the collections ADDREPLICA command to add as many
replicas as you want.
Best,
Erick
On Tue, Jan 29, 2019 at 8:14 PM Bharath Kumar <bh...@gmail.com> wrote:
>
> Hi All,
>
> I am trying to create a shard using solr 7.6.0 using just core.properties
> file (like auto-discovering the shard) with legacyCloud set to false. But i
> am getting an error message like below even though i specify the
> coreNodeName in the core.properties file:-
>
> "coreNodeName " + coreNodeName + " does not exist in shard " +
> cloudDesc.getShardId() +
> ", ignore the exception if the replica was deleted");
>
> Please note my zookeeper state is new and does not have any state
> registered earlier. Can you please help? The reason i need this is, we are
> trying to migrate from 6.1 to 7.6.0 and i have a single shard with 2
> replicas created using core.properties and not using the collection api.
> --
> Thanks & Regards,
> Bharath MV Kumar
>
> "Life is short, enjoy every moment of it"