You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Alex Pircher <al...@yahoo.de.INVALID> on 2024/06/01 13:36:28 UTC
Avoiding _shard1_replica_n1-directories in Solr cloud mode
Hello all,
in Solr Cloud mode I always get an additional data-directory named *_shard1_replica_n1 for every core which is then used as data-directory.
In Standalone mode everything is working fine.
- Standalone
bin/solr start -force
curl http://localhost:8983/solr/admin/cores
curl http://localhost:8983/solr/testcore/update?commit=true --data-binary '[ { "id": "1", "text": "example" } ]' -H 'Content-type:application/json'
curl http://localhost:8983/solr/testcore/select?q=*
- Cloud mode
With an empty core.properties I get
1. No coreNodeName for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
2. No shard id for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
So core.properties needs to be populated:
cat <<EOF > /var/solr/data/testcore/core.properties
coreNodeName=core_node1
collection.configName=testcore
name=testcore
numShards=1
shard=shard1
collection=testcore
replicaType=NRT
EOF
Start in cloud mode:
bin/solr start -c -force
An error is occurs, because I think the collection needs to be created:
> coreNodeName core_node1 does not exist in shard shard1, ignore the exception if the replica was deleted
So creating the collection:
bin/solr create -c testcore -d /var/solr/data/testcore -force
> Created collection 'testcore' with 1 shard(s), 1 replica(s) with config-set 'testcore'
After that there is an additional data-directory named *_shard1_replica_n1:
ls /var/solr/data/
> testcore testcore_shard1_replica_n1 zoo_data
The original directory "testcore" is never used again and can even be deleted.
Does anyone has an idea how Solr in cloud mode can be configured to use the original directory without creating a new one?
Thanks,
Alex
My test-environment for Solr with Docker:
docker run --rm -it --user root solr:latest /bin/bash
mkdir -p /var/solr/data/testcore/conf
touch /var/solr/data/testcore/core.properties
cat <<EOF > /var/solr/data/testcore/conf/solrconfig.xml
<?xml version="1.0" encoding="UTF-8"?>
<config>
<luceneMatchVersion>9.8</luceneMatchVersion>
<schemaFactory class="ClassicIndexSchemaFactory"/>
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">\${solr.ulog.dir:}</str>
</updateLog>
</updateHandler>
<requestHandler name="/select" class="solr.SearchHandler" default="true">
<lst name="defaults">
<str name="defType">edismax</str>
<str name="df">text</str>
<str name="q.op">AND</str>
</lst>
</requestHandler>
</config>
EOF
cat <<EOF > /var/solr/data/testcore/conf/schema.xml
<?xml version="1.0" encoding="UTF-8"?>
<schema name="testcore" version="1.2">
<uniqueKey>id</uniqueKey>
<field name="id" type="string" indexed="true" required="true" stored="true"/>
<field name="_version_" type="long" indexed="true" stored="true" multiValued="false"/>
<fieldType name="int" class="solr.IntPointField" docValues="true"/>
<fieldType name="long" class="solr.LongPointField" docValues="true"/>
<fieldType name="string" class="solr.StrField" omitNorms="true"/>
<fieldType name="text" class="solr.TextField" positionIncrementGap="100"/>
<field name="text" type="text" multiValued="true" indexed="true" stored="true"/>
</schema>
EOF
Re: Avoiding _shard1_replica_n1-directories in Solr cloud mode
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Cloud mode is not designed for you to micro manage core.properties. In fact you are not supposed to work on the "core" level at all, but rather issue API requests to create a collection, and let Solr create all the underlying cores. You should not need to care what the names of those core directories are, since it is all managed by Solr. You will index and search using the collection name only.
So if you have a standalone Solr today and want to convert it into a SolrCloud instance, please start from scratch with an empty install, then upload the configSet, then create the collection, then index all your data from scratch. Do not attempt to convert a standalone install into a cloud install.
Jan
> 1. juni 2024 kl. 15:36 skrev Alex Pircher <al...@yahoo.de.INVALID>:
>
> Hello all,
>
> in Solr Cloud mode I always get an additional data-directory named *_shard1_replica_n1 for every core which is then used as data-directory.
> In Standalone mode everything is working fine.
>
> - Standalone
> bin/solr start -force
> curl http://localhost:8983/solr/admin/cores
> curl http://localhost:8983/solr/testcore/update?commit=true --data-binary '[ { "id": "1", "text": "example" } ]' -H 'Content-type:application/json'
> curl http://localhost:8983/solr/testcore/select?q=*
>
> - Cloud mode
> With an empty core.properties I get
> 1. No coreNodeName for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
> 2. No shard id for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
>
> So core.properties needs to be populated:
> cat <<EOF > /var/solr/data/testcore/core.properties
> coreNodeName=core_node1
> collection.configName=testcore
> name=testcore
> numShards=1
> shard=shard1
> collection=testcore
> replicaType=NRT
> EOF
>
> Start in cloud mode:
> bin/solr start -c -force
>
> An error is occurs, because I think the collection needs to be created:
>> coreNodeName core_node1 does not exist in shard shard1, ignore the exception if the replica was deleted
>
> So creating the collection:
> bin/solr create -c testcore -d /var/solr/data/testcore -force
>> Created collection 'testcore' with 1 shard(s), 1 replica(s) with config-set 'testcore'
>
> After that there is an additional data-directory named *_shard1_replica_n1:
> ls /var/solr/data/
>> testcore testcore_shard1_replica_n1 zoo_data
>
> The original directory "testcore" is never used again and can even be deleted.
>
> Does anyone has an idea how Solr in cloud mode can be configured to use the original directory without creating a new one?
>
> Thanks,
> Alex
>
>
> My test-environment for Solr with Docker:
> docker run --rm -it --user root solr:latest /bin/bash
>
> mkdir -p /var/solr/data/testcore/conf
> touch /var/solr/data/testcore/core.properties
>
> cat <<EOF > /var/solr/data/testcore/conf/solrconfig.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <config>
> <luceneMatchVersion>9.8</luceneMatchVersion>
> <schemaFactory class="ClassicIndexSchemaFactory"/>
> <updateHandler class="solr.DirectUpdateHandler2">
> <updateLog>
> <str name="dir">\${solr.ulog.dir:}</str>
> </updateLog>
> </updateHandler>
> <requestHandler name="/select" class="solr.SearchHandler" default="true">
> <lst name="defaults">
> <str name="defType">edismax</str>
> <str name="df">text</str>
> <str name="q.op">AND</str>
> </lst>
> </requestHandler>
> </config>
> EOF
>
> cat <<EOF > /var/solr/data/testcore/conf/schema.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <schema name="testcore" version="1.2">
> <uniqueKey>id</uniqueKey>
> <field name="id" type="string" indexed="true" required="true" stored="true"/>
> <field name="_version_" type="long" indexed="true" stored="true" multiValued="false"/>
> <fieldType name="int" class="solr.IntPointField" docValues="true"/>
> <fieldType name="long" class="solr.LongPointField" docValues="true"/>
> <fieldType name="string" class="solr.StrField" omitNorms="true"/>
> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"/>
> <field name="text" type="text" multiValued="true" indexed="true" stored="true"/>
> </schema>
> EOF
>