You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Alex Pircher <al...@yahoo.de.INVALID> on 2024/06/01 13:36:28 UTC

Avoiding _shard1_replica_n1-directories in Solr cloud mode

Hello all,

in Solr Cloud mode I always get an additional data-directory named *_shard1_replica_n1 for every core which is then used as data-directory.
In Standalone mode everything is working fine.

- Standalone
bin/solr start -force
curl http://localhost:8983/solr/admin/cores
curl http://localhost:8983/solr/testcore/update?commit=true --data-binary '[ { "id": "1", "text": "example" } ]' -H 'Content-type:application/json'
curl http://localhost:8983/solr/testcore/select?q=*

- Cloud mode
With an empty core.properties I get
1. No coreNodeName for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
2. No shard id for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]

So core.properties needs to be populated:
cat <<EOF > /var/solr/data/testcore/core.properties
coreNodeName=core_node1
collection.configName=testcore
name=testcore
numShards=1
shard=shard1
collection=testcore
replicaType=NRT
EOF

Start in cloud mode:
bin/solr start -c -force

An error is occurs, because I think the collection needs to be created:
> coreNodeName core_node1 does not exist in shard shard1, ignore the exception if the replica was deleted

So creating the collection:
bin/solr create -c testcore -d /var/solr/data/testcore -force
> Created collection 'testcore' with 1 shard(s), 1 replica(s) with config-set 'testcore'

After that there is an additional data-directory named *_shard1_replica_n1:
ls /var/solr/data/
> testcore  testcore_shard1_replica_n1  zoo_data

The original directory "testcore" is never used again and can even be deleted.

Does anyone has an idea how Solr in cloud mode can be configured to use the original directory without creating a new one?

Thanks,
 Alex


My test-environment for Solr with Docker:
docker run --rm -it --user root solr:latest /bin/bash

mkdir -p /var/solr/data/testcore/conf
touch /var/solr/data/testcore/core.properties

cat <<EOF > /var/solr/data/testcore/conf/solrconfig.xml
<?xml version="1.0" encoding="UTF-8"?>
<config>
  <luceneMatchVersion>9.8</luceneMatchVersion>
  <schemaFactory class="ClassicIndexSchemaFactory"/>
  <updateHandler class="solr.DirectUpdateHandler2">
    <updateLog>
      <str name="dir">\${solr.ulog.dir:}</str>
    </updateLog>
  </updateHandler>
  <requestHandler name="/select" class="solr.SearchHandler" default="true">
    <lst name="defaults">
      <str name="defType">edismax</str>
      <str name="df">text</str>
      <str name="q.op">AND</str>
    </lst>
  </requestHandler>
</config>
EOF

cat <<EOF > /var/solr/data/testcore/conf/schema.xml
<?xml version="1.0" encoding="UTF-8"?>
<schema name="testcore" version="1.2">
  <uniqueKey>id</uniqueKey>
  <field name="id" type="string" indexed="true" required="true" stored="true"/>
  <field name="_version_" type="long" indexed="true" stored="true" multiValued="false"/>
  <fieldType name="int" class="solr.IntPointField" docValues="true"/>
  <fieldType name="long" class="solr.LongPointField" docValues="true"/>
  <fieldType name="string" class="solr.StrField" omitNorms="true"/>
  <fieldType name="text" class="solr.TextField" positionIncrementGap="100"/>
  <field name="text" type="text" multiValued="true" indexed="true" stored="true"/>
</schema>
EOF


Re: Avoiding _shard1_replica_n1-directories in Solr cloud mode

Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,

Cloud mode is not designed for you to micro manage core.properties. In fact you are not supposed to work on the "core" level at all, but rather issue API requests to create a collection, and let Solr create all the underlying cores. You should not need to care what the names of those core directories are, since it is all managed by Solr. You will index and search using the collection name only.

So if you have a standalone Solr today and want to convert it into a SolrCloud instance, please start from scratch with an empty install, then upload the configSet, then create the collection, then index all your data from scratch. Do not attempt to convert a standalone install into a cloud install.

Jan

> 1. juni 2024 kl. 15:36 skrev Alex Pircher <al...@yahoo.de.INVALID>:
> 
> Hello all,
> 
> in Solr Cloud mode I always get an additional data-directory named *_shard1_replica_n1 for every core which is then used as data-directory.
> In Standalone mode everything is working fine.
> 
> - Standalone
> bin/solr start -force
> curl http://localhost:8983/solr/admin/cores
> curl http://localhost:8983/solr/testcore/update?commit=true --data-binary '[ { "id": "1", "text": "example" } ]' -H 'Content-type:application/json'
> curl http://localhost:8983/solr/testcore/select?q=*
> 
> - Cloud mode
> With an empty core.properties I get
> 1. No coreNodeName for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
> 2. No shard id for CoreDescriptor[name=testcore;instanceDir=/var/solr/data/testcore]
> 
> So core.properties needs to be populated:
> cat <<EOF > /var/solr/data/testcore/core.properties
> coreNodeName=core_node1
> collection.configName=testcore
> name=testcore
> numShards=1
> shard=shard1
> collection=testcore
> replicaType=NRT
> EOF
> 
> Start in cloud mode:
> bin/solr start -c -force
> 
> An error is occurs, because I think the collection needs to be created:
>> coreNodeName core_node1 does not exist in shard shard1, ignore the exception if the replica was deleted
> 
> So creating the collection:
> bin/solr create -c testcore -d /var/solr/data/testcore -force
>> Created collection 'testcore' with 1 shard(s), 1 replica(s) with config-set 'testcore'
> 
> After that there is an additional data-directory named *_shard1_replica_n1:
> ls /var/solr/data/
>> testcore  testcore_shard1_replica_n1  zoo_data
> 
> The original directory "testcore" is never used again and can even be deleted.
> 
> Does anyone has an idea how Solr in cloud mode can be configured to use the original directory without creating a new one?
> 
> Thanks,
>  Alex
> 
> 
> My test-environment for Solr with Docker:
> docker run --rm -it --user root solr:latest /bin/bash
> 
> mkdir -p /var/solr/data/testcore/conf
> touch /var/solr/data/testcore/core.properties
> 
> cat <<EOF > /var/solr/data/testcore/conf/solrconfig.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <config>
>   <luceneMatchVersion>9.8</luceneMatchVersion>
>   <schemaFactory class="ClassicIndexSchemaFactory"/>
>   <updateHandler class="solr.DirectUpdateHandler2">
>     <updateLog>
>       <str name="dir">\${solr.ulog.dir:}</str>
>     </updateLog>
>   </updateHandler>
>   <requestHandler name="/select" class="solr.SearchHandler" default="true">
>     <lst name="defaults">
>       <str name="defType">edismax</str>
>       <str name="df">text</str>
>       <str name="q.op">AND</str>
>     </lst>
>   </requestHandler>
> </config>
> EOF
> 
> cat <<EOF > /var/solr/data/testcore/conf/schema.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <schema name="testcore" version="1.2">
>   <uniqueKey>id</uniqueKey>
>   <field name="id" type="string" indexed="true" required="true" stored="true"/>
>   <field name="_version_" type="long" indexed="true" stored="true" multiValued="false"/>
>   <fieldType name="int" class="solr.IntPointField" docValues="true"/>
>   <fieldType name="long" class="solr.LongPointField" docValues="true"/>
>   <fieldType name="string" class="solr.StrField" omitNorms="true"/>
>   <fieldType name="text" class="solr.TextField" positionIncrementGap="100"/>
>   <field name="text" type="text" multiValued="true" indexed="true" stored="true"/>
> </schema>
> EOF
>