You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zheng Lin Edwin Yeo <ed...@gmail.com> on 2017/07/06 03:34:47 UTC
Placing different collections on different hard disk/folder
Hi,
Would like to check, how can we place the indexed files of different
collections on different hard disk/folder, but they are in the same node?
For example, I want collection1 to be placed in C: drive, collection2 to be
placed in D: drive, and collection3 to be placed in E: drive.
I am using Solr 6.5.1
Regards,
Edwin
Re: Placing different collections on different hard disk/folder
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/12/2017 8:25 PM, Zheng Lin Edwin Yeo wrote:
> Thanks Shawn and Erick.
>
> We are planning to migrate the data in two of the largest collections to
> another hard disk, while the rest of the collections remains at the default
> one for the main core.
> So there are already data indexed in the collections.
> Will this method work, or we have to create a new collection, edit the
> core.properties files, then transfer the index files to this new collection?
Yes.
You can use the method Erick mentioned, or you can use the alternate
method outlined in the paragraph below. For Erick's idea, you use the
Collections API to add a new replica with a new dataDir property, and
let Solr copy the index data for you, then use DELETEREPLICA to get rid
of the one in the wrong location. No downtime for any Solr instances,
and you don't have to stop indexing.
For the alternate method, you'll need to stop Solr, edit
core.properties, and copy the contents of the corename/data directory to
the location pointed at by the added dataDir property in the
core.properties file. Then when you restart Solr, it will use the
existing data in the new location. You can check the core overview in
the admin UI to make sure that the dataDir has been updated before you
delete the old data directory.
Thanks,
Shawn
Re: Placing different collections on different hard disk/folder
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Thanks Shawn and Erick.
We are planning to migrate the data in two of the largest collections to
another hard disk, while the rest of the collections remains at the default
one for the main core.
So there are already data indexed in the collections.
Will this method work, or we have to create a new collection, edit the
core.properties files, then transfer the index files to this new collection?
Regards,
Edwin
On 12 July 2017 at 23:07, Erick Erickson <er...@gmail.com> wrote:
> Shawn's way will work, of course you have to be sure you didn't index
> any data before editing all the core.properties files.
>
> There's another way to set the dataDir per core though that has the
> advantage of not entailing any down time or hand editing files:
> > create the collection with createNodeSet=EMPTY. No replicas are created
> > Now add each replica with ADDREPLICA. In addition to all the regular
> params, use property.dataDir=place_you_want_the_index
>
> The ADDREPLICA property.blah param is intended exactly to set
> arbitrary properties in core.properties for the replica...
>
> Best,
> Erick
>
> On Wed, Jul 12, 2017 at 6:24 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> > On 7/12/2017 12:38 AM, Zheng Lin Edwin Yeo wrote:
> >> I found that we can set the path under <dataDir> in solrconfig.xml
> >>
> >> However, this seems to work only if there is one replica. How do we set
> it
> >> if we have 2 or more replica?
> >
> > Setting dataDir in solrconfig.xml is something that really only works in
> > standalone Solr. For SolrCloud, this method has issues that are
> > difficult to get around.
> >
> > Another option that works in ANY Solr mode is changing dataDir in the
> > core.properties file that every core uses. Create the collection,
> > allowing Solr to create the cores in the default way. Shut down Solr
> > and edit the core.properties file for each core that you want to have
> > the data in a different location. Add a dataDir property pointing at
> > the new location for that core's data. If the core actually has any
> > contents, you can move the data to that location, but if not, you can
> > simply let Solr create the data itself when it starts back up.
> >
> > The core.properties file is in Java properties format, which is well
> > documented in multiple places around the Internet.
> >
> > https://www.google.com/search?q=java+properties+format&ie=utf-8&oe=utf-8
> >
> > If the dataDir location is not an absolute path, then it will be
> > relative to the instanceDir -- the place where core.properties is. The
> > dataDir value defaults to a simple value of "data".
> >
> > Thanks,
> > Shawn
> >
>
Re: Placing different collections on different hard disk/folder
Posted by Erick Erickson <er...@gmail.com>.
Shawn's way will work, of course you have to be sure you didn't index
any data before editing all the core.properties files.
There's another way to set the dataDir per core though that has the
advantage of not entailing any down time or hand editing files:
> create the collection with createNodeSet=EMPTY. No replicas are created
> Now add each replica with ADDREPLICA. In addition to all the regular params, use property.dataDir=place_you_want_the_index
The ADDREPLICA property.blah param is intended exactly to set
arbitrary properties in core.properties for the replica...
Best,
Erick
On Wed, Jul 12, 2017 at 6:24 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/12/2017 12:38 AM, Zheng Lin Edwin Yeo wrote:
>> I found that we can set the path under <dataDir> in solrconfig.xml
>>
>> However, this seems to work only if there is one replica. How do we set it
>> if we have 2 or more replica?
>
> Setting dataDir in solrconfig.xml is something that really only works in
> standalone Solr. For SolrCloud, this method has issues that are
> difficult to get around.
>
> Another option that works in ANY Solr mode is changing dataDir in the
> core.properties file that every core uses. Create the collection,
> allowing Solr to create the cores in the default way. Shut down Solr
> and edit the core.properties file for each core that you want to have
> the data in a different location. Add a dataDir property pointing at
> the new location for that core's data. If the core actually has any
> contents, you can move the data to that location, but if not, you can
> simply let Solr create the data itself when it starts back up.
>
> The core.properties file is in Java properties format, which is well
> documented in multiple places around the Internet.
>
> https://www.google.com/search?q=java+properties+format&ie=utf-8&oe=utf-8
>
> If the dataDir location is not an absolute path, then it will be
> relative to the instanceDir -- the place where core.properties is. The
> dataDir value defaults to a simple value of "data".
>
> Thanks,
> Shawn
>
Re: Placing different collections on different hard disk/folder
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/12/2017 12:38 AM, Zheng Lin Edwin Yeo wrote:
> I found that we can set the path under <dataDir> in solrconfig.xml
>
> However, this seems to work only if there is one replica. How do we set it
> if we have 2 or more replica?
Setting dataDir in solrconfig.xml is something that really only works in
standalone Solr. For SolrCloud, this method has issues that are
difficult to get around.
Another option that works in ANY Solr mode is changing dataDir in the
core.properties file that every core uses. Create the collection,
allowing Solr to create the cores in the default way. Shut down Solr
and edit the core.properties file for each core that you want to have
the data in a different location. Add a dataDir property pointing at
the new location for that core's data. If the core actually has any
contents, you can move the data to that location, but if not, you can
simply let Solr create the data itself when it starts back up.
The core.properties file is in Java properties format, which is well
documented in multiple places around the Internet.
https://www.google.com/search?q=java+properties+format&ie=utf-8&oe=utf-8
If the dataDir location is not an absolute path, then it will be
relative to the instanceDir -- the place where core.properties is. The
dataDir value defaults to a simple value of "data".
Thanks,
Shawn
Re: Placing different collections on different hard disk/folder
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
I found that we can set the path under <dataDir> in solrconfig.xml
However, this seems to work only if there is one replica. How do we set it
if we have 2 or more replica?
Regards,
Edwin
On 6 July 2017 at 11:34, Zheng Lin Edwin Yeo <ed...@gmail.com> wrote:
> Hi,
>
> Would like to check, how can we place the indexed files of different
> collections on different hard disk/folder, but they are in the same node?
>
> For example, I want collection1 to be placed in C: drive, collection2 to
> be placed in D: drive, and collection3 to be placed in E: drive.
>
> I am using Solr 6.5.1
>
> Regards,
> Edwin
>