You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Em <ma...@yahoo.de> on 2011/01/20 11:39:57 UTC
Multicore Relaod Theoretical Question
Hello list,
I got a theoretical question about a Multicore-Situation:
I got two cores: active, inactive
The active core serves all the queries.
The inactive core is the tricky thing:
I create an optimized index outside the environment and want to insert that
optimized index 1 to 1 into the inactive core, which means replacing
everything inside the index-directory.
After this is done, I would like to reload the inactive core, so that it is
ready for a core-swap and ready for serving queries on top of the new
inserted optimized index.
Is it possible to handle such a situation?
Thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2293999.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multicore Relaod Theoretical Question
Posted by Em <ma...@yahoo.de>.
Thanks Alexander, what a valuable ressource :).
- Em
--
View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2321335.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multicore Relaod Theoretical Question
Posted by Alexander Kanarsky <ka...@gmail.com>.
Em,
that's correct. You can use 'lsof' to see file handles still in use.
See http://0xfe.blogspot.com/2006/03/troubleshooting-unix-systems-with-lsof.html,
"Recipe #11".
-Alexander
On Sun, Jan 23, 2011 at 1:52 AM, Em <ma...@yahoo.de> wrote:
>
> Hi Alexander,
>
> thank you for your response.
>
> You said that the old index files were still in use. That means Linux does
> not *really* delete them until Solr frees its locks from it, which happens
> while reloading?
>
>
>
> Thank you for sharing your experiences!
>
> Kind regards,
> Em
>
>
> Alexander Kanarsky wrote:
>>
>> Em,
>>
>> yes, you can replace the index (get the new one into a separate folder
>> like index.new and then rename it to the index folder) outside the
>> Solr, then just do the http call to reload the core.
>>
>> Note that the old index files may still be in use (continue to serve
>> the queries while reloading), even if the old index folder is deleted
>> - that is on Linux filesystems, not sure about NTFS.
>> That means the space on disk will be freed only when the old files are
>> not referenced by Solr searcher any longer.
>>
>> -Alexander
>>
>> On Sat, Jan 22, 2011 at 1:51 PM, Em <ma...@yahoo.de> wrote:
>>>
>>> Hi Erick,
>>>
>>> thanks for your response.
>>>
>>> Yes, it's really not that easy.
>>>
>>> However, the target is to avoid any kind of master-slave-setup.
>>>
>>> The most recent idea i got is to create a new core with a data-dir
>>> pointing
>>> to an already existing directory with a fully optimized index.
>>>
>>> Regards,
>>> Em
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2310709.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2312778.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Multicore Relaod Theoretical Question
Posted by Em <ma...@yahoo.de>.
Hi Alexander,
thank you for your response.
You said that the old index files were still in use. That means Linux does
not *really* delete them until Solr frees its locks from it, which happens
while reloading?
Thank you for sharing your experiences!
Kind regards,
Em
Alexander Kanarsky wrote:
>
> Em,
>
> yes, you can replace the index (get the new one into a separate folder
> like index.new and then rename it to the index folder) outside the
> Solr, then just do the http call to reload the core.
>
> Note that the old index files may still be in use (continue to serve
> the queries while reloading), even if the old index folder is deleted
> - that is on Linux filesystems, not sure about NTFS.
> That means the space on disk will be freed only when the old files are
> not referenced by Solr searcher any longer.
>
> -Alexander
>
> On Sat, Jan 22, 2011 at 1:51 PM, Em <ma...@yahoo.de> wrote:
>>
>> Hi Erick,
>>
>> thanks for your response.
>>
>> Yes, it's really not that easy.
>>
>> However, the target is to avoid any kind of master-slave-setup.
>>
>> The most recent idea i got is to create a new core with a data-dir
>> pointing
>> to an already existing directory with a fully optimized index.
>>
>> Regards,
>> Em
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2310709.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2312778.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multicore Relaod Theoretical Question
Posted by Alexander Kanarsky <ka...@gmail.com>.
Em,
yes, you can replace the index (get the new one into a separate folder
like index.new and then rename it to the index folder) outside the
Solr, then just do the http call to reload the core.
Note that the old index files may still be in use (continue to serve
the queries while reloading), even if the old index folder is deleted
- that is on Linux filesystems, not sure about NTFS.
That means the space on disk will be freed only when the old files are
not referenced by Solr searcher any longer.
-Alexander
On Sat, Jan 22, 2011 at 1:51 PM, Em <ma...@yahoo.de> wrote:
>
> Hi Erick,
>
> thanks for your response.
>
> Yes, it's really not that easy.
>
> However, the target is to avoid any kind of master-slave-setup.
>
> The most recent idea i got is to create a new core with a data-dir pointing
> to an already existing directory with a fully optimized index.
>
> Regards,
> Em
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2310709.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Multicore Relaod Theoretical Question
Posted by Em <ma...@yahoo.de>.
Hi Erick,
thanks for your response.
Yes, it's really not that easy.
However, the target is to avoid any kind of master-slave-setup.
The most recent idea i got is to create a new core with a data-dir pointing
to an already existing directory with a fully optimized index.
Regards,
Em
--
View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2310709.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multicore Relaod Theoretical Question
Posted by Erick Erickson <er...@gmail.com>.
This seems far too complex to me. Why not just optimize on the master
and let replication do all the rest for you?
Best
Erick
On Fri, Jan 21, 2011 at 1:07 PM, Em <ma...@yahoo.de> wrote:
>
> Hi,
>
> are there no experiences or thoughts?
> How would you solve this at Lucene-Level?
>
> Regards
>
>
> Em wrote:
> >
> > Hello list,
> >
> > I got a theoretical question about a Multicore-Situation:
> >
> > I got two cores: active, inactive
> >
> > The active core serves all the queries.
> >
> > The inactive core is the tricky thing:
> > I create an optimized index outside the environment and want to insert
> > that optimized index 1 to 1 into the inactive core, which means replacing
> > everything inside the index-directory.
> > After this is done, I would like to reload the inactive core, so that it
> > is ready for a core-swap and ready for serving queries on top of the new
> > inserted optimized index.
> >
> > Is it possible to handle such a situation?
> >
> > Thank you.
> >
> >
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2303585.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Multicore Relaod Theoretical Question
Posted by Em <ma...@yahoo.de>.
Hi,
are there no experiences or thoughts?
How would you solve this at Lucene-Level?
Regards
Em wrote:
>
> Hello list,
>
> I got a theoretical question about a Multicore-Situation:
>
> I got two cores: active, inactive
>
> The active core serves all the queries.
>
> The inactive core is the tricky thing:
> I create an optimized index outside the environment and want to insert
> that optimized index 1 to 1 into the inactive core, which means replacing
> everything inside the index-directory.
> After this is done, I would like to reload the inactive core, so that it
> is ready for a core-swap and ready for serving queries on top of the new
> inserted optimized index.
>
> Is it possible to handle such a situation?
>
> Thank you.
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Multicore-Relaod-Theoretical-Question-tp2293999p2303585.html
Sent from the Solr - User mailing list archive at Nabble.com.