You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Willie Wong <Wi...@METALOGIC-INC.COM> on 2008/07/08 16:25:42 UTC

Automated Index Creation

Hi,

Sorry if this question sounds daft.... but I was wondering if there was 
anything built into Solr that allows you to automate the creation of new 
indexes once they reach a certain size or point in time.  I looked briefly 
at the documentation on CollectionDestribution, but it seems more geared 
to towards replicatting to other production servers...    I'm looking for 
something that is more along the lines of archiving indexes for later 
use...


Thanks,

Willie


Re: Automated Index Creation

Posted by Norberto Meijome <fr...@meijome.net>.
On Wed, 9 Jul 2008 08:48:35 +0530
"Shalin Shekhar Mangar" <sh...@gmail.com> wrote:

> Yes, SOLR-350 added that capability. Look at
> http://wiki.apache.org/solr/MultiCore for details.

ahh loving SOLR more every day :P

thx

_________________________
{Beto|Norberto|Numard} Meijome

I used to hate weddings; all the Grandmas would poke me and
say, "You're next sonny!" They stopped doing that when i
started to do it to them at funerals.

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

Re: Automated Index Creation

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yes, SOLR-350 added that capability. Look at
http://wiki.apache.org/solr/MultiCore for details.

On Wed, Jul 9, 2008 at 6:26 AM, Norberto Meijome <fr...@meijome.net>
wrote:

> On Tue, 8 Jul 2008 12:05:45 -0400
> Willie Wong <Wi...@METALOGIC-INC.COM> wrote:
>
> > I think the snapshooter will work fine for creating the indexes and then
> I
> > can use the multicore capabilities to make them available to users....
> one
> > final question though, after snapshot has been created is there a way to
> > totally clear out the contents in the master index - or have solr
> recreate
> > the data directory?
>
> yup. But you would have to have predefined cores where to publish the data
> to... would be interesting ( useful ?? ) if cores could be dynamically
> created
> and added/registered to a running instance of SOLR. Is that supported
> already?
>
> B
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Ask not what's inside your head, but what your head's inside of."
>   J. J. Gibson
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet.
> Reading disclaimers makes you go blind. Writing them is worse. You have
> been
> Warned.
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: Automated Index Creation

Posted by Norberto Meijome <fr...@meijome.net>.
On Tue, 8 Jul 2008 12:05:45 -0400
Willie Wong <Wi...@METALOGIC-INC.COM> wrote:

> I think the snapshooter will work fine for creating the indexes and then I 
> can use the multicore capabilities to make them available to users.... one 
> final question though, after snapshot has been created is there a way to 
> totally clear out the contents in the master index - or have solr recreate 
> the data directory?

yup. But you would have to have predefined cores where to publish the data
to... would be interesting ( useful ?? ) if cores could be dynamically created
and added/registered to a running instance of SOLR. Is that supported already?

B
_________________________
{Beto|Norberto|Numard} Meijome

"Ask not what's inside your head, but what your head's inside of."
   J. J. Gibson

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.

Re: Automated Index Creation

Posted by Ryan McKinley <ry...@gmail.com>.
<delete><query>*:*</query></delete>

will wipe all data in the index...


On Jul 8, 2008, at 12:05 PM, Willie Wong wrote:
> Thanks Sahlin and Ryan for your posts...
>
> I think the snapshooter will work fine for creating the indexes and  
> then I
> can use the multicore capabilities to make them available to  
> users.... one
> final question though, after snapshot has been created is there a  
> way to
> totally clear out the contents in the master index - or have solr  
> recreate
> the data directory?
>
>
>
> Thanks,
>
> Willie
>
>
>
>
> Ryan McKinley <ry...@gmail.com>
> 08/07/2008 11:17 AM
> Please respond to
> solr-user@lucene.apache.org
>
>
> To
> solr-user@lucene.apache.org
> cc
>
> Subject
> Re: Automated Index Creation
>
>
>
>
>
>
> re-reading your post...
>
> Shalin is correct, just use the snapshooter script to create a point-
> in-time snapshot of the index.  The multicore stuff will not help with
> this.
>
> ryan
>
>
> On Jul 8, 2008, at 11:09 AM, Shalin Shekhar Mangar wrote:
>> Hi Willie,
>>
>> If you want to have backups (point-in-time snapshots) then you'd need
>> something similar to the snapshooter script used in replication. I
>> believe
>> it creates hard links to files of the current index in a new  
>> directory
>> marked with the timestamp. You can either use snapshooter itself or
>> create
>> your own script by modifying snapshooter to create copies instead of
>> hardlinks if you want. You can use the RunExecutableListener to run
>> your
>> script on every commit or optimize and use the snapshots for backup
>> purposes.
>>
>> On Tue, Jul 8, 2008 at 7:55 PM, Willie Wong
> <Willie_Wong@metalogic-inc.com
>>>
>> wrote:
>>
>>> Hi,
>>>
>>> Sorry if this question sounds daft.... but I was wondering if there
>>> was
>>> anything built into Solr that allows you to automate the creation
>>> of new
>>> indexes once they reach a certain size or point in time.  I looked
>>> briefly
>>> at the documentation on CollectionDestribution, but it seems more
>>> geared
>>> to towards replicatting to other production servers...    I'm
>>> looking for
>>> something that is more along the lines of archiving indexes for  
>>> later
>>> use...
>>>
>>>
>>> Thanks,
>>>
>>> Willie
>>>
>>>
>>
>>
>> -- 
>> Regards,
>> Shalin Shekhar Mangar.
>
>
>


Re: Automated Index Creation

Posted by Willie Wong <Wi...@METALOGIC-INC.COM>.
Thanks Sahlin and Ryan for your posts...

I think the snapshooter will work fine for creating the indexes and then I 
can use the multicore capabilities to make them available to users.... one 
final question though, after snapshot has been created is there a way to 
totally clear out the contents in the master index - or have solr recreate 
the data directory?



Thanks,

Willie




Ryan McKinley <ry...@gmail.com> 
08/07/2008 11:17 AM
Please respond to
solr-user@lucene.apache.org


To
solr-user@lucene.apache.org
cc

Subject
Re: Automated Index Creation






re-reading your post...

Shalin is correct, just use the snapshooter script to create a point- 
in-time snapshot of the index.  The multicore stuff will not help with 
this.

ryan


On Jul 8, 2008, at 11:09 AM, Shalin Shekhar Mangar wrote:
> Hi Willie,
>
> If you want to have backups (point-in-time snapshots) then you'd need
> something similar to the snapshooter script used in replication. I 
> believe
> it creates hard links to files of the current index in a new directory
> marked with the timestamp. You can either use snapshooter itself or 
> create
> your own script by modifying snapshooter to create copies instead of
> hardlinks if you want. You can use the RunExecutableListener to run 
> your
> script on every commit or optimize and use the snapshots for backup
> purposes.
>
> On Tue, Jul 8, 2008 at 7:55 PM, Willie Wong 
<Willie_Wong@metalogic-inc.com 
> >
> wrote:
>
>> Hi,
>>
>> Sorry if this question sounds daft.... but I was wondering if there 
>> was
>> anything built into Solr that allows you to automate the creation 
>> of new
>> indexes once they reach a certain size or point in time.  I looked 
>> briefly
>> at the documentation on CollectionDestribution, but it seems more 
>> geared
>> to towards replicatting to other production servers...    I'm 
>> looking for
>> something that is more along the lines of archiving indexes for later
>> use...
>>
>>
>> Thanks,
>>
>> Willie
>>
>>
>
>
> -- 
> Regards,
> Shalin Shekhar Mangar.




Re: Automated Index Creation

Posted by Ryan McKinley <ry...@gmail.com>.
re-reading your post...

Shalin is correct, just use the snapshooter script to create a point- 
in-time snapshot of the index.  The multicore stuff will not help with  
this.

ryan


On Jul 8, 2008, at 11:09 AM, Shalin Shekhar Mangar wrote:
> Hi Willie,
>
> If you want to have backups (point-in-time snapshots) then you'd need
> something similar to the snapshooter script used in replication. I  
> believe
> it creates hard links to files of the current index in a new directory
> marked with the timestamp. You can either use snapshooter itself or  
> create
> your own script by modifying snapshooter to create copies instead of
> hardlinks if you want. You can use the RunExecutableListener to run  
> your
> script on every commit or optimize and use the snapshots for backup
> purposes.
>
> On Tue, Jul 8, 2008 at 7:55 PM, Willie Wong <Willie_Wong@metalogic-inc.com 
> >
> wrote:
>
>> Hi,
>>
>> Sorry if this question sounds daft.... but I was wondering if there  
>> was
>> anything built into Solr that allows you to automate the creation  
>> of new
>> indexes once they reach a certain size or point in time.  I looked  
>> briefly
>> at the documentation on CollectionDestribution, but it seems more  
>> geared
>> to towards replicatting to other production servers...    I'm  
>> looking for
>> something that is more along the lines of archiving indexes for later
>> use...
>>
>>
>> Thanks,
>>
>> Willie
>>
>>
>
>
> -- 
> Regards,
> Shalin Shekhar Mangar.


Re: Automated Index Creation

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Willie,

If you want to have backups (point-in-time snapshots) then you'd need
something similar to the snapshooter script used in replication. I believe
it creates hard links to files of the current index in a new directory
marked with the timestamp. You can either use snapshooter itself or create
your own script by modifying snapshooter to create copies instead of
hardlinks if you want. You can use the RunExecutableListener to run your
script on every commit or optimize and use the snapshots for backup
purposes.

On Tue, Jul 8, 2008 at 7:55 PM, Willie Wong <Wi...@metalogic-inc.com>
wrote:

> Hi,
>
> Sorry if this question sounds daft.... but I was wondering if there was
> anything built into Solr that allows you to automate the creation of new
> indexes once they reach a certain size or point in time.  I looked briefly
> at the documentation on CollectionDestribution, but it seems more geared
> to towards replicatting to other production servers...    I'm looking for
> something that is more along the lines of archiving indexes for later
> use...
>
>
> Thanks,
>
> Willie
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Automated Index Creation

Posted by Ryan McKinley <ry...@gmail.com>.
nothing to automatically create a new index, but check the multicore  
stuff to see how you could implement this:
http://wiki.apache.org/solr/MultiCore


On Jul 8, 2008, at 10:25 AM, Willie Wong wrote:
> Hi,
>
> Sorry if this question sounds daft.... but I was wondering if there  
> was
> anything built into Solr that allows you to automate the creation of  
> new
> indexes once they reach a certain size or point in time.  I looked  
> briefly
> at the documentation on CollectionDestribution, but it seems more  
> geared
> to towards replicatting to other production servers...    I'm  
> looking for
> something that is more along the lines of archiving indexes for later
> use...
>
>
> Thanks,
>
> Willie
>