Re: Can we point a Solr server to index directory dynamically at runtime..

[Michael <so...@gmail.com>]

Using a multicore approach, you could send a "create a core named
'core3weeksold' pointing to '/datadirs/3weeksold' " command to a live Solr,
which would spin it up on the fly.  Then you query it, and maybe keep it
spun up until it's not queried for 60 seconds or something, then send a
"remove core 'core3weeksold' " command.
See http://wiki.apache.org/solr/CoreAdmin#CoreAdminHandler .

Michael

On Thu, Sep 24, 2009 at 12:31 AM, Silent Surfer <si...@yahoo.com>wrote:

> Hi,
>
> Is there any way to dynamically point the Solr servers to an index/data
> directories at run time?
>
> We are generating 200 GB worth of index per day and we want to retain the
> index for approximately 1 month. So our idea is to keep the first 1 week of
> index available at anytime for the users i.e have set of Solr servers up and
> running and handle request to get the past 1 week of date.
>
> But when user tries to query data which is older than 7 days old, we want
> to dynamically point the existing Solr instances to the inactive/dormant
> indexes and get the results.
>
> The main intention is to limit the number of Solr Slave instances and there
> by limit the # of Servers required.
>
> If the index directory and Solr instances are tightly coupled, then most of
> the Solr instances are just up and running and may hardly used, as most of
> the users are mainly interested in past 1 week data and not beyond that.
>
> Any thoughts or any other approaches to tackle this would be greatly
> appreciated.
>
> Thanks,
> sS
>
>
>
>
>

Re: Can we point a Solr server to index directory dynamically at runtime..

[Chris Hostetter <ho...@fucit.org>]

: Using a multicore approach, you could send a "create a core named
: 'core3weeksold' pointing to '/datadirs/3weeksold' " command to a live Solr,
: which would spin it up on the fly.  Then you query it, and maybe keep it
: spun up until it's not queried for 60 seconds or something, then send a
: "remove core 'core3weeksold' " command.
: See http://wiki.apache.org/solr/CoreAdmin#CoreAdminHandler .

something that seems implicit in the question is what to do when the 
request spans all of the data ... this is where (in theory) distributed 
searching could help you out.

index each days worth of data into it's own core, that makes it really 
easy to expire the old data (just UNLOAD and delete an entire core once 
it's more then 30 days old) if your user is only searching "current" dta 
then your app can directly query the core containing the most current data 
-- but if they want to query the last week, or last two weeks worth of 
data, you do a distributed request for all of the shards needed to search 
the appropriate amount of data.

Between the ALIAS and SWAP commands it on the CoreAdmin screen it should 
be pretty easy have cores with names like "today","1dayold","2dayold" so 
that your app can configure simple shard params for all the perumations 
you'll need to query.


-Hoss