near realtime search and dih

["giovanni.bricconi@banzai.it" <gi...@banzai.it>]

I would like to understand if  near realtime search is applicable to my 
configuration, or if I should change the way I load data.

Currently my application uses data import handler to load new documents 
every 15 minutes. This is acceptable, but  it would be interesting to 
bring online some changes within a minute.

Is it possible to configure the DIH to run in soft commit mode? if not, 
is it possible to use the update handler to made changes with soft 
commit and still use the dih to load other changes and commit modifications?

I have a replica of this core, soft commited data gets replicated too or 
should I send soft updates to both the servers?

I have just updated to solr4 alpha.

thanks


Giovanni

Re: near realtime search and dih

[Mark Miller <ma...@gmail.com>]

Not sure what options DiH has in terms of controlling params - but at the least you could add an update proc that added a commitWithin param. commitWithin is a soft commit on Solr 4. 

You could also use autoSoftCommit and set it to n seconds. 

Sent from my iPhone

On Aug 9, 2012, at 6:02 AM, "giovanni.bricconi@banzai.it" <gi...@banzai.it> wrote:

> I would like to understand if  near realtime search is applicable to my configuration, or if I should change the way I load data.
> 
> Currently my application uses data import handler to load new documents every 15 minutes. This is acceptable, but  it would be interesting to bring online some changes within a minute.
> 
> Is it possible to configure the DIH to run in soft commit mode? if not, is it possible to use the update handler to made changes with soft commit and still use the dih to load other changes and commit modifications?
> 
> I have a replica of this core, soft commited data gets replicated too or should I send soft updates to both the servers?
> 
> I have just updated to solr4 alpha.
> 
> thanks
> 
> 
> Giovanni

Re: near realtime search and dih

[Tomás Fernández Löbbe <to...@gmail.com>]

Master-Slave architectures don't get along very well with NRT. One minute
may be achieved if your index is small and you don't have many updates per
minute, but in other case, I would go with Solr Cloud and distributed
indexing (you can run DIH in one of the nodes and every document will be
indexed in both replicas at the same time).
I don't know if you can configure DIH to use soft commits, but you could
use autosoftcommit.
While I was testing DIH + Solr Cloud I had some memory issues, those were
solved with https://issues.apache.org/jira/browse/SOLR-3658 but the fix is
not in the ALPHA, you should get a more recent revision (or wait for the
BETA).

Tomás

On Thu, Aug 9, 2012 at 7:02 AM, giovanni.bricconi@banzai.it <
giovanni.bricconi@banzai.it> wrote:

> I would like to understand if  near realtime search is applicable to my
> configuration, or if I should change the way I load data.
>
> Currently my application uses data import handler to load new documents
> every 15 minutes. This is acceptable, but  it would be interesting to bring
> online some changes within a minute.
>
> Is it possible to configure the DIH to run in soft commit mode? if not, is
> it possible to use the update handler to made changes with soft commit and
> still use the dih to load other changes and commit modifications?
>
> I have a replica of this core, soft commited data gets replicated too or
> should I send soft updates to both the servers?
>
> I have just updated to solr4 alpha.
>
> thanks
>
>
> Giovanni
>