Delta Replication in SOLR

[mechravi25 <me...@yahoo.co.in>]

We would like know whether it is possible to replicate only a certain
documents from master to slave. More like a Delta Replication process. 

In our application, the master solr instances is used for indexing purpose
and the slave solr is for user search request. Hence the replication has to
happen on regular interval of time. Master solr has around 1.4 million
document(Size : 2.7 GB) and it takes more than 900 seconds for replication.
Even if we update few documents in the master, we have to replicate that to
the slave to make the Slave in sync with master, in this process its taking
too much. 

We have a field in the master SOLR which will denote the last added or
updated time (<field name="lastindexedtimestamp" type="date" indexed="false"
stored="true" default="NOW" multiValued="false"/>), so we thought whether we
can replicate the documents from master which were added/updated after the
last Replication time of slave instance which will be available in
replication.properties file. We don’t want all the documents from master to
be replicated to slave. The ultimate purpose is to reduce the time taken for
replication. 

Thanks in advance. Any pointers would be of great help. 


--
View this message in context: http://lucene.472066.n3.nabble.com/Delta-Replication-in-SOLR-tp3587745p3587745.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Delta Replication in SOLR

[Walter Underwood <wu...@wunderwood.org>]

On Dec 14, 2011, at 9:58 PM, mechravi25 wrote:

> We would like know whether it is possible to replicate only a certain
> documents from master to slave. More like a Delta Replication process. 

No, it is not.

wunder
--
Walter Underwood
wunder@wunderwood.org

Re: Delta Replication in SOLR

[Bob Stewart <bs...@gmail.com>]

Replication only copies new segment files so unless you are optimizing on commit it will not copy entire index.  Make sure you do not optimize your index.  Optimizing merges to a single segment and is not necessary.  When new docs are added new small segment files are created so typical replication will only copy a few small segments from master to slave.

On Dec 15, 2011, at 12:58 AM, mechravi25 <me...@yahoo.co.in> wrote:

> We would like know whether it is possible to replicate only a certain
> documents from master to slave. More like a Delta Replication process. 
> 
> In our application, the master solr instances is used for indexing purpose
> and the slave solr is for user search request. Hence the replication has to
> happen on regular interval of time. Master solr has around 1.4 million
> document(Size : 2.7 GB) and it takes more than 900 seconds for replication.
> Even if we update few documents in the master, we have to replicate that to
> the slave to make the Slave in sync with master, in this process its taking
> too much. 
> 
> We have a field in the master SOLR which will denote the last added or
> updated time (<field name="lastindexedtimestamp" type="date" indexed="false"
> stored="true" default="NOW" multiValued="false"/>), so we thought whether we
> can replicate the documents from master which were added/updated after the
> last Replication time of slave instance which will be available in
> replication.properties file. We don’t want all the documents from master to
> be replicated to slave. The ultimate purpose is to reduce the time taken for
> replication. 
> 
> Thanks in advance. Any pointers would be of great help. 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Delta-Replication-in-SOLR-tp3587745p3587745.html
> Sent from the Solr - User mailing list archive at Nabble.com.