You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Victor Ruiz <bi...@gmail.com> on 2013/03/11 18:53:07 UTC
Solr replication takes long time
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have 2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
is below 50 queries per second
Replication has been working for long time with any problem, but in the last
weeks the replication cycles started to take long and long time for the big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>LUCENE_40</luceneMatchVersion>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<indexConfig>
<mergeFactor>3</mergeFactor>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="maxCommitsToKeep">10</str>
<str name="maxOptimizedCommitsToKeep">1</str>
<str name="maxCommitAge">6HOUR</str>
</deletionPolicy>
</indexConfig>
<jmx/>
<updateHandler class="solr.DirectUpdateHandler2">
<autoCommit>
<maxDocs>2000</maxDocs>
<maxTime>30000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxTime>500</maxTime>
</autoSoftCommit>
<updateLog>
<str name="dir">${solr.data.dir:}</str>
</updateLog>
</updateHandler>
<query>
<maxBooleanClauses>2048</maxBooleanClauses>
<filterCache
class="solr.FastLRUCache"
size="2048"
initialSize="1024"
autowarmCount="1024"/>
<queryResultCache
class="solr.LRUCache"
size="2048"
initialSize="1024"
autowarmCount="1024"/>
<documentCache
class="solr.LRUCache"
size="2048"
initialSize="1024"
autowarmCount="1024"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>50</queryResultWindowSize>
<queryResultMaxDocsCached>50</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">*:*</str>
<str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str>
<str name="rows">1000</str>
</lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">*:*</str>
<str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str>
<str name="rows">1000</str>
</lst>
</arr>
</listener>
<useColdSearcher>true</useColdSearcher>
<maxWarmingSearchers>4</maxWarmingSearchers>
</query>
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="enable">${enable.master:false}</str>
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="backupAfter">startup</str>
<str
name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt</str>
</lst>
<lst name="slave">
<str name="enable">${enable.slave:false}</str>
<str
name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}</str>
<str name="pollInterval">05:00</str>
<str name="httpBasicAuthUser">${MASTER_HTTP_USER}</str>
<str name="httpBasicAuthPassword">${MASTER_HTTP_PWD}</str>
</lst>
</requestHandler>
<admin>
<defaultQuery>*:*</defaultQuery>
</admin>
</config>
Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
increase it up to 10, with no effect, the replication is always taking so
long., even with a poll time of 2 minutes, when there are only a few megas
to replicate.
Any idea suggestion about what could be the problem?
Thanks in advance,
Victor
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Victor Ruiz <bi...@gmail.com>.
Thanks for your answer Mark. I think I'll try to update to 4.2. I'll keep you
updated.
Anyway, I'd not say that the full index is replicated, I've been monitoring
the replication process in the Solr admin console and there I see that
usually not more than 50-100 files are transferrend, the total size is
rarely greater than 50MB. Is this info trustable?
Victor
Mark Miller-3 wrote
> Okay - yes, 4.0 is a better choice for replication than 4.1.
>
> It almost sounds like you may be replicating the full index rather than
> just changes or something. 4.0 had a couple issues as well - a couple
> things that were discovered while writing stronger tests for 4.2.
>
> 4.2 is spreading onto mirrors now.
>
> - Mark
>
> On Mar 11, 2013, at 2:00 PM, Victor Ruiz <
> bik1979@
> > wrote:
>
>> no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was
>> an
>> issue with the replication, so I decided not to try it for now
>>
>>
>> Mark Miller-3 wrote
>>> Are you using Solr 4.1?
>>>
>>> - Mark
>>>
>>> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
>>
>>> bik1979@
>>
>>> > wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I have a problem with Solr replication. I have 2 solr servers (Solr
>>>> 4.0.0) 1
>>>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
>>>> every server, there are 2 independent instances of solr running (I
>>>> tried
>>>> also multicore config, but having independent instances has for me
>>>> better
>>>> performance), every instance having a differente collection. So, we
>>>> have
>>>> 2
>>>> masters in server 1, and 2 slaves in server 2.
>>>>
>>>> Index size is currently (for the biggest collection) around 17 million
>>>> documents, with a total size near 12 GB. The files transferred every
>>>> replication cycle are typically not more than 100, with a total size
>>>> not
>>>> bigger than 50MB. The other collection is not that big, just around 1
>>>> million docs and not bigger than 2 GB and not a high update ratio. The
>>>> big
>>>> collection has a load around 200 queries per second (MoreLikeThis,
>>>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>>>> it
>>>> is below 50 queries per second
>>>>
>>>> Replication has been working for long time with any problem, but in the
>>>> last
>>>> weeks the replication cycles started to take long and long time for the
>>>> big
>>>> collection, even more than 2 minutes, some times even more. During that
>>>> time, slaves are so overloaded, that many queries are timing out,
>>>> despite
>>>> the timeout in my clients is 30 seconds.
>>>>
>>>> The servers are in same LAN, gigabit ethernet, so the broadband should
>>>> not
>>>> be the bottleneck.
>>>>
>>>> Since the index is receiving frequents updates and deletes (update
>>>> handler
>>>> receives more than 200 request per second for the big collection, but
>>>> not
>>>> more than 5 per second for the small one), I tried to use the
>>>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>>>> replication, but it has no effect.
>>>>
>>>> My solrconfig.xml in the big collection is like that:
>>>>
>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>
>>>>
>>>
> <config>
>>>>
>>>>
>>>
> <luceneMatchVersion>
>>> LUCENE_40
>>>
> </luceneMatchVersion>
>>>>
>>>>
>>>
> <directoryFactory name="DirectoryFactory"
>>
>>>
>>>
>>> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>>>
>>>>
>>>>
>>>
> <indexConfig>
>>>>
>>>
> <mergeFactor>
>>> 3
>>>
> </mergeFactor>
>>>>
>>>>
>>>
> <deletionPolicy class="solr.SolrDeletionPolicy">
>>>>
>>>>
>>>
> <str name="maxCommitsToKeep">
>>> 10
>>>
> </str>
>>>>
>>>
> <str name="maxOptimizedCommitsToKeep">
>>> 1
>>>
> </str>
>>>>
>>>>
>>>
> <str name="maxCommitAge">
>>> 6HOUR
>>>
> </str>
>>>>
>>>>
>>>
> </deletionPolicy>
>>>>
>>>>
>>>
> </indexConfig>
>>>>
>>>>
>>>
> <jmx/>
>>>>
>>>>
>>>
> <updateHandler class="solr.DirectUpdateHandler2">
>>>>
>>>>
>>>
> <autoCommit>
>>>>
>>>
> <maxDocs>
>>> 2000
>>>
> </maxDocs>
>>>>
>>>
> <maxTime>
>>> 30000
>>>
> </maxTime>
>>>>
>>>
> </autoCommit>
>>>>
>>>>
>>>
> <autoSoftCommit>
>>>>
>>>
> <maxTime>
>>> 500
>>>
> </maxTime>
>>>>
>>>
> </autoSoftCommit>
>>>>
>>>>
>>>
> <updateLog>
>>>>
>>>
> <str name="dir">
>>> ${solr.data.dir:}
>>>
> </str>
>>>>
>>>
> </updateLog>
>>>>
>>>>
>>>
> </updateHandler>
>>>>
>>>>
>>>
> <query>
>>>>
>>>
> <maxBooleanClauses>
>>> 2048
>>>
> </maxBooleanClauses>
>>>>
>>>>
>>>
> <filterCache
>>
>>>
>>> class="solr.FastLRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>
> <queryResultCache
>>
>>>
>>> class="solr.LRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>>
>>>
> <documentCache
>>
>>>
>>> class="solr.LRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>
> <enableLazyFieldLoading>
>>> true
>>>
> </enableLazyFieldLoading>
>>>>
>>>>
>>>
> <queryResultWindowSize>
>>> 50
>>>
> </queryResultWindowSize>
>>>>
>>>>
>>>
> <queryResultMaxDocsCached>
>>> 50
>>>
> </queryResultMaxDocsCached>
>>>>
>>>>
>>>
> <listener event="newSearcher" class="solr.QuerySenderListener">
>>>>
>>>
> <arr name="queries">
>>>>
>>>
> <lst>
>>>>
>>>
> <str name="q">
>>> *:*
>>>
> </str>
>>>>
>>>
> <str name="fq">
>>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>>>
> </str>
>>>>
>>>
> <str name="rows">
>>> 1000
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </arr>
>>>>
>>>
> </listener>
>>>>
>>>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
>>>>
>>>
> <arr name="queries">
>>>>
>>>
> <lst>
>>>>
>>>
> <str name="q">
>>> *:*
>>>
> </str>
>>>>
>>>
> <str name="fq">
>>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>>>
> </str>
>>>>
>>>
> <str name="rows">
>>> 1000
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </arr>
>>>>
>>>
> </listener>
>>>>
>>>>
>>>
> <useColdSearcher>
>>> true
>>>
> </useColdSearcher>
>>>>
>>>>
>>>
> <maxWarmingSearchers>
>>> 4
>>>
> </maxWarmingSearchers>
>>>>
>>>
> </query>
>>>>
>>>>
>>>>
>>>>
>>>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>>>
>>>
> <lst name="master">
>>>>
>>>
> <str name="enable">
>>> ${enable.master:false}
>>>
> </str>
>>>>
>>>
> <str name="replicateAfter">
>>> commit
>>>
> </str>
>>>>
>>>
> <str name="replicateAfter">
>>> startup
>>>
> </str>
>>>>
>>>
> <str name="backupAfter">
>>> startup
>>>
> </str>
>>>>
>>>
> <str
>>
>>>
>>>
>>> name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> <lst name="slave">
>>>>
>>>
> <str name="enable">
>>> ${enable.slave:false}
>>>
> </str>
>>>>
>>>
> <str
>>
>>>
>>> name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}
>>>
> </str>
>>>>
>>>
> <str name="pollInterval">
>>> 05:00
>>>
> </str>
>>>>
>>>
> <str name="httpBasicAuthUser">
>>> ${MASTER_HTTP_USER}
>>>
> </str>
>>>>
>>>
> <str name="httpBasicAuthPassword">
>>> ${MASTER_HTTP_PWD}
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </requestHandler>
>>>>
>>>>
>>>
> <admin>
>>>>
>>>
> <defaultQuery>
>>> *:*
>>>
> </defaultQuery>
>>>>
>>>
> </admin>
>>>>
>>>>
>>>
> </config>
>>>>
>>>> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
>>>> increase it up to 10, with no effect, the replication is always taking
>>>> so
>>>> long., even with a poll time of 2 minutes, when there are only a few
>>>> megas
>>>> to replicate.
>>>>
>>>> Any idea suggestion about what could be the problem?
>>>>
>>>> Thanks in advance,
>>>> Victor
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046392.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046428.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Victor Ruiz <bi...@gmail.com>.
While looking at Solr logs, I found a java.lang.OutOfMemoryError: Java heap
space that was happening 2 times per hour
So I tried to increase the max memory heap assigned to JVM (-Xmx) and since
then the servers are not crashing, even though the replication takes still
long time to complete. But for now, the 2 slaves can handle with no problems
all the queries.
Regards,
Victor
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046993.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Victor Ruiz <bi...@gmail.com>.
<http://lucene.472066.n3.nabble.com/file/n4046958/solr_replication.jpg>
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046958.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Victor Ruiz <bi...@gmail.com>.
After upgrading to 4.2, the problem is not yet solved, in this image you can
see, how slow is the transfer speed. At least, after the update the master
is not blocked during replication
<http://lucene.472066.n3.nabble.com/file/n4046951/solr_replication.jpg>
Any idea?
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046951.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Mark Miller <ma...@gmail.com>.
Okay - yes, 4.0 is a better choice for replication than 4.1.
It almost sounds like you may be replicating the full index rather than just changes or something. 4.0 had a couple issues as well - a couple things that were discovered while writing stronger tests for 4.2.
4.2 is spreading onto mirrors now.
- Mark
On Mar 11, 2013, at 2:00 PM, Victor Ruiz <bi...@gmail.com> wrote:
> no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
> issue with the replication, so I decided not to try it for now
>
>
> Mark Miller-3 wrote
>> Are you using Solr 4.1?
>>
>> - Mark
>>
>> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
>
>> bik1979@
>
>> > wrote:
>>
>>> Hi guys,
>>>
>>> I have a problem with Solr replication. I have 2 solr servers (Solr
>>> 4.0.0) 1
>>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
>>> every server, there are 2 independent instances of solr running (I tried
>>> also multicore config, but having independent instances has for me better
>>> performance), every instance having a differente collection. So, we have
>>> 2
>>> masters in server 1, and 2 slaves in server 2.
>>>
>>> Index size is currently (for the biggest collection) around 17 million
>>> documents, with a total size near 12 GB. The files transferred every
>>> replication cycle are typically not more than 100, with a total size not
>>> bigger than 50MB. The other collection is not that big, just around 1
>>> million docs and not bigger than 2 GB and not a high update ratio. The
>>> big
>>> collection has a load around 200 queries per second (MoreLikeThis,
>>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>>> it
>>> is below 50 queries per second
>>>
>>> Replication has been working for long time with any problem, but in the
>>> last
>>> weeks the replication cycles started to take long and long time for the
>>> big
>>> collection, even more than 2 minutes, some times even more. During that
>>> time, slaves are so overloaded, that many queries are timing out, despite
>>> the timeout in my clients is 30 seconds.
>>>
>>> The servers are in same LAN, gigabit ethernet, so the broadband should
>>> not
>>> be the bottleneck.
>>>
>>> Since the index is receiving frequents updates and deletes (update
>>> handler
>>> receives more than 200 request per second for the big collection, but not
>>> more than 5 per second for the small one), I tried to use the
>>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>>> replication, but it has no effect.
>>>
>>> My solrconfig.xml in the big collection is like that:
>>>
>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>
>>>
>> <config>
>>>
>>>
>> <luceneMatchVersion>
>> LUCENE_40
>> </luceneMatchVersion>
>>>
>>>
>> <directoryFactory name="DirectoryFactory"
>>>
>> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>>
>>>
>>>
>> <indexConfig>
>>>
>> <mergeFactor>
>> 3
>> </mergeFactor>
>>>
>>>
>> <deletionPolicy class="solr.SolrDeletionPolicy">
>>>
>>>
>> <str name="maxCommitsToKeep">
>> 10
>> </str>
>>>
>> <str name="maxOptimizedCommitsToKeep">
>> 1
>> </str>
>>>
>>>
>> <str name="maxCommitAge">
>> 6HOUR
>> </str>
>>>
>>>
>> </deletionPolicy>
>>>
>>>
>> </indexConfig>
>>>
>>>
>> <jmx/>
>>>
>>>
>> <updateHandler class="solr.DirectUpdateHandler2">
>>>
>>>
>> <autoCommit>
>>>
>> <maxDocs>
>> 2000
>> </maxDocs>
>>>
>> <maxTime>
>> 30000
>> </maxTime>
>>>
>> </autoCommit>
>>>
>>>
>> <autoSoftCommit>
>>>
>> <maxTime>
>> 500
>> </maxTime>
>>>
>> </autoSoftCommit>
>>>
>>>
>> <updateLog>
>>>
>> <str name="dir">
>> ${solr.data.dir:}
>> </str>
>>>
>> </updateLog>
>>>
>>>
>> </updateHandler>
>>>
>>>
>> <query>
>>>
>> <maxBooleanClauses>
>> 2048
>> </maxBooleanClauses>
>>>
>>>
>> <filterCache
>>>
>> class="solr.FastLRUCache"
>>> size="2048"
>>> initialSize="1024"
>>> autowarmCount="1024"/>
>>>
>>>
>> <queryResultCache
>>>
>> class="solr.LRUCache"
>>> size="2048"
>>> initialSize="1024"
>>> autowarmCount="1024"/>
>>>
>>>
>>>
>> <documentCache
>>>
>> class="solr.LRUCache"
>>> size="2048"
>>> initialSize="1024"
>>> autowarmCount="1024"/>
>>>
>>>
>> <enableLazyFieldLoading>
>> true
>> </enableLazyFieldLoading>
>>>
>>>
>> <queryResultWindowSize>
>> 50
>> </queryResultWindowSize>
>>>
>>>
>> <queryResultMaxDocsCached>
>> 50
>> </queryResultMaxDocsCached>
>>>
>>>
>> <listener event="newSearcher" class="solr.QuerySenderListener">
>>>
>> <arr name="queries">
>>>
>> <lst>
>>>
>> <str name="q">
>> *:*
>> </str>
>>>
>> <str name="fq">
>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>> </str>
>>>
>> <str name="rows">
>> 1000
>> </str>
>>>
>> </lst>
>>>
>> </arr>
>>>
>> </listener>
>>>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>>>
>> <arr name="queries">
>>>
>> <lst>
>>>
>> <str name="q">
>> *:*
>> </str>
>>>
>> <str name="fq">
>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>> </str>
>>>
>> <str name="rows">
>> 1000
>> </str>
>>>
>> </lst>
>>>
>> </arr>
>>>
>> </listener>
>>>
>>>
>> <useColdSearcher>
>> true
>> </useColdSearcher>
>>>
>>>
>> <maxWarmingSearchers>
>> 4
>> </maxWarmingSearchers>
>>>
>> </query>
>>>
>>>
>>>
>>>
>> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>>
>> <lst name="master">
>>>
>> <str name="enable">
>> ${enable.master:false}
>> </str>
>>>
>> <str name="replicateAfter">
>> commit
>> </str>
>>>
>> <str name="replicateAfter">
>> startup
>> </str>
>>>
>> <str name="backupAfter">
>> startup
>> </str>
>>>
>> <str
>>>
>>
>> name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt
>> </str>
>>>
>> </lst>
>>>
>> <lst name="slave">
>>>
>> <str name="enable">
>> ${enable.slave:false}
>> </str>
>>>
>> <str
>>>
>> name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}
>> </str>
>>>
>> <str name="pollInterval">
>> 05:00
>> </str>
>>>
>> <str name="httpBasicAuthUser">
>> ${MASTER_HTTP_USER}
>> </str>
>>>
>> <str name="httpBasicAuthPassword">
>> ${MASTER_HTTP_PWD}
>> </str>
>>>
>> </lst>
>>>
>> </requestHandler>
>>>
>>>
>> <admin>
>>>
>> <defaultQuery>
>> *:*
>> </defaultQuery>
>>>
>> </admin>
>>>
>>>
>> </config>
>>>
>>> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
>>> increase it up to 10, with no effect, the replication is always taking so
>>> long., even with a poll time of 2 minutes, when there are only a few
>>> megas
>>> to replicate.
>>>
>>> Any idea suggestion about what could be the problem?
>>>
>>> Thanks in advance,
>>> Victor
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046392.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Victor Ruiz <bi...@gmail.com>.
no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
issue with the replication, so I decided not to try it for now
Mark Miller-3 wrote
> Are you using Solr 4.1?
>
> - Mark
>
> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
> bik1979@
> > wrote:
>
>> Hi guys,
>>
>> I have a problem with Solr replication. I have 2 solr servers (Solr
>> 4.0.0) 1
>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
>> every server, there are 2 independent instances of solr running (I tried
>> also multicore config, but having independent instances has for me better
>> performance), every instance having a differente collection. So, we have
>> 2
>> masters in server 1, and 2 slaves in server 2.
>>
>> Index size is currently (for the biggest collection) around 17 million
>> documents, with a total size near 12 GB. The files transferred every
>> replication cycle are typically not more than 100, with a total size not
>> bigger than 50MB. The other collection is not that big, just around 1
>> million docs and not bigger than 2 GB and not a high update ratio. The
>> big
>> collection has a load around 200 queries per second (MoreLikeThis,
>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>> it
>> is below 50 queries per second
>>
>> Replication has been working for long time with any problem, but in the
>> last
>> weeks the replication cycles started to take long and long time for the
>> big
>> collection, even more than 2 minutes, some times even more. During that
>> time, slaves are so overloaded, that many queries are timing out, despite
>> the timeout in my clients is 30 seconds.
>>
>> The servers are in same LAN, gigabit ethernet, so the broadband should
>> not
>> be the bottleneck.
>>
>> Since the index is receiving frequents updates and deletes (update
>> handler
>> receives more than 200 request per second for the big collection, but not
>> more than 5 per second for the small one), I tried to use the
>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>> replication, but it has no effect.
>>
>> My solrconfig.xml in the big collection is like that:
>>
>> <?xml version="1.0" encoding="UTF-8" ?>
>>
>>
> <config>
>>
>>
> <luceneMatchVersion>
> LUCENE_40
> </luceneMatchVersion>
>>
>>
> <directoryFactory name="DirectoryFactory"
>>
> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>
>>
>>
> <indexConfig>
>>
> <mergeFactor>
> 3
> </mergeFactor>
>>
>>
> <deletionPolicy class="solr.SolrDeletionPolicy">
>>
>>
> <str name="maxCommitsToKeep">
> 10
> </str>
>>
> <str name="maxOptimizedCommitsToKeep">
> 1
> </str>
>>
>>
> <str name="maxCommitAge">
> 6HOUR
> </str>
>>
>>
> </deletionPolicy>
>>
>>
> </indexConfig>
>>
>>
> <jmx/>
>>
>>
> <updateHandler class="solr.DirectUpdateHandler2">
>>
>>
> <autoCommit>
>>
> <maxDocs>
> 2000
> </maxDocs>
>>
> <maxTime>
> 30000
> </maxTime>
>>
> </autoCommit>
>>
>>
> <autoSoftCommit>
>>
> <maxTime>
> 500
> </maxTime>
>>
> </autoSoftCommit>
>>
>>
> <updateLog>
>>
> <str name="dir">
> ${solr.data.dir:}
> </str>
>>
> </updateLog>
>>
>>
> </updateHandler>
>>
>>
> <query>
>>
> <maxBooleanClauses>
> 2048
> </maxBooleanClauses>
>>
>>
> <filterCache
>>
> class="solr.FastLRUCache"
>> size="2048"
>> initialSize="1024"
>> autowarmCount="1024"/>
>>
>>
> <queryResultCache
>>
> class="solr.LRUCache"
>> size="2048"
>> initialSize="1024"
>> autowarmCount="1024"/>
>>
>>
>>
> <documentCache
>>
> class="solr.LRUCache"
>> size="2048"
>> initialSize="1024"
>> autowarmCount="1024"/>
>>
>>
> <enableLazyFieldLoading>
> true
> </enableLazyFieldLoading>
>>
>>
> <queryResultWindowSize>
> 50
> </queryResultWindowSize>
>>
>>
> <queryResultMaxDocsCached>
> 50
> </queryResultMaxDocsCached>
>>
>>
> <listener event="newSearcher" class="solr.QuerySenderListener">
>>
> <arr name="queries">
>>
> <lst>
>>
> <str name="q">
> *:*
> </str>
>>
> <str name="fq">
> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
> </str>
>>
> <str name="rows">
> 1000
> </str>
>>
> </lst>
>>
> </arr>
>>
> </listener>
>>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
>>
> <arr name="queries">
>>
> <lst>
>>
> <str name="q">
> *:*
> </str>
>>
> <str name="fq">
> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
> </str>
>>
> <str name="rows">
> 1000
> </str>
>>
> </lst>
>>
> </arr>
>>
> </listener>
>>
>>
> <useColdSearcher>
> true
> </useColdSearcher>
>>
>>
> <maxWarmingSearchers>
> 4
> </maxWarmingSearchers>
>>
> </query>
>>
>>
>>
>>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>
> <lst name="master">
>>
> <str name="enable">
> ${enable.master:false}
> </str>
>>
> <str name="replicateAfter">
> commit
> </str>
>>
> <str name="replicateAfter">
> startup
> </str>
>>
> <str name="backupAfter">
> startup
> </str>
>>
> <str
>>
>
> name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt
> </str>
>>
> </lst>
>>
> <lst name="slave">
>>
> <str name="enable">
> ${enable.slave:false}
> </str>
>>
> <str
>>
> name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}
> </str>
>>
> <str name="pollInterval">
> 05:00
> </str>
>>
> <str name="httpBasicAuthUser">
> ${MASTER_HTTP_USER}
> </str>
>>
> <str name="httpBasicAuthPassword">
> ${MASTER_HTTP_PWD}
> </str>
>>
> </lst>
>>
> </requestHandler>
>>
>>
> <admin>
>>
> <defaultQuery>
> *:*
> </defaultQuery>
>>
> </admin>
>>
>>
> </config>
>>
>> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
>> increase it up to 10, with no effect, the replication is always taking so
>> long., even with a poll time of 2 minutes, when there are only a few
>> megas
>> to replicate.
>>
>> Any idea suggestion about what could be the problem?
>>
>> Thanks in advance,
>> Victor
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046392.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
Posted by Mark Miller <ma...@gmail.com>.
Are you using Solr 4.1?
- Mark
On Mar 11, 2013, at 1:53 PM, Victor Ruiz <bi...@gmail.com> wrote:
> Hi guys,
>
> I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
> every server, there are 2 independent instances of solr running (I tried
> also multicore config, but having independent instances has for me better
> performance), every instance having a differente collection. So, we have 2
> masters in server 1, and 2 slaves in server 2.
>
> Index size is currently (for the biggest collection) around 17 million
> documents, with a total size near 12 GB. The files transferred every
> replication cycle are typically not more than 100, with a total size not
> bigger than 50MB. The other collection is not that big, just around 1
> million docs and not bigger than 2 GB and not a high update ratio. The big
> collection has a load around 200 queries per second (MoreLikeThis,
> RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
> is below 50 queries per second
>
> Replication has been working for long time with any problem, but in the last
> weeks the replication cycles started to take long and long time for the big
> collection, even more than 2 minutes, some times even more. During that
> time, slaves are so overloaded, that many queries are timing out, despite
> the timeout in my clients is 30 seconds.
>
> The servers are in same LAN, gigabit ethernet, so the broadband should not
> be the bottleneck.
>
> Since the index is receiving frequents updates and deletes (update handler
> receives more than 200 request per second for the big collection, but not
> more than 5 per second for the small one), I tried to use the
> maxCommitsToKeep attribute, to ensure that no file was deleted during
> replication, but it has no effect.
>
> My solrconfig.xml in the big collection is like that:
>
> <?xml version="1.0" encoding="UTF-8" ?>
>
> <config>
>
> <luceneMatchVersion>LUCENE_40</luceneMatchVersion>
>
> <directoryFactory name="DirectoryFactory"
> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>
>
> <indexConfig>
> <mergeFactor>3</mergeFactor>
>
> <deletionPolicy class="solr.SolrDeletionPolicy">
>
> <str name="maxCommitsToKeep">10</str>
> <str name="maxOptimizedCommitsToKeep">1</str>
>
> <str name="maxCommitAge">6HOUR</str>
>
> </deletionPolicy>
>
> </indexConfig>
>
> <jmx/>
>
> <updateHandler class="solr.DirectUpdateHandler2">
>
> <autoCommit>
> <maxDocs>2000</maxDocs>
> <maxTime>30000</maxTime>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>500</maxTime>
> </autoSoftCommit>
>
> <updateLog>
> <str name="dir">${solr.data.dir:}</str>
> </updateLog>
>
> </updateHandler>
>
> <query>
> <maxBooleanClauses>2048</maxBooleanClauses>
>
> <filterCache
> class="solr.FastLRUCache"
> size="2048"
> initialSize="1024"
> autowarmCount="1024"/>
>
> <queryResultCache
> class="solr.LRUCache"
> size="2048"
> initialSize="1024"
> autowarmCount="1024"/>
>
>
> <documentCache
> class="solr.LRUCache"
> size="2048"
> initialSize="1024"
> autowarmCount="1024"/>
>
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>
> <queryResultWindowSize>50</queryResultWindowSize>
>
> <queryResultMaxDocsCached>50</queryResultMaxDocsCached>
>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst>
> <str name="q">*:*</str>
> <str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str>
> <str name="rows">1000</str>
> </lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst>
> <str name="q">*:*</str>
> <str name="fq">date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]</str>
> <str name="rows">1000</str>
> </lst>
> </arr>
> </listener>
>
> <useColdSearcher>true</useColdSearcher>
>
> <maxWarmingSearchers>4</maxWarmingSearchers>
> </query>
>
>
>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
> <lst name="master">
> <str name="enable">${enable.master:false}</str>
> <str name="replicateAfter">commit</str>
> <str name="replicateAfter">startup</str>
> <str name="backupAfter">startup</str>
> <str
> name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt</str>
> </lst>
> <lst name="slave">
> <str name="enable">${enable.slave:false}</str>
> <str
> name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}</str>
> <str name="pollInterval">05:00</str>
> <str name="httpBasicAuthUser">${MASTER_HTTP_USER}</str>
> <str name="httpBasicAuthPassword">${MASTER_HTTP_PWD}</str>
> </lst>
> </requestHandler>
>
> <admin>
> <defaultQuery>*:*</defaultQuery>
> </admin>
>
> </config>
>
> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
> increase it up to 10, with no effect, the replication is always taking so
> long., even with a poll time of 2 minutes, when there are only a few megas
> to replicate.
>
> Any idea suggestion about what could be the problem?
>
> Thanks in advance,
> Victor
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
> Sent from the Solr - User mailing list archive at Nabble.com.