You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Isart Montane <is...@gmail.com> on 2017/06/06 08:51:03 UTC
Slow inserting with SolrCloud when increasing replicas
Hello,
We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
problem we are seeing is a huge drop on writes when the number of replicas
increase.
When we index (using DIH and batches) a collection with no replicas, we are
able to index at 1800 inserts/sec. That number decreases to 1200 with 1
replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
worst when more replicas are added.
We've been reading about it and it seems that the `replicationFactor` plays
a big role on that, but we've got it set to 1, so I'm not sure why it keeps
decreasing when more replicas are added. In fact, we don't need the data to
be replicated in real time (we can even afford minutes of delay), but I've
been unable to find how to tune that.
Has anyone experienced a similar behaviour? is there any way to increase
the indexing performance when using SolrCloud?
We've seen posts about people having +100 replicas, so my feeling is that
there's something to tune that we are not doing.
Thanks
Isart Montane Mogas
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Susheel Kumar <su...@gmail.com>.
There is some indexing performance penalty is by adding a replica but then
onwards adding more replica(s) shouldn't have much difference. So
something very wrong. Can you watch your logs when indexing happening and
see what's going when gets replicated etc.
Thanks,
Susheel
On Tue, Jun 6, 2017 at 9:28 AM, Isart Montane <is...@gmail.com>
wrote:
> Hi Sisheel!
>
> We are using Solr 6.5.
>
> We've already looked at Issue 7333, but none of the params seem to change
> the behaviour.
>
> Also, I'm not sure having more parallelism will improve performance since
> the problem seems to be related to replication. It looks like the writes
> need to get to all the replicas before the indexing can continue with the
> next batch
>
> Isart
>
>
>
> On Tue, Jun 6, 2017 at 2:31 PM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Which version of Solr are you using. See
> >
> > https://lucidworks.com/2015/06/10/indexing-performance-solr-
> > 5-2-now-twice-fast/
> >
> >
> > https://issues.apache.org/jira/browse/SOLR-7333
> >
> > Also would suggest to index using SolrJ with parallelism (multiple
> threads
> > and/or machines) to increase indexing thru-put further.
> >
> > On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane <is...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> > > problem we are seeing is a huge drop on writes when the number of
> > replicas
> > > increase.
> > >
> > > When we index (using DIH and batches) a collection with no replicas, we
> > are
> > > able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> > > replica, 800 with 2 replicas and 400 with 3 replicas and it keeps
> getting
> > > worst when more replicas are added.
> > >
> > > We've been reading about it and it seems that the `replicationFactor`
> > plays
> > > a big role on that, but we've got it set to 1, so I'm not sure why it
> > keeps
> > > decreasing when more replicas are added. In fact, we don't need the
> data
> > to
> > > be replicated in real time (we can even afford minutes of delay), but
> > I've
> > > been unable to find how to tune that.
> > >
> > > Has anyone experienced a similar behaviour? is there any way to
> increase
> > > the indexing performance when using SolrCloud?
> > >
> > > We've seen posts about people having +100 replicas, so my feeling is
> that
> > > there's something to tune that we are not doing.
> > >
> > > Thanks
> > >
> > >
> > > Isart Montane Mogas
> > >
> >
>
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Isart Montane <is...@gmail.com>.
Hi Sisheel!
We are using Solr 6.5.
We've already looked at Issue 7333, but none of the params seem to change
the behaviour.
Also, I'm not sure having more parallelism will improve performance since
the problem seems to be related to replication. It looks like the writes
need to get to all the replicas before the indexing can continue with the
next batch
Isart
On Tue, Jun 6, 2017 at 2:31 PM, Susheel Kumar <su...@gmail.com> wrote:
> Which version of Solr are you using. See
>
> https://lucidworks.com/2015/06/10/indexing-performance-solr-
> 5-2-now-twice-fast/
>
>
> https://issues.apache.org/jira/browse/SOLR-7333
>
> Also would suggest to index using SolrJ with parallelism (multiple threads
> and/or machines) to increase indexing thru-put further.
>
> On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane <is...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> > problem we are seeing is a huge drop on writes when the number of
> replicas
> > increase.
> >
> > When we index (using DIH and batches) a collection with no replicas, we
> are
> > able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> > replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
> > worst when more replicas are added.
> >
> > We've been reading about it and it seems that the `replicationFactor`
> plays
> > a big role on that, but we've got it set to 1, so I'm not sure why it
> keeps
> > decreasing when more replicas are added. In fact, we don't need the data
> to
> > be replicated in real time (we can even afford minutes of delay), but
> I've
> > been unable to find how to tune that.
> >
> > Has anyone experienced a similar behaviour? is there any way to increase
> > the indexing performance when using SolrCloud?
> >
> > We've seen posts about people having +100 replicas, so my feeling is that
> > there's something to tune that we are not doing.
> >
> > Thanks
> >
> >
> > Isart Montane Mogas
> >
>
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Susheel Kumar <su...@gmail.com>.
Which version of Solr are you using. See
https://lucidworks.com/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/
https://issues.apache.org/jira/browse/SOLR-7333
Also would suggest to index using SolrJ with parallelism (multiple threads
and/or machines) to increase indexing thru-put further.
On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane <is...@gmail.com>
wrote:
> Hello,
>
> We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> problem we are seeing is a huge drop on writes when the number of replicas
> increase.
>
> When we index (using DIH and batches) a collection with no replicas, we are
> able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
> worst when more replicas are added.
>
> We've been reading about it and it seems that the `replicationFactor` plays
> a big role on that, but we've got it set to 1, so I'm not sure why it keeps
> decreasing when more replicas are added. In fact, we don't need the data to
> be replicated in real time (we can even afford minutes of delay), but I've
> been unable to find how to tune that.
>
> Has anyone experienced a similar behaviour? is there any way to increase
> the indexing performance when using SolrCloud?
>
> We've seen posts about people having +100 replicas, so my feeling is that
> there's something to tune that we are not doing.
>
> Thanks
>
>
> Isart Montane Mogas
>
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Susheel Kumar <su...@gmail.com>.
Does 50K batch size is what are you using to ingest into Solr? If that's
the case it may be too high and you may want to start with 100-1000 batch
size depending on your document size and gradually increase until it starts
degrading the performance.
On Wed, Jun 7, 2017 at 5:51 AM, Isart Montane <is...@gmail.com>
wrote:
> Hi,
>
> The cluster is running on EC2 using 5x r3.xlarge instances and disks are
> 1TB gp2 EBS.
>
> I will try to get the logs that Susheel requested but it's not an easy
> task.
>
> When indexing there's very few IO.
>
> Solr is started with the following flags:
> ```
> /usr/lib/jvm/java-8-oracle/bin/java
> -server
> -Xms15340m
> -Xmx15340m
> -XX:NewRatio=3
> -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -XX:CompressedClassSpaceSize=250m
> -verbose:gc
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/data/solr/logs/solr_gc.log
> -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=9
> -XX:GCLogFileSize=20M
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=18983
> -Dcom.sun.management.jmxremote.rmi.port=18983
> -DzkClientTimeout=15000
> -DzkHost=zk1,zk2,zk3
> -Dsolr.log.dir=/data/solr/logs
> -Djetty.port=8983
> -DSTOP.PORT=7983
> -DSTOP.KEY=solrrocks
> -Duser.timezone=UTC
> -Djetty.home=/home/solr/solr/server
> -Dsolr.solr.home=/data/solr/data
> -Dsolr.install.dir=/home/solr/solr
> -Dlog4j.configuration=file:/data/solr/log4j.properties
> -Xss256k
> -Dsolr.log.muteconsole
> -XX:OnOutOfMemoryError=/home/solr/solr/bin/oom_solr.sh 8983
> /data/solr/logs
> -jar start.jar
> --module=http
> ```
>
> Not sure if it's related, but when the batches get replicated on the
> replicas they don't seem to respect the batch size on the primary.
>
> That's the insert on the primary (batch is 50k)
> ```
> 2017-06-07 09:46:00.629 INFO (qtp592179046-260) [c:collection1 s:shard1
> r:core_node17 x:collection1_shard1_replica6] o.a.s.h.d.DocBuilder Import
> completed successfully
> 2017-06-07 09:46:00.638 INFO (qtp592179046-260) [c:collection1 s:shard1
> r:core_node17 x:collection1_shard1_replica6] o.a.s.h.d.DocBuilder Time
> taken = 0:0:27.717
> 2017-06-07 09:46:00.655 INFO (qtp592179046-260) [c:collection1 s:shard1
> r:core_node17 x:collection1_shard1_replica6]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica6]
> webapp=/solr path=/dataimport
> params={optimize=false&startId=489247153&synchronous=
> true&limit=50000&commit=false&clean=false&command=full-
> import&entity=instagram-users-incremental}{add=[489247178,
> 489247179, 489247191, 489247238, 489247256, 489247260, 489247279,
> 489247325, 489247368, 489247369, ... (50000 adds)]} 0 27743
> ```
>
> And that get's replicated to the replicas as many rows like that. I thought
> that replicating batches of 50k rows to batches of 10-20 rows might be a
> problem, but I couldn't find a way to tune that behaviour (and I'm not sure
> there's one)
> ```
> 2017-06-07 09:45:23.640 INFO (qtp592179046-1808) [c:collection1 s:shard1
> r:core_node13 x:collection1_shard1_replica3]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica3]
> webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
> http://10.0.0.159:8983/solr/collection1_shard1_replica2/&
> wt=javabin&version=2}{add=[488928327
> (1569538675702759424), 488928344 (1569538675703808000), 488928391
> (1569538675703808001), 488928406 (1569538675704856576), 488928418
> (1569538675704856577), 488928451 (1569538675705905152), 488928456
> (1569538675706953728), 488928495 (1569538675706953729), 488928538
> (1569538675708002304), 488928548 (1569538675708002305), ... (15 adds)]} 0 1
> 2017-06-07 09:45:23.671 INFO (qtp592179046-1832) [c:collection1 s:shard2
> r:core_node14 x:collection1_shard2_replica2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
> webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
> http://10.0.0.159:8983/solr/collection1_shard2_replica1/&
> wt=javabin&version=2}{add=[488928306
> (1569538675703808000), 488928329 (1569538675717439488), 488928331
> (1569538675718488064), 488928332 (1569538675719536640), 488928378
> (1569538675734216704), 488928383 (1569538675735265280), 488928399
> (1569538675735265281), 488928426 (1569538675736313856), 488928438
> (1569538675742605312), 488928471 (1569538675743653888), ... (13 adds)]} 0 1
> 2017-06-07 09:45:23.686 INFO (qtp592179046-1811) [c:collection1 s:shard1
> r:core_node13 x:collection1_shard1_replica3]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica3]
> webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
> http://10.0.0.159:8983/solr/collection1_shard1_replica2/&
> wt=javabin&version=2}{add=[488928827
> (1569538675750993920), 488928833 (1569538675753091072), 488928842
> (1569538675754139648), 488928888 (1569538675754139649), 488928914
> (1569538675755188224), 488928953 (1569538675755188225), 488928958
> (1569538675756236800), 488928969 (1569538675756236801), 488928977
> (1569538675757285376), 488928996 (1569538675757285377), ... (15 adds)]} 0 1
> 2017-06-07 09:45:23.706 INFO (qtp592179046-1861) [c:collection1 s:shard2
> r:core_node14 x:collection1_shard2_replica2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
> webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
> http://10.0.0.159:8983/solr/collection1_shard2_replica1/&
> wt=javabin&version=2}{add=[488929020
> (1569538675758333952), 488929023 (1569538675761479680), 488929027
> (1569538675762528256), 488929032 (1569538675763576832), 488929035
> (1569538675763576833), 488929046 (1569538675764625408), 488929051
> (1569538675764625409), 488929131 (1569538675765673984), 488929141
> (1569538675766722560), 488929145 (1569538675766722561), ... (21 adds)]} 0
> 18
> 2017-06-07 09:45:25.213 INFO (qtp592179046-1845) [c:collection1 s:shard2
> r:core_node14 x:collection1_shard2_replica2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
> webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
> http://10.0.0.159:8983/solr/collection1_shard2_replica1/&
> wt=javabin&version=2}{add=[488929441
> (1569538675793985536), 488929535 (1569538675795034112), 488929540
> (1569538675795034113), 488929560 (1569538675796082688), 488929597
> (1569538675796082689), 488929639 (1569538677352169472), 488929684
> (1569538677352169473), 488929713 (1569538677353218048), 488929777
> (1569538677353218049), 488929791 (1569538677353218050), ... (19 adds)]} 0
> 9```
>
> On Wed, Jun 7, 2017 at 10:00 AM, Toke Eskildsen <to...@kb.dk> wrote:
>
> > On Tue, 2017-06-06 at 10:51 +0200, Isart Montane wrote:
> > > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each.
> > > The problem we are seeing is a huge drop on writes when the number of
> > > replicas increase.
> > >
> > > When we index (using DIH and batches) a collection with no replicas,
> > > we are able to index at 1800 inserts/sec. That number decreases to
> > > 1200 with 1 replica, 800 with 2 replicas and 400 with 3 replicas and
> > > it keeps getting worst when more replicas are added.
> >
> > That is, as Susheel says, not expected behaviour. If you are running
> > everything on a single physical machine that could be an explanation.
> > What is your hardware-setup?
> > --
> > Toke Eskildsen, Royal Danish Library
> >
>
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Isart Montane <is...@gmail.com>.
Hi,
The cluster is running on EC2 using 5x r3.xlarge instances and disks are
1TB gp2 EBS.
I will try to get the logs that Susheel requested but it's not an easy task.
When indexing there's very few IO.
Solr is started with the following flags:
```
/usr/lib/jvm/java-8-oracle/bin/java
-server
-Xms15340m
-Xmx15340m
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4
-XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:CompressedClassSpaceSize=250m
-verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xloggc:/data/solr/logs/solr_gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=9
-XX:GCLogFileSize=20M
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=18983
-Dcom.sun.management.jmxremote.rmi.port=18983
-DzkClientTimeout=15000
-DzkHost=zk1,zk2,zk3
-Dsolr.log.dir=/data/solr/logs
-Djetty.port=8983
-DSTOP.PORT=7983
-DSTOP.KEY=solrrocks
-Duser.timezone=UTC
-Djetty.home=/home/solr/solr/server
-Dsolr.solr.home=/data/solr/data
-Dsolr.install.dir=/home/solr/solr
-Dlog4j.configuration=file:/data/solr/log4j.properties
-Xss256k
-Dsolr.log.muteconsole
-XX:OnOutOfMemoryError=/home/solr/solr/bin/oom_solr.sh 8983
/data/solr/logs
-jar start.jar
--module=http
```
Not sure if it's related, but when the batches get replicated on the
replicas they don't seem to respect the batch size on the primary.
That's the insert on the primary (batch is 50k)
```
2017-06-07 09:46:00.629 INFO (qtp592179046-260) [c:collection1 s:shard1
r:core_node17 x:collection1_shard1_replica6] o.a.s.h.d.DocBuilder Import
completed successfully
2017-06-07 09:46:00.638 INFO (qtp592179046-260) [c:collection1 s:shard1
r:core_node17 x:collection1_shard1_replica6] o.a.s.h.d.DocBuilder Time
taken = 0:0:27.717
2017-06-07 09:46:00.655 INFO (qtp592179046-260) [c:collection1 s:shard1
r:core_node17 x:collection1_shard1_replica6]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica6]
webapp=/solr path=/dataimport
params={optimize=false&startId=489247153&synchronous=true&limit=50000&commit=false&clean=false&command=full-import&entity=instagram-users-incremental}{add=[489247178,
489247179, 489247191, 489247238, 489247256, 489247260, 489247279,
489247325, 489247368, 489247369, ... (50000 adds)]} 0 27743
```
And that get's replicated to the replicas as many rows like that. I thought
that replicating batches of 50k rows to batches of 10-20 rows might be a
problem, but I couldn't find a way to tune that behaviour (and I'm not sure
there's one)
```
2017-06-07 09:45:23.640 INFO (qtp592179046-1808) [c:collection1 s:shard1
r:core_node13 x:collection1_shard1_replica3]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica3]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://10.0.0.159:8983/solr/collection1_shard1_replica2/&wt=javabin&version=2}{add=[488928327
(1569538675702759424), 488928344 (1569538675703808000), 488928391
(1569538675703808001), 488928406 (1569538675704856576), 488928418
(1569538675704856577), 488928451 (1569538675705905152), 488928456
(1569538675706953728), 488928495 (1569538675706953729), 488928538
(1569538675708002304), 488928548 (1569538675708002305), ... (15 adds)]} 0 1
2017-06-07 09:45:23.671 INFO (qtp592179046-1832) [c:collection1 s:shard2
r:core_node14 x:collection1_shard2_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://10.0.0.159:8983/solr/collection1_shard2_replica1/&wt=javabin&version=2}{add=[488928306
(1569538675703808000), 488928329 (1569538675717439488), 488928331
(1569538675718488064), 488928332 (1569538675719536640), 488928378
(1569538675734216704), 488928383 (1569538675735265280), 488928399
(1569538675735265281), 488928426 (1569538675736313856), 488928438
(1569538675742605312), 488928471 (1569538675743653888), ... (13 adds)]} 0 1
2017-06-07 09:45:23.686 INFO (qtp592179046-1811) [c:collection1 s:shard1
r:core_node13 x:collection1_shard1_replica3]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica3]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://10.0.0.159:8983/solr/collection1_shard1_replica2/&wt=javabin&version=2}{add=[488928827
(1569538675750993920), 488928833 (1569538675753091072), 488928842
(1569538675754139648), 488928888 (1569538675754139649), 488928914
(1569538675755188224), 488928953 (1569538675755188225), 488928958
(1569538675756236800), 488928969 (1569538675756236801), 488928977
(1569538675757285376), 488928996 (1569538675757285377), ... (15 adds)]} 0 1
2017-06-07 09:45:23.706 INFO (qtp592179046-1861) [c:collection1 s:shard2
r:core_node14 x:collection1_shard2_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://10.0.0.159:8983/solr/collection1_shard2_replica1/&wt=javabin&version=2}{add=[488929020
(1569538675758333952), 488929023 (1569538675761479680), 488929027
(1569538675762528256), 488929032 (1569538675763576832), 488929035
(1569538675763576833), 488929046 (1569538675764625408), 488929051
(1569538675764625409), 488929131 (1569538675765673984), 488929141
(1569538675766722560), 488929145 (1569538675766722561), ... (21 adds)]} 0 18
2017-06-07 09:45:25.213 INFO (qtp592179046-1845) [c:collection1 s:shard2
r:core_node14 x:collection1_shard2_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://10.0.0.159:8983/solr/collection1_shard2_replica1/&wt=javabin&version=2}{add=[488929441
(1569538675793985536), 488929535 (1569538675795034112), 488929540
(1569538675795034113), 488929560 (1569538675796082688), 488929597
(1569538675796082689), 488929639 (1569538677352169472), 488929684
(1569538677352169473), 488929713 (1569538677353218048), 488929777
(1569538677353218049), 488929791 (1569538677353218050), ... (19 adds)]} 0
9```
On Wed, Jun 7, 2017 at 10:00 AM, Toke Eskildsen <to...@kb.dk> wrote:
> On Tue, 2017-06-06 at 10:51 +0200, Isart Montane wrote:
> > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each.
> > The problem we are seeing is a huge drop on writes when the number of
> > replicas increase.
> >
> > When we index (using DIH and batches) a collection with no replicas,
> > we are able to index at 1800 inserts/sec. That number decreases to
> > 1200 with 1 replica, 800 with 2 replicas and 400 with 3 replicas and
> > it keeps getting worst when more replicas are added.
>
> That is, as Susheel says, not expected behaviour. If you are running
> everything on a single physical machine that could be an explanation.
> What is your hardware-setup?
> --
> Toke Eskildsen, Royal Danish Library
>
Re: Slow inserting with SolrCloud when increasing replicas
Posted by Toke Eskildsen <to...@kb.dk>.
On Tue, 2017-06-06 at 10:51 +0200, Isart Montane wrote:
> We are using SolrCloud with 5 nodes, 2 collections, 2 shards each.
> The problem we are seeing is a huge drop on writes when the number of
> replicas increase.
>
> When we index (using DIH and batches) a collection with no replicas,
> we are able to index at 1800 inserts/sec. That number decreases to
> 1200 with 1 replica, 800 with 2 replicas and 400 with 3 replicas and
> it keeps getting worst when more replicas are added.
That is, as Susheel says, not expected behaviour. If you are running
everything on a single physical machine that could be an explanation.
What is your hardware-setup?
--
Toke Eskildsen, Royal Danish Library