You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ankit Shah <as...@gmail.com> on 2020/09/03 16:18:46 UTC
Query performance degrades when TLOG replica
We have the following setup , solr 7.7.2 with 1 TLOG Leader & 1 TLOG
replica with a single shard. We have about 34.5 million documents with an
approximate index size of 600GB. I have noticed a degraded query
performance whenever the replica is trying to (guessing here) sync or
perform actual replication. To test this, I fire a very basic query using
solrj client & the query comes back right away, but whenever the
replication is trying to see how far behind it is by comparing the
generation ids the same queries take longer. In production we do not make
these simple queries, but rather complex queries with filter queries &
sorting. These queries take too long as compared to our previous
(standalone solr 6.1.0)
Any help here is appreciated
20-09-02 16:35:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:35:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:00 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:00 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:30 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458909
status=0 QTime=0
*2020-09-02 16:37:01* INFO [db_shard1_replica_t3] webapp=/solr
path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2}
hits=34458909 status=0 QTime=*1011*
*2020-09-02 16:37:01* INFO [db_shard1_replica_t3] webapp=/solr
path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2}
hits=34458909 status=0 QTime=*758*
*2020-09-02 16:37:32* INFO [db_shard1_replica_t3] webapp=/solr
path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2}
hits=34458957 status=0 QTime=*1077*
*2020-09-02 16:37:32* INFO [db_shard1_replica_t3] webapp=/solr
path=/select params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2}
hits=34458957 status=0 QTime=*1081*
2020-09-02 16:38:02 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957
status=0 QTime=*668*
2020-09-02 16:38:03 INFO [db_shard1_replica_t3] webapp=/solr path=/select
params={q=*:*&fl=id&sort=id+desc&rows=1&wt=xml&version=2.2} hits=34458957
status=0 QTime=*1001*
*2020-09-02 16:37:01* INFO Master's generation: 263116
*2020-09-02 16:37:01* INFO Master's version: 1599064577322
*2020-09-02 16:37:01* INFO Slave's generation: 263116
*2020-09-02 16:37:01* INFO Slave's version: 1599064577322
*2020-09-02 16:37:01* INFO Slave in sync with master.
2020-09-02 16:37:02 INFO Master's generation: 104189
2020-09-02 16:37:02 INFO Master's version: 1599064620532
2020-09-02 16:37:02 INFO Slave's generation: 104188
2020-09-02 16:37:02 INFO Slave's version: 1599064560341
2020-09-02 16:37:02 INFO Starting replication process
2020-09-02 16:37:02 INFO Number of files in latest index in master: 1010
2020-09-02 16:37:02 INFO Starting download (fullCopy=false) to
NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2020-09-02 16:37:02 INFO Bytes downloaded: 837587, Bytes skipped
downloading: 0
2020-09-02 16:37:02 INFO Total time taken for download
(fullCopy=false,bytesDownloaded=837587) : 0 secs (null bytes/sec) to
NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2020-09-02 16:37:03 INFO New IndexWriter is ready to be used.
2020-09-02 16:37:03 INFO Master's generation: 124002
2020-09-02 16:37:03 INFO Master's version: 1599064617242
2020-09-02 16:37:03 INFO Slave's generation: 124000
2020-09-02 16:37:03 INFO Slave's version: 1599064492914
2020-09-02 16:37:03 INFO Starting replication process
2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=
http://178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2}{add=[11907382419
(1676740784884285440), 11907383701 (1676740784889528320), 11907383253
(1676740784900014080), 11907379290 (1676740785002774528), 11907382623
(1676740785005920256), 11907378461 (1676740785011163136), 11907382429
(1676740785012211712), 11907380739 (1676740785023746048), 11907381184
(1676740785038426112), 11907380614 (1676740785042620416), ... (36 adds)]} 0
2227
2020-09-02 16:37:04 INFO Number of files in latest index in master: 1227
2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://
178.33.234.1:8983/solr/db_shard1_replica_t25/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=
<http://178.33.234.1:8983/solr/db_shard1_replica_t25/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false%7D%7Bcommit=>}
0 0
2020-09-02 16:37:04 INFO Starting download (fullCopy=false) to
NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/db_shard1_replica_t3/data/index.20200902163704048
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2020-09-02 16:37:04 INFO [db_shard1_replica_t3] webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=http://
178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2}{add=[10350978095
<http://178.33.234.1:8983/solr/db_shard1_replica_t25/&wt=javabin&version=2%7D%7Badd=%5B10350978095>
(1676740051134840833), 10350978170 (1676740051136937985), 10350978182
(1676740051137986560), 10350978368 (1676740051140083713), 10350978377
(1676740051142180865), 10350979169 (1676740051153715200), 10350979195
(1676740051380207616), 10350979757 (1676740051398033408), 10350980088
(1676740051407470593), 10350980128 (1676740051410616321), ... (50000
adds)]} 0 702638