You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ganesh Sethuraman <ga...@gmail.com> on 2020/05/01 03:03:01 UTC
Re: SolrCloud degraded during backup and batch CSV update
Any other JVM settings change possible?
On Tue, Apr 28, 2020, 10:15 PM Sethuraman, Ganesh
<Se...@dnb.com.invalid> wrote:
> Hi
>
> We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92
> collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes,
> with JVM size of 18GB (G1 GC). We need your help with the Issue we faced
> today: The issue is SolrCloud server went into a degraded collections (for
> few collections) when the Solr backup and the Solr batch CSV update load
> happened at the same time as backup. The CSV data load was about ~5 GB per
> shard/replica. We think this happened after zkClient disconnect happened as
> noted below. We had to restart Solr to bring it back to normal.
>
>
> 1. Is it not suggested to run backup and Solr batch CSV update large
> load at the same time?
> 2. In the past we have seen two CSV batch update load in parallel
> causes issues, is this also not suggested (this issue is not related to
> that)?
> 3. Do you think we should increase Zookeeper timeout?
> 4. How do we know if we need to up the JVM Max memory, and by how much?
> 5. We also see that once the Solr goes into degraded collection and
> recovery failed, it NEVER get back to normal, even after when there is no
> load. Is this a bug?
>
> The GC information and Solr Log below
>
>
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
>
>
> 2020-04-27 07:34:07.322 WARN
> (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181))
> [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server
> in 10775ms for sessionid 0x171a6fb51310008
> ....
> 2020-04-27 07:34:07.426 WARN
> (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread)
> [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
>
>
>
>
> SOLR Log Below (Curtailed WARN log)
> --------------------
> 2020-04-27 07:26:45.402 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.405 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.406 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.408 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.410 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error fetching file, doing one
> retry...:org.apache.solr.common.SolrException: Unable to download
> _1kr_r.liv completely. Downloaded 0!=587
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.414 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.414 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> java.nio.file.NoSuchFileException:
> /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> at
> org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> at
> org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> at
> org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.449 WARN
> (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is
> 3934327071 and actual is checksum 3095753427. expected length is 33235371
> and actual length is 54180642
> 2020-04-27 07:26:45.449 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is
> 138861107 and actual is checksum 188997082. expected length is 107460 and
> actual length is 107460
> 2020-04-27 07:26:45.457 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected
> checksum is 2947323803 and actual is checksum 979431341. expected length is
> 32176 and actual length is 26089
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected
> checksum is 2087731087 and actual is checksum 3540174119. expected length
> is 110 and actual length is 110
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3)
> [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1]
> o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> 777834759 and actual is checksum 289576304. expected length is 59980991 and
> actual length is 6236424
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is
> 1321210176 and actual is checksum 224636564. expected length is 114 and
> actual length is 106
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is
> 3790133021 and actual is checksum 2516047051. expected length is 1477755
> and actual length is 1477850
> 2020-04-27 07:26:45.578 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected
> checksum is 3690299979 and actual is checksum 411724865. expected length is
> 4392270 and actual length is 4396498
> 2020-04-27 07:26:45.588 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected
> checksum is 937224718 and actual is checksum 1176441880. expected length is
> 110 and actual length is 110
> 2020-04-27 07:26:45.733 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected
> checksum is 1429458386 and actual is checksum 4155623862. expected length
> is 2213909 and actual length is 2213939
> 2020-04-27 07:26:45.754 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.ReplicationHandler Index fetch failed
> :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv
> completely. Downloaded 0!=587
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.c.RecoveryStrategy Error while trying to
> recover:org.apache.solr.common.SolrException: Replication for recovery
> failed.
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> 2020-04-27 07:26:45.780 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected
> checksum is 1474431645 and actual is checksum 1271168044. expected length
> is 275570 and actual length is 274506
> 2020-04-27 07:26:45.834 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is
> 4129522944 and actual is checksum 4080519415. expected length is 6626 and
> actual length is 6626
> 2020-04-27 07:26:45.843 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is
> 2493366746 and actual is checksum 31587394. expected length is 546 and
> actual length is 546
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> 1392813257 and actual is checksum 3455988143. expected length is 103 and
> actual length is 111
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected
> checksum is 2301226889 and actual is checksum 29696594. expected length is
> 86459 and actual length is 85786
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected
> checksum is 112676849 and actual is checksum 354222195. expected length is
> 286474 and actual length is 286546
> 2020-04-27 07:26:45.911 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> 2203892485 and actual is checksum 1165086749. expected length is 107460 and
> actual length is 106580
> 2020-04-27 07:26:45.911 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is
> 3175370827 and actual is checksum 520836483. expected length is 51396 and
> actual length is 51404
> 2020-04-27 07:26:45.912 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is
> 967260408 and actual is checksum 969726862. expected length is 3180 and
> actual length is 3180
> 2020-04-27 07:26:45.914 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> 719562508 and actual is checksum 956062095. expected length is 16717 and
> actual length is 21454
> 2020-04-27 07:26:45.915 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected
> checksum is 3833337361 and actual is checksum 493358852. expected length is
> 43996 and actual length is 44026
> 2020-04-27 07:26:45.916 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> checksum is 1533628328 and actual is checksum 3735455395. expected length
> is 110 and actual length is 110
> 2020-04-27 07:26:45.917 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected
> checksum is 75152145 and actual is checksum 1665090946. expected length is
> 497241188 and actual length is 497254814
> 2020-04-27 07:26:45.918 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> checksum is 1349491281 and actual is checksum 3808036883. expected length
> is 25153 and actual length is 32322
> 2020-04-27 07:26:45.920 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> checksum is 3820733560 and actual is checksum 2819887160. expected length
> is 571 and actual length is 725
> 2020-04-27 07:26:46.034 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> checksum is 1547030244 and actual is checksum 1122196351. expected length
> is 324313 and actual length is 323553
> 2020-04-27 07:26:46.170 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> 751850967 and actual is checksum 1026527313. expected length is 508 and
> actual length is 508
> and actual length is 319133
> 2020-04-27 07:27:30.567 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected
> checksum is 3435029878 and actual is checksum 1202327807. expected length
> is 3337719 and actual length is 3375106
> 2020-04-27 07:27:30.578 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is
> 2264443519 and actual is checksum 2299249501. expected length is 104 and
> actual length is 102
> 2020-04-27 07:27:30.580 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is
> 1184027188 and actual is checksum 2810465617. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.622 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> 2246788287 and actual is checksum 2682862227. expected length is 92 and
> actual length is 96
> 2020-04-27 07:27:30.625 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> 3227520043 and actual is checksum 2778425164. expected length is 107137 and
> actual length is 107137
> 2020-04-27 07:27:30.627 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> 1964200539 and actual is checksum 3997289541. expected length is 7398 and
> actual length is 9042
> 2020-04-27 07:27:30.630 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> checksum is 2165407577 and actual is checksum 2563543943. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.632 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> checksum is 1427836076 and actual is checksum 225462388. expected length is
> 11162 and actual length is 13623
> 2020-04-27 07:27:30.634 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> checksum is 2998276982 and actual is checksum 1966428087. expected length
> is 296 and actual length is 353
> 2020-04-27 07:27:30.637 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> checksum is 2717609594 and actual is checksum 673086945. expected length is
> 315491 and actual length is 312531
> 2020-04-27 07:27:30.640 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> 2885031416 and actual is checksum 762513018. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.643 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected
> checksum is 602612626 and actual is checksum 2826680041. expected length is
> 1292092 and actual length is 1655171
> 2020-04-27 07:27:30.669 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected
> checksum is 401385741 and actual is checksum 3373755803. expected length is
> 621 and actual length is 715
> 2020-04-27 07:27:30.671 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is
> 3708405765 and actual is checksum 1057460545. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.674 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected
> checksum is 2755907323 and actual is checksum 1931314059. expected length
> is 327125 and actual length is 318737
> 2020-04-27 07:27:30.677 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is
> 77505520 and actual is checksum 2602905677. expected length is 107240 and
> actual length is 107460
> 2020-04-27 07:27:30.680 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected
> checksum is 4204960097 and actual is checksum 2276943505. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.682 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is
> 3414743450 and actual is checksum 3890346633. expected length is 18415 and
> actual length is 21922
> 2020-04-27 07:27:30.685 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected
> checksum is 2879829129 and actual is checksum 790212338. expected length is
> 27860 and actual length is 33235
> 2020-04-27 07:27:30.687 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is
> 3356980932 and actual is checksum 1544608917. expected length is 108 and
> actual length is 112
> 2020-04-27 07:27:30.690 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected
> checksum is 1979425912 and actual is checksum 584894070. expected length is
> 3726110 and actual length is 4689653
> 2020-04-27 07:27:30.700 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is
> 3754169108 and actual is checksum 527001228. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.702 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected
> checksum is 3981710860 and actual is checksum 1749011618. expected length
> is 1938123 and actual length is 33694
> 2020-04-27 07:27:30.710 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is
> 1515379146 and actual is checksum 3212722427. expected length is 98 and
> actual length is 84
> 2020-04-27 07:27:30.712 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is
> 1819400085 and actual is checksum 3372088644. expected length is 107020 and
> actual length is 103887
> 2020-04-27 07:27:30.715 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected
> checksum is 1401562358 and actual is checksum 2224139078. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.718 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected
> checksum is 3386701421 and actual is checksum 1434467863. expected length
> is 15944 and actual length is 685
> 2020-04-27 07:27:30.720 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is
> 4259543754 and actual is checksum 3203989793. expected length is 10616 and
> actual length is 439
> 2020-04-27 07:27:30.722 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected
> checksum is 725882371 and actual is checksum 3650374401. expected length is
> 383 and actual length is 114
> 2020-04-27 07:27:30.725 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected
> checksum is 3717717087 and actual is checksum 3712473868. expected length
> is 303729 and actual length is 117582
> 2020-04-27 07:27:30.728 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is
> 3506718582 and actual is checksum 342046803. expected length is 93 and
> actual length is 104
> 2020-04-27 07:27:30.731 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is
> 2920971663 and actual is checksum 2142622028. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.733 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected
> checksum is 723455769 and actual is checksum 2731538349. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:30.735 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected
> checksum is 2624342764 and actual is checksum 1283943229. expected length
> is 11424 and actual length is 26688
> 2020-04-27 07:27:30.738 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected
> checksum is 978416968 and actual is checksum 1807767534. expected length is
> 330 and actual length is 597
> 2020-04-27 07:27:30.740 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected
> checksum is 3599774596 and actual is checksum 2764190632. expected length
> is 318017 and actual length is 311617
> 2020-04-27 07:27:30.743 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected
> checksum is 3141314417 and actual is checksum 3847788889. expected length
> is 1258075 and actual length is 3539994
> 2020-04-27 07:27:30.749 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is
> 1404317797 and actual is checksum 2471843701. expected length is 105295 and
> actual length is 107020
> 2020-04-27 07:27:30.752 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is
> 3412288246 and actual is checksum 922481525. expected length is 7580 and
> actual length is 17733
> 2020-04-27 07:27:30.754 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected
> checksum is 2675540020 and actual is checksum 1452209674. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.757 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected
> checksum is 665797990 and actual is checksum 2134373365. expected length is
> 10715 and actual length is 3418
> 2020-04-27 07:27:30.759 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is
> 2006427884 and actual is checksum 527297847. expected length is 7178 and
> actual length is 2326
> 2020-04-27 07:27:30.761 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is
> 200552893 and actual is checksum 3290522141. expected length is 106580 and
> actual length is 105206
> 2020-04-27 07:27:30.764 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected
> checksum is 1511033560 and actual is checksum 3689083691. expected length
> is 278 and actual length is 154
> 2020-04-27 07:27:30.767 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected
> checksum is 1435050119 and actual is checksum 6864243. expected length is
> 323217 and actual length is 338063
> 2020-04-27 07:27:30.770 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is
> 1107038327 and actual is checksum 3068870234. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.772 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is
> 2853806777 and actual is checksum 459866651. expected length is 93 and
> actual length is 87
> 2020-04-27 07:27:30.775 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected
> checksum is 1356537478 and actual is checksum 2594306289. expected length
> is 1179215 and actual length is 249458
> 2020-04-27 07:27:30.780 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is
> 1785795479 and actual is checksum 65014613. expected length is 106580 and
> actual length is 106800
> 2020-04-27 07:27:30.783 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected
> checksum is 1824666616 and actual is checksum 2857527783. expected length
> is 507 and actual length is 637
> 2020-04-27 07:27:30.785 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is
> 1384446542 and actual is checksum 3981590407. expected length is 13686 and
> actual length is 18516
> 2020-04-27 07:27:30.788 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected
> checksum is 3380549662 and actual is checksum 2002436369. expected length
> is 317777 and actual length is 319213
> 2020-04-27 07:27:30.791 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected
> checksum is 1594954666 and actual is checksum 1892723425. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.794 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected
> checksum is 305271241 and actual is checksum 854278263. expected length is
> 20522 and actual length is 27836
> 2020-04-27 07:27:30.796 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is
> 2220527769 and actual is checksum 2016379681. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.799 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is
> 1648054590 and actual is checksum 137025071. expected length is 100 and
> actual length is 105
> 2020-04-27 07:27:30.801 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected
> checksum is 262113808 and actual is checksum 327562225. expected length is
> 2731808 and actual length is 3909012
> 2020-04-27 07:27:30.809 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected
> checksum is 2157667750 and actual is checksum 3043505208. expected length
> is 36906 and actual length is 163443
> 2020-04-27 07:27:30.812 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is
> 3456088243 and actual is checksum 1715362430. expected length is 453 and
> actual length is 1636
> 2020-04-27 07:27:30.814 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is
> 4028738113 and actual is checksum 4245480574. expected length is 104876 and
> actual length is 105515
> 2020-04-27 07:27:30.817 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected
> checksum is 3120632072 and actual is checksum 1945528118. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.819 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected
> checksum is 3303166970 and actual is checksum 1740028553. expected length
> is 114 and actual length is 137
> 2020-04-27 07:27:30.822 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected
> checksum is 126700369 and actual is checksum 162509958. expected length is
> 700 and actual length is 2398
> 2020-04-27 07:27:30.825 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is
> 3157051968 and actual is checksum 137679212. expected length is 84 and
> actual length is 86
> 2020-04-27 07:27:30.827 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected
> checksum is 3969090178 and actual is checksum 2364978843. expected length
> is 127576 and actual length is 266581
> 2020-04-27 07:27:30.830 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is
> 3730892058 and actual is checksum 2734980133. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.832 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is
> 3259266514 and actual is checksum 2051283039. expected length is 104855 and
> actual length is 106819
> 2020-04-27 07:27:30.835 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected
> checksum is 380153843 and actual is checksum 580580220. expected length is
> 1822 and actual length is 1737
> 2020-04-27 07:27:30.838 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected
> checksum is 4094454161 and actual is checksum 421930789. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:30.840 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is
> 911575783 and actual is checksum 2090965125. expected length is 1244 and
> actual length is 1201
> 2020-04-27 07:27:30.842 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected
> checksum is 1279063865 and actual is checksum 765869778. expected length is
> 128 and actual length is 130
> 2020-04-27 07:27:30.845 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected
> checksum is 1689040450 and actual is checksum 627554920. expected length is
> 223473 and actual length is 230157
> 2020-04-27 07:27:30.848 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is
> 1278110969 and actual is checksum 1444534251. expected length is 86 and
> actual length is 86
> 2020-04-27 07:27:30.850 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected
> checksum is 1992192696 and actual is checksum 3455180678. expected length
> is 119844 and actual length is 117860
> 2020-04-27 07:27:30.853 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is
> 2776236859 and actual is checksum 2572132550. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:35.866 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is
> 2573424453 and actual is checksum 3612352961. expected length is 546 and
> actual length is 437
> 2020-04-27 07:27:35.868 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is
> 2592320603 and actual is checksum 1374033039. expected length is 289 and
> actual length is 289
> 2020-04-27 07:27:35.872 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is
> 2911257988 and actual is checksum 2011202433. expected length is 437 and
> actual length is 437
> 2020-04-27 07:27:35.874 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is
> 1304802713 and actual is checksum 454030092. expected length is 7598425 and
> actual length is 49265169
> 2020-04-27 07:27:37.234 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is
> 1422458436 and actual is checksum 703149874. expected length is 437 and
> actual length is 437
> 2020-04-27 07:27:38.117 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> 1892273984 and actual is checksum 2087644984. expected length is 50710602
> and actual length is 10402264
> 2020-04-27 07:27:38.532 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is
> 3864394245 and actual is checksum 1846889307. expected length is 289 and
> actual length is 289
> 2020-04-27 07:27:38.575 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is
> 660375856 and actual is checksum 571052523. expected length is 114 and
> actual length is 84
> 2020-04-27 07:27:38.590 WARN
> (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744
> deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> 2020-04-27 07:27:39.259 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected
> checksum is 421440641 and actual is checksum 2075160524. expected length is
> 337361 and actual length is 141820
> 2020-04-27 07:27:44.881 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected
> checksum is 47189924 and actual is checksum 3640637009. expected length is
> 5235629 and actual length is 48679
> 2020-04-27 07:27:45.335 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected
> checksum is 2305624674 and actual is checksum 704932512. expected length is
> 693 and actual length is 115
> 2020-04-27 07:27:45.939 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is
> 1157089232 and actual is checksum 2388254324. expected length is 23435 and
> actual length is 673
> 2020-04-27 07:27:45.941 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is
> 2811885732 and actual is checksum 2854129172. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:45.943 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected
> checksum is 1925626873 and actual is checksum 1921615605. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:45.945 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected
> checksum is 3627283142 and actual is checksum 3796132842. expected length
> is 34596 and actual length is 958
> 2020-04-27 07:27:45.947 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is
> 1291024098 and actual is checksum 2700484857. expected length is 107020 and
> actual length is 104209
> 2020-04-27 07:27:45.949 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected
> checksum is 2785033346 and actual is checksum 1911387906. expected length
> is 816 and actual length is 760
> 2020-04-27 07:27:45.951 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected
> checksum is 942596692 and actual is checksum 3039336477. expected length is
> 5557044 and actual length is 4847564
> 2020-04-27 07:27:45.974 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is
> 143536399 and actual is checksum 2962535892. expected length is 116 and
> actual length is 111
> 2020-04-27 07:27:45.976 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected
> checksum is 3884436961 and actual is checksum 1345841996. expected length
> is 339801 and actual length is 336809
> 2020-04-27 07:27:45.980 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is
> 3902419774 and actual is checksum 3791519088. expected length is 23306 and
> actual length is 21047
> 2020-04-27 07:27:45.981 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is
> 45470103 and actual is checksum 235515609. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:45.983 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected
> checksum is 408594907 and actual is checksum 2650340971. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:45.985 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected
> checksum is 761408352 and actual is checksum 685575313. expected length is
> 35256 and actual length is 31644
> 2020-04-27 07:27:45.987 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is
> 811204953 and actual is checksum 1244308773. expected length is 107460 and
> actual length is 107460
> 2020-04-27 07:27:45.989 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is
> 3813441918 and actual is checksum 2133371346. expected length is 107460 and
> actual length is 106360
> 2020-04-27 07:27:45.991 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is
> 4263416408 and actual is checksum 3722461284. expected length is 16698 and
> actual length is 21392
> 2020-04-27 07:27:45.993 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected
> checksum is 632422377 and actual is checksum 501799146. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:45.994 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected
> checksum is 3170165892 and actual is checksum 4152937488. expected length
> is 25090 and actual length is 32328
> 2020-04-27 07:27:46.000 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected
> checksum is 2679120655 and actual is checksum 3243853760. expected length
> is 553 and actual length is 726
> 2020-04-27 07:27:46.001 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected
> checksum is 2991034554 and actual is checksum 1980969905. expected length
> is 321785 and actual length is 322117
> 2020-04-27 07:27:46.004 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _
Re: SolrCloud degraded during backup and batch CSV update
Posted by matthew sporleder <ms...@gmail.com>.
If the errors happen with garbage collection then potentially, yes.
You should never pause longer than your zk timeout (both sides).
On Thu, Apr 30, 2020 at 11:03 PM Ganesh Sethuraman
<ga...@gmail.com> wrote:
>
> Any other JVM settings change possible?
>
> On Tue, Apr 28, 2020, 10:15 PM Sethuraman, Ganesh
> <Se...@dnb.com.invalid> wrote:
>
> > Hi
> >
> > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92
> > collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes,
> > with JVM size of 18GB (G1 GC). We need your help with the Issue we faced
> > today: The issue is SolrCloud server went into a degraded collections (for
> > few collections) when the Solr backup and the Solr batch CSV update load
> > happened at the same time as backup. The CSV data load was about ~5 GB per
> > shard/replica. We think this happened after zkClient disconnect happened as
> > noted below. We had to restart Solr to bring it back to normal.
> >
> >
> > 1. Is it not suggested to run backup and Solr batch CSV update large
> > load at the same time?
> > 2. In the past we have seen two CSV batch update load in parallel
> > causes issues, is this also not suggested (this issue is not related to
> > that)?
> > 3. Do you think we should increase Zookeeper timeout?
> > 4. How do we know if we need to up the JVM Max memory, and by how much?
> > 5. We also see that once the Solr goes into degraded collection and
> > recovery failed, it NEVER get back to normal, even after when there is no
> > load. Is this a bug?
> >
> > The GC information and Solr Log below
> >
> >
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
> >
> >
> > 2020-04-27 07:34:07.322 WARN
> > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181))
> > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server
> > in 10775ms for sessionid 0x171a6fb51310008
> > ....
> > 2020-04-27 07:34:07.426 WARN
> > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread)
> > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> >
> >
> >
> >
> > SOLR Log Below (Curtailed WARN log)
> > --------------------
> > 2020-04-27 07:26:45.402 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.405 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.406 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.408 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.410 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error fetching file, doing one
> > retry...:org.apache.solr.common.SolrException: Unable to download
> > _1kr_r.liv completely. Downloaded 0!=587
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.414 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.414 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> > java.nio.file.NoSuchFileException:
> > /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> > at
> > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> > at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> > at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> > at
> > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> > at
> > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> > at java.nio.file.Files.delete(Files.java:1126)
> > at
> > org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> > at
> > org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> > at
> > org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> > at
> > org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.449 WARN
> > (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> > o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is
> > 3934327071 and actual is checksum 3095753427. expected length is 33235371
> > and actual length is 54180642
> > 2020-04-27 07:26:45.449 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is
> > 138861107 and actual is checksum 188997082. expected length is 107460 and
> > actual length is 107460
> > 2020-04-27 07:26:45.457 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected
> > checksum is 2947323803 and actual is checksum 979431341. expected length is
> > 32176 and actual length is 26089
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected
> > checksum is 2087731087 and actual is checksum 3540174119. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3)
> > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1]
> > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> > 777834759 and actual is checksum 289576304. expected length is 59980991 and
> > actual length is 6236424
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is
> > 1321210176 and actual is checksum 224636564. expected length is 114 and
> > actual length is 106
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is
> > 3790133021 and actual is checksum 2516047051. expected length is 1477755
> > and actual length is 1477850
> > 2020-04-27 07:26:45.578 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected
> > checksum is 3690299979 and actual is checksum 411724865. expected length is
> > 4392270 and actual length is 4396498
> > 2020-04-27 07:26:45.588 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected
> > checksum is 937224718 and actual is checksum 1176441880. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:26:45.733 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected
> > checksum is 1429458386 and actual is checksum 4155623862. expected length
> > is 2213909 and actual length is 2213939
> > 2020-04-27 07:26:45.754 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.ReplicationHandler Index fetch failed
> > :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv
> > completely. Downloaded 0!=587
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.c.RecoveryStrategy Error while trying to
> > recover:org.apache.solr.common.SolrException: Replication for recovery
> > failed.
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> > 2020-04-27 07:26:45.780 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected
> > checksum is 1474431645 and actual is checksum 1271168044. expected length
> > is 275570 and actual length is 274506
> > 2020-04-27 07:26:45.834 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is
> > 4129522944 and actual is checksum 4080519415. expected length is 6626 and
> > actual length is 6626
> > 2020-04-27 07:26:45.843 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is
> > 2493366746 and actual is checksum 31587394. expected length is 546 and
> > actual length is 546
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> > 1392813257 and actual is checksum 3455988143. expected length is 103 and
> > actual length is 111
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected
> > checksum is 2301226889 and actual is checksum 29696594. expected length is
> > 86459 and actual length is 85786
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected
> > checksum is 112676849 and actual is checksum 354222195. expected length is
> > 286474 and actual length is 286546
> > 2020-04-27 07:26:45.911 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> > 2203892485 and actual is checksum 1165086749. expected length is 107460 and
> > actual length is 106580
> > 2020-04-27 07:26:45.911 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is
> > 3175370827 and actual is checksum 520836483. expected length is 51396 and
> > actual length is 51404
> > 2020-04-27 07:26:45.912 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is
> > 967260408 and actual is checksum 969726862. expected length is 3180 and
> > actual length is 3180
> > 2020-04-27 07:26:45.914 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> > 719562508 and actual is checksum 956062095. expected length is 16717 and
> > actual length is 21454
> > 2020-04-27 07:26:45.915 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected
> > checksum is 3833337361 and actual is checksum 493358852. expected length is
> > 43996 and actual length is 44026
> > 2020-04-27 07:26:45.916 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> > checksum is 1533628328 and actual is checksum 3735455395. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:26:45.917 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected
> > checksum is 75152145 and actual is checksum 1665090946. expected length is
> > 497241188 and actual length is 497254814
> > 2020-04-27 07:26:45.918 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> > checksum is 1349491281 and actual is checksum 3808036883. expected length
> > is 25153 and actual length is 32322
> > 2020-04-27 07:26:45.920 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> > checksum is 3820733560 and actual is checksum 2819887160. expected length
> > is 571 and actual length is 725
> > 2020-04-27 07:26:46.034 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> > checksum is 1547030244 and actual is checksum 1122196351. expected length
> > is 324313 and actual length is 323553
> > 2020-04-27 07:26:46.170 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> > 751850967 and actual is checksum 1026527313. expected length is 508 and
> > actual length is 508
> > and actual length is 319133
> > 2020-04-27 07:27:30.567 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected
> > checksum is 3435029878 and actual is checksum 1202327807. expected length
> > is 3337719 and actual length is 3375106
> > 2020-04-27 07:27:30.578 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is
> > 2264443519 and actual is checksum 2299249501. expected length is 104 and
> > actual length is 102
> > 2020-04-27 07:27:30.580 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is
> > 1184027188 and actual is checksum 2810465617. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.622 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> > 2246788287 and actual is checksum 2682862227. expected length is 92 and
> > actual length is 96
> > 2020-04-27 07:27:30.625 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> > 3227520043 and actual is checksum 2778425164. expected length is 107137 and
> > actual length is 107137
> > 2020-04-27 07:27:30.627 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> > 1964200539 and actual is checksum 3997289541. expected length is 7398 and
> > actual length is 9042
> > 2020-04-27 07:27:30.630 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> > checksum is 2165407577 and actual is checksum 2563543943. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.632 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> > checksum is 1427836076 and actual is checksum 225462388. expected length is
> > 11162 and actual length is 13623
> > 2020-04-27 07:27:30.634 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> > checksum is 2998276982 and actual is checksum 1966428087. expected length
> > is 296 and actual length is 353
> > 2020-04-27 07:27:30.637 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> > checksum is 2717609594 and actual is checksum 673086945. expected length is
> > 315491 and actual length is 312531
> > 2020-04-27 07:27:30.640 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> > 2885031416 and actual is checksum 762513018. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.643 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected
> > checksum is 602612626 and actual is checksum 2826680041. expected length is
> > 1292092 and actual length is 1655171
> > 2020-04-27 07:27:30.669 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected
> > checksum is 401385741 and actual is checksum 3373755803. expected length is
> > 621 and actual length is 715
> > 2020-04-27 07:27:30.671 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is
> > 3708405765 and actual is checksum 1057460545. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.674 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected
> > checksum is 2755907323 and actual is checksum 1931314059. expected length
> > is 327125 and actual length is 318737
> > 2020-04-27 07:27:30.677 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is
> > 77505520 and actual is checksum 2602905677. expected length is 107240 and
> > actual length is 107460
> > 2020-04-27 07:27:30.680 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected
> > checksum is 4204960097 and actual is checksum 2276943505. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.682 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is
> > 3414743450 and actual is checksum 3890346633. expected length is 18415 and
> > actual length is 21922
> > 2020-04-27 07:27:30.685 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected
> > checksum is 2879829129 and actual is checksum 790212338. expected length is
> > 27860 and actual length is 33235
> > 2020-04-27 07:27:30.687 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is
> > 3356980932 and actual is checksum 1544608917. expected length is 108 and
> > actual length is 112
> > 2020-04-27 07:27:30.690 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected
> > checksum is 1979425912 and actual is checksum 584894070. expected length is
> > 3726110 and actual length is 4689653
> > 2020-04-27 07:27:30.700 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is
> > 3754169108 and actual is checksum 527001228. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.702 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected
> > checksum is 3981710860 and actual is checksum 1749011618. expected length
> > is 1938123 and actual length is 33694
> > 2020-04-27 07:27:30.710 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is
> > 1515379146 and actual is checksum 3212722427. expected length is 98 and
> > actual length is 84
> > 2020-04-27 07:27:30.712 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is
> > 1819400085 and actual is checksum 3372088644. expected length is 107020 and
> > actual length is 103887
> > 2020-04-27 07:27:30.715 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected
> > checksum is 1401562358 and actual is checksum 2224139078. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.718 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected
> > checksum is 3386701421 and actual is checksum 1434467863. expected length
> > is 15944 and actual length is 685
> > 2020-04-27 07:27:30.720 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is
> > 4259543754 and actual is checksum 3203989793. expected length is 10616 and
> > actual length is 439
> > 2020-04-27 07:27:30.722 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected
> > checksum is 725882371 and actual is checksum 3650374401. expected length is
> > 383 and actual length is 114
> > 2020-04-27 07:27:30.725 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected
> > checksum is 3717717087 and actual is checksum 3712473868. expected length
> > is 303729 and actual length is 117582
> > 2020-04-27 07:27:30.728 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is
> > 3506718582 and actual is checksum 342046803. expected length is 93 and
> > actual length is 104
> > 2020-04-27 07:27:30.731 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is
> > 2920971663 and actual is checksum 2142622028. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.733 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected
> > checksum is 723455769 and actual is checksum 2731538349. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:30.735 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected
> > checksum is 2624342764 and actual is checksum 1283943229. expected length
> > is 11424 and actual length is 26688
> > 2020-04-27 07:27:30.738 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected
> > checksum is 978416968 and actual is checksum 1807767534. expected length is
> > 330 and actual length is 597
> > 2020-04-27 07:27:30.740 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected
> > checksum is 3599774596 and actual is checksum 2764190632. expected length
> > is 318017 and actual length is 311617
> > 2020-04-27 07:27:30.743 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected
> > checksum is 3141314417 and actual is checksum 3847788889. expected length
> > is 1258075 and actual length is 3539994
> > 2020-04-27 07:27:30.749 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is
> > 1404317797 and actual is checksum 2471843701. expected length is 105295 and
> > actual length is 107020
> > 2020-04-27 07:27:30.752 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is
> > 3412288246 and actual is checksum 922481525. expected length is 7580 and
> > actual length is 17733
> > 2020-04-27 07:27:30.754 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected
> > checksum is 2675540020 and actual is checksum 1452209674. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.757 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected
> > checksum is 665797990 and actual is checksum 2134373365. expected length is
> > 10715 and actual length is 3418
> > 2020-04-27 07:27:30.759 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is
> > 2006427884 and actual is checksum 527297847. expected length is 7178 and
> > actual length is 2326
> > 2020-04-27 07:27:30.761 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is
> > 200552893 and actual is checksum 3290522141. expected length is 106580 and
> > actual length is 105206
> > 2020-04-27 07:27:30.764 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected
> > checksum is 1511033560 and actual is checksum 3689083691. expected length
> > is 278 and actual length is 154
> > 2020-04-27 07:27:30.767 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected
> > checksum is 1435050119 and actual is checksum 6864243. expected length is
> > 323217 and actual length is 338063
> > 2020-04-27 07:27:30.770 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is
> > 1107038327 and actual is checksum 3068870234. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.772 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is
> > 2853806777 and actual is checksum 459866651. expected length is 93 and
> > actual length is 87
> > 2020-04-27 07:27:30.775 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected
> > checksum is 1356537478 and actual is checksum 2594306289. expected length
> > is 1179215 and actual length is 249458
> > 2020-04-27 07:27:30.780 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is
> > 1785795479 and actual is checksum 65014613. expected length is 106580 and
> > actual length is 106800
> > 2020-04-27 07:27:30.783 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected
> > checksum is 1824666616 and actual is checksum 2857527783. expected length
> > is 507 and actual length is 637
> > 2020-04-27 07:27:30.785 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is
> > 1384446542 and actual is checksum 3981590407. expected length is 13686 and
> > actual length is 18516
> > 2020-04-27 07:27:30.788 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected
> > checksum is 3380549662 and actual is checksum 2002436369. expected length
> > is 317777 and actual length is 319213
> > 2020-04-27 07:27:30.791 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected
> > checksum is 1594954666 and actual is checksum 1892723425. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.794 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected
> > checksum is 305271241 and actual is checksum 854278263. expected length is
> > 20522 and actual length is 27836
> > 2020-04-27 07:27:30.796 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is
> > 2220527769 and actual is checksum 2016379681. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.799 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is
> > 1648054590 and actual is checksum 137025071. expected length is 100 and
> > actual length is 105
> > 2020-04-27 07:27:30.801 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected
> > checksum is 262113808 and actual is checksum 327562225. expected length is
> > 2731808 and actual length is 3909012
> > 2020-04-27 07:27:30.809 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected
> > checksum is 2157667750 and actual is checksum 3043505208. expected length
> > is 36906 and actual length is 163443
> > 2020-04-27 07:27:30.812 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is
> > 3456088243 and actual is checksum 1715362430. expected length is 453 and
> > actual length is 1636
> > 2020-04-27 07:27:30.814 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is
> > 4028738113 and actual is checksum 4245480574. expected length is 104876 and
> > actual length is 105515
> > 2020-04-27 07:27:30.817 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected
> > checksum is 3120632072 and actual is checksum 1945528118. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.819 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected
> > checksum is 3303166970 and actual is checksum 1740028553. expected length
> > is 114 and actual length is 137
> > 2020-04-27 07:27:30.822 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected
> > checksum is 126700369 and actual is checksum 162509958. expected length is
> > 700 and actual length is 2398
> > 2020-04-27 07:27:30.825 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is
> > 3157051968 and actual is checksum 137679212. expected length is 84 and
> > actual length is 86
> > 2020-04-27 07:27:30.827 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected
> > checksum is 3969090178 and actual is checksum 2364978843. expected length
> > is 127576 and actual length is 266581
> > 2020-04-27 07:27:30.830 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is
> > 3730892058 and actual is checksum 2734980133. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.832 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is
> > 3259266514 and actual is checksum 2051283039. expected length is 104855 and
> > actual length is 106819
> > 2020-04-27 07:27:30.835 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected
> > checksum is 380153843 and actual is checksum 580580220. expected length is
> > 1822 and actual length is 1737
> > 2020-04-27 07:27:30.838 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected
> > checksum is 4094454161 and actual is checksum 421930789. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:30.840 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is
> > 911575783 and actual is checksum 2090965125. expected length is 1244 and
> > actual length is 1201
> > 2020-04-27 07:27:30.842 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected
> > checksum is 1279063865 and actual is checksum 765869778. expected length is
> > 128 and actual length is 130
> > 2020-04-27 07:27:30.845 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected
> > checksum is 1689040450 and actual is checksum 627554920. expected length is
> > 223473 and actual length is 230157
> > 2020-04-27 07:27:30.848 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is
> > 1278110969 and actual is checksum 1444534251. expected length is 86 and
> > actual length is 86
> > 2020-04-27 07:27:30.850 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected
> > checksum is 1992192696 and actual is checksum 3455180678. expected length
> > is 119844 and actual length is 117860
> > 2020-04-27 07:27:30.853 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is
> > 2776236859 and actual is checksum 2572132550. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:35.866 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is
> > 2573424453 and actual is checksum 3612352961. expected length is 546 and
> > actual length is 437
> > 2020-04-27 07:27:35.868 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is
> > 2592320603 and actual is checksum 1374033039. expected length is 289 and
> > actual length is 289
> > 2020-04-27 07:27:35.872 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is
> > 2911257988 and actual is checksum 2011202433. expected length is 437 and
> > actual length is 437
> > 2020-04-27 07:27:35.874 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is
> > 1304802713 and actual is checksum 454030092. expected length is 7598425 and
> > actual length is 49265169
> > 2020-04-27 07:27:37.234 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is
> > 1422458436 and actual is checksum 703149874. expected length is 437 and
> > actual length is 437
> > 2020-04-27 07:27:38.117 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> > 1892273984 and actual is checksum 2087644984. expected length is 50710602
> > and actual length is 10402264
> > 2020-04-27 07:27:38.532 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is
> > 3864394245 and actual is checksum 1846889307. expected length is 289 and
> > actual length is 289
> > 2020-04-27 07:27:38.575 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is
> > 660375856 and actual is checksum 571052523. expected length is 114 and
> > actual length is 84
> > 2020-04-27 07:27:38.590 WARN
> > (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744
> > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> > 2020-04-27 07:27:39.259 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected
> > checksum is 421440641 and actual is checksum 2075160524. expected length is
> > 337361 and actual length is 141820
> > 2020-04-27 07:27:44.881 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected
> > checksum is 47189924 and actual is checksum 3640637009. expected length is
> > 5235629 and actual length is 48679
> > 2020-04-27 07:27:45.335 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected
> > checksum is 2305624674 and actual is checksum 704932512. expected length is
> > 693 and actual length is 115
> > 2020-04-27 07:27:45.939 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is
> > 1157089232 and actual is checksum 2388254324. expected length is 23435 and
> > actual length is 673
> > 2020-04-27 07:27:45.941 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is
> > 2811885732 and actual is checksum 2854129172. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:45.943 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected
> > checksum is 1925626873 and actual is checksum 1921615605. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:45.945 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected
> > checksum is 3627283142 and actual is checksum 3796132842. expected length
> > is 34596 and actual length is 958
> > 2020-04-27 07:27:45.947 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is
> > 1291024098 and actual is checksum 2700484857. expected length is 107020 and
> > actual length is 104209
> > 2020-04-27 07:27:45.949 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected
> > checksum is 2785033346 and actual is checksum 1911387906. expected length
> > is 816 and actual length is 760
> > 2020-04-27 07:27:45.951 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected
> > checksum is 942596692 and actual is checksum 3039336477. expected length is
> > 5557044 and actual length is 4847564
> > 2020-04-27 07:27:45.974 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is
> > 143536399 and actual is checksum 2962535892. expected length is 116 and
> > actual length is 111
> > 2020-04-27 07:27:45.976 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected
> > checksum is 3884436961 and actual is checksum 1345841996. expected length
> > is 339801 and actual length is 336809
> > 2020-04-27 07:27:45.980 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is
> > 3902419774 and actual is checksum 3791519088. expected length is 23306 and
> > actual length is 21047
> > 2020-04-27 07:27:45.981 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is
> > 45470103 and actual is checksum 235515609. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:45.983 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected
> > checksum is 408594907 and actual is checksum 2650340971. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:45.985 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected
> > checksum is 761408352 and actual is checksum 685575313. expected length is
> > 35256 and actual length is 31644
> > 2020-04-27 07:27:45.987 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is
> > 811204953 and actual is checksum 1244308773. expected length is 107460 and
> > actual length is 107460
> > 2020-04-27 07:27:45.989 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is
> > 3813441918 and actual is checksum 2133371346. expected length is 107460 and
> > actual length is 106360
> > 2020-04-27 07:27:45.991 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is
> > 4263416408 and actual is checksum 3722461284. expected length is 16698 and
> > actual length is 21392
> > 2020-04-27 07:27:45.993 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected
> > checksum is 632422377 and actual is checksum 501799146. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:45.994 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected
> > checksum is 3170165892 and actual is checksum 4152937488. expected length
> > is 25090 and actual length is 32328
> > 2020-04-27 07:27:46.000 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected
> > checksum is 2679120655 and actual is checksum 3243853760. expected length
> > is 553 and actual length is 726
> > 2020-04-27 07:27:46.001 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected
> > checksum is 2991034554 and actual is checksum 1980969905. expected length
> > is 321785 and actual length is 322117
> > 2020-04-27 07:27:46.004 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _