You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Sethuraman, Ganesh" <Se...@DNB.com.INVALID> on 2020/04/29 02:14:50 UTC
SolrCloud degraded during backup and batch CSV update
Hi
We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: The issue is SolrCloud server went into a degraded collections (for few collections) when the Solr backup and the Solr batch CSV update load happened at the same time as backup. The CSV data load was about ~5 GB per shard/replica. We think this happened after zkClient disconnect happened as noted below. We had to restart Solr to bring it back to normal.
1. Is it not suggested to run backup and Solr batch CSV update large load at the same time?
2. In the past we have seen two CSV batch update load in parallel causes issues, is this also not suggested (this issue is not related to that)?
3. Do you think we should increase Zookeeper timeout?
4. How do we know if we need to up the JVM Max memory, and by how much?
5. We also see that once the Solr goes into degraded collection and recovery failed, it NEVER get back to normal, even after when there is no load. Is this a bug?
The GC information and Solr Log below
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
....
2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
SOLR Log Below (Curtailed WARN log)
--------------------
2020-04-27 07:26:45.402 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.405 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.406 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.408 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.410 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.412 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.412 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error fetching file, doing one retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.414 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.414 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
java.nio.file.NoSuchFileException: /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
at org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is 3934327071 and actual is checksum 3095753427. expected length is 33235371 and actual length is 54180642
2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is 138861107 and actual is checksum 188997082. expected length is 107460 and actual length is 107460
2020-04-27 07:26:45.457 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected checksum is 2947323803 and actual is checksum 979431341. expected length is 32176 and actual length is 26089
2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected checksum is 2087731087 and actual is checksum 3540174119. expected length is 110 and actual length is 110
2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 777834759 and actual is checksum 289576304. expected length is 59980991 and actual length is 6236424
2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is 1321210176 and actual is checksum 224636564. expected length is 114 and actual length is 106
2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is 3790133021 and actual is checksum 2516047051. expected length is 1477755 and actual length is 1477850
2020-04-27 07:26:45.578 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected checksum is 3690299979 and actual is checksum 411724865. expected length is 4392270 and actual length is 4396498
2020-04-27 07:26:45.588 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected checksum is 937224718 and actual is checksum 1176441880. expected length is 110 and actual length is 110
2020-04-27 07:26:45.733 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected checksum is 1429458386 and actual is checksum 4155623862. expected length is 2213909 and actual length is 2213939
2020-04-27 07:26:45.754 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
2020-04-27 07:26:45.780 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected checksum is 1474431645 and actual is checksum 1271168044. expected length is 275570 and actual length is 274506
2020-04-27 07:26:45.834 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is 4129522944 and actual is checksum 4080519415. expected length is 6626 and actual length is 6626
2020-04-27 07:26:45.843 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is 2493366746 and actual is checksum 31587394. expected length is 546 and actual length is 546
2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 1392813257 and actual is checksum 3455988143. expected length is 103 and actual length is 111
2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected checksum is 2301226889 and actual is checksum 29696594. expected length is 86459 and actual length is 85786
2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected checksum is 112676849 and actual is checksum 354222195. expected length is 286474 and actual length is 286546
2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 2203892485 and actual is checksum 1165086749. expected length is 107460 and actual length is 106580
2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is 3175370827 and actual is checksum 520836483. expected length is 51396 and actual length is 51404
2020-04-27 07:26:45.912 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is 967260408 and actual is checksum 969726862. expected length is 3180 and actual length is 3180
2020-04-27 07:26:45.914 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 719562508 and actual is checksum 956062095. expected length is 16717 and actual length is 21454
2020-04-27 07:26:45.915 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected checksum is 3833337361 and actual is checksum 493358852. expected length is 43996 and actual length is 44026
2020-04-27 07:26:45.916 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 1533628328 and actual is checksum 3735455395. expected length is 110 and actual length is 110
2020-04-27 07:26:45.917 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected checksum is 75152145 and actual is checksum 1665090946. expected length is 497241188 and actual length is 497254814
2020-04-27 07:26:45.918 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1349491281 and actual is checksum 3808036883. expected length is 25153 and actual length is 32322
2020-04-27 07:26:45.920 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 3820733560 and actual is checksum 2819887160. expected length is 571 and actual length is 725
2020-04-27 07:26:46.034 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 1547030244 and actual is checksum 1122196351. expected length is 324313 and actual length is 323553
2020-04-27 07:26:46.170 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 751850967 and actual is checksum 1026527313. expected length is 508 and actual length is 508
and actual length is 319133
2020-04-27 07:27:30.567 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 3435029878 and actual is checksum 1202327807. expected length is 3337719 and actual length is 3375106
2020-04-27 07:27:30.578 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 2264443519 and actual is checksum 2299249501. expected length is 104 and actual length is 102
2020-04-27 07:27:30.580 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 1184027188 and actual is checksum 2810465617. expected length is 508 and actual length is 508
2020-04-27 07:27:30.622 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 2246788287 and actual is checksum 2682862227. expected length is 92 and actual length is 96
2020-04-27 07:27:30.625 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 3227520043 and actual is checksum 2778425164. expected length is 107137 and actual length is 107137
2020-04-27 07:27:30.627 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 1964200539 and actual is checksum 3997289541. expected length is 7398 and actual length is 9042
2020-04-27 07:27:30.630 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 2165407577 and actual is checksum 2563543943. expected length is 110 and actual length is 110
2020-04-27 07:27:30.632 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1427836076 and actual is checksum 225462388. expected length is 11162 and actual length is 13623
2020-04-27 07:27:30.634 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 2998276982 and actual is checksum 1966428087. expected length is 296 and actual length is 353
2020-04-27 07:27:30.637 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 2717609594 and actual is checksum 673086945. expected length is 315491 and actual length is 312531
2020-04-27 07:27:30.640 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 2885031416 and actual is checksum 762513018. expected length is 508 and actual length is 508
2020-04-27 07:27:30.643 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected checksum is 602612626 and actual is checksum 2826680041. expected length is 1292092 and actual length is 1655171
2020-04-27 07:27:30.669 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 401385741 and actual is checksum 3373755803. expected length is 621 and actual length is 715
2020-04-27 07:27:30.671 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 3708405765 and actual is checksum 1057460545. expected length is 508 and actual length is 508
2020-04-27 07:27:30.674 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 2755907323 and actual is checksum 1931314059. expected length is 327125 and actual length is 318737
2020-04-27 07:27:30.677 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 77505520 and actual is checksum 2602905677. expected length is 107240 and actual length is 107460
2020-04-27 07:27:30.680 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 4204960097 and actual is checksum 2276943505. expected length is 110 and actual length is 110
2020-04-27 07:27:30.682 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 3414743450 and actual is checksum 3890346633. expected length is 18415 and actual length is 21922
2020-04-27 07:27:30.685 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 2879829129 and actual is checksum 790212338. expected length is 27860 and actual length is 33235
2020-04-27 07:27:30.687 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 3356980932 and actual is checksum 1544608917. expected length is 108 and actual length is 112
2020-04-27 07:27:30.690 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 1979425912 and actual is checksum 584894070. expected length is 3726110 and actual length is 4689653
2020-04-27 07:27:30.700 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is 3754169108 and actual is checksum 527001228. expected length is 508 and actual length is 508
2020-04-27 07:27:30.702 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected checksum is 3981710860 and actual is checksum 1749011618. expected length is 1938123 and actual length is 33694
2020-04-27 07:27:30.710 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is 1515379146 and actual is checksum 3212722427. expected length is 98 and actual length is 84
2020-04-27 07:27:30.712 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is 1819400085 and actual is checksum 3372088644. expected length is 107020 and actual length is 103887
2020-04-27 07:27:30.715 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected checksum is 1401562358 and actual is checksum 2224139078. expected length is 110 and actual length is 110
2020-04-27 07:27:30.718 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected checksum is 3386701421 and actual is checksum 1434467863. expected length is 15944 and actual length is 685
2020-04-27 07:27:30.720 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is 4259543754 and actual is checksum 3203989793. expected length is 10616 and actual length is 439
2020-04-27 07:27:30.722 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected checksum is 725882371 and actual is checksum 3650374401. expected length is 383 and actual length is 114
2020-04-27 07:27:30.725 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected checksum is 3717717087 and actual is checksum 3712473868. expected length is 303729 and actual length is 117582
2020-04-27 07:27:30.728 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 3506718582 and actual is checksum 342046803. expected length is 93 and actual length is 104
2020-04-27 07:27:30.731 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 2920971663 and actual is checksum 2142622028. expected length is 508 and actual length is 508
2020-04-27 07:27:30.733 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 723455769 and actual is checksum 2731538349. expected length is 110 and actual length is 110
2020-04-27 07:27:30.735 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 2624342764 and actual is checksum 1283943229. expected length is 11424 and actual length is 26688
2020-04-27 07:27:30.738 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 978416968 and actual is checksum 1807767534. expected length is 330 and actual length is 597
2020-04-27 07:27:30.740 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 3599774596 and actual is checksum 2764190632. expected length is 318017 and actual length is 311617
2020-04-27 07:27:30.743 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3141314417 and actual is checksum 3847788889. expected length is 1258075 and actual length is 3539994
2020-04-27 07:27:30.749 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 1404317797 and actual is checksum 2471843701. expected length is 105295 and actual length is 107020
2020-04-27 07:27:30.752 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 3412288246 and actual is checksum 922481525. expected length is 7580 and actual length is 17733
2020-04-27 07:27:30.754 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected checksum is 2675540020 and actual is checksum 1452209674. expected length is 110 and actual length is 110
2020-04-27 07:27:30.757 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected checksum is 665797990 and actual is checksum 2134373365. expected length is 10715 and actual length is 3418
2020-04-27 07:27:30.759 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is 2006427884 and actual is checksum 527297847. expected length is 7178 and actual length is 2326
2020-04-27 07:27:30.761 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is 200552893 and actual is checksum 3290522141. expected length is 106580 and actual length is 105206
2020-04-27 07:27:30.764 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected checksum is 1511033560 and actual is checksum 3689083691. expected length is 278 and actual length is 154
2020-04-27 07:27:30.767 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected checksum is 1435050119 and actual is checksum 6864243. expected length is 323217 and actual length is 338063
2020-04-27 07:27:30.770 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is 1107038327 and actual is checksum 3068870234. expected length is 508 and actual length is 508
2020-04-27 07:27:30.772 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is 2853806777 and actual is checksum 459866651. expected length is 93 and actual length is 87
2020-04-27 07:27:30.775 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected checksum is 1356537478 and actual is checksum 2594306289. expected length is 1179215 and actual length is 249458
2020-04-27 07:27:30.780 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 1785795479 and actual is checksum 65014613. expected length is 106580 and actual length is 106800
2020-04-27 07:27:30.783 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1824666616 and actual is checksum 2857527783. expected length is 507 and actual length is 637
2020-04-27 07:27:30.785 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 1384446542 and actual is checksum 3981590407. expected length is 13686 and actual length is 18516
2020-04-27 07:27:30.788 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 3380549662 and actual is checksum 2002436369. expected length is 317777 and actual length is 319213
2020-04-27 07:27:30.791 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 1594954666 and actual is checksum 1892723425. expected length is 110 and actual length is 110
2020-04-27 07:27:30.794 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 305271241 and actual is checksum 854278263. expected length is 20522 and actual length is 27836
2020-04-27 07:27:30.796 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 2220527769 and actual is checksum 2016379681. expected length is 508 and actual length is 508
2020-04-27 07:27:30.799 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1648054590 and actual is checksum 137025071. expected length is 100 and actual length is 105
2020-04-27 07:27:30.801 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 262113808 and actual is checksum 327562225. expected length is 2731808 and actual length is 3909012
2020-04-27 07:27:30.809 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected checksum is 2157667750 and actual is checksum 3043505208. expected length is 36906 and actual length is 163443
2020-04-27 07:27:30.812 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is 3456088243 and actual is checksum 1715362430. expected length is 453 and actual length is 1636
2020-04-27 07:27:30.814 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is 4028738113 and actual is checksum 4245480574. expected length is 104876 and actual length is 105515
2020-04-27 07:27:30.817 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected checksum is 3120632072 and actual is checksum 1945528118. expected length is 110 and actual length is 110
2020-04-27 07:27:30.819 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected checksum is 3303166970 and actual is checksum 1740028553. expected length is 114 and actual length is 137
2020-04-27 07:27:30.822 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected checksum is 126700369 and actual is checksum 162509958. expected length is 700 and actual length is 2398
2020-04-27 07:27:30.825 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is 3157051968 and actual is checksum 137679212. expected length is 84 and actual length is 86
2020-04-27 07:27:30.827 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected checksum is 3969090178 and actual is checksum 2364978843. expected length is 127576 and actual length is 266581
2020-04-27 07:27:30.830 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is 3730892058 and actual is checksum 2734980133. expected length is 508 and actual length is 508
2020-04-27 07:27:30.832 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is 3259266514 and actual is checksum 2051283039. expected length is 104855 and actual length is 106819
2020-04-27 07:27:30.835 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected checksum is 380153843 and actual is checksum 580580220. expected length is 1822 and actual length is 1737
2020-04-27 07:27:30.838 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected checksum is 4094454161 and actual is checksum 421930789. expected length is 110 and actual length is 110
2020-04-27 07:27:30.840 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is 911575783 and actual is checksum 2090965125. expected length is 1244 and actual length is 1201
2020-04-27 07:27:30.842 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected checksum is 1279063865 and actual is checksum 765869778. expected length is 128 and actual length is 130
2020-04-27 07:27:30.845 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected checksum is 1689040450 and actual is checksum 627554920. expected length is 223473 and actual length is 230157
2020-04-27 07:27:30.848 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is 1278110969 and actual is checksum 1444534251. expected length is 86 and actual length is 86
2020-04-27 07:27:30.850 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected checksum is 1992192696 and actual is checksum 3455180678. expected length is 119844 and actual length is 117860
2020-04-27 07:27:30.853 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is 2776236859 and actual is checksum 2572132550. expected length is 508 and actual length is 508
2020-04-27 07:27:35.866 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is 2573424453 and actual is checksum 3612352961. expected length is 546 and actual length is 437
2020-04-27 07:27:35.868 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is 2592320603 and actual is checksum 1374033039. expected length is 289 and actual length is 289
2020-04-27 07:27:35.872 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is 2911257988 and actual is checksum 2011202433. expected length is 437 and actual length is 437
2020-04-27 07:27:35.874 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is 1304802713 and actual is checksum 454030092. expected length is 7598425 and actual length is 49265169
2020-04-27 07:27:37.234 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is 1422458436 and actual is checksum 703149874. expected length is 437 and actual length is 437
2020-04-27 07:27:38.117 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 1892273984 and actual is checksum 2087644984. expected length is 50710602 and actual length is 10402264
2020-04-27 07:27:38.532 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is 3864394245 and actual is checksum 1846889307. expected length is 289 and actual length is 289
2020-04-27 07:27:38.575 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is 660375856 and actual is checksum 571052523. expected length is 114 and actual length is 84
2020-04-27 07:27:38.590 WARN (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
2020-04-27 07:27:39.259 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected checksum is 421440641 and actual is checksum 2075160524. expected length is 337361 and actual length is 141820
2020-04-27 07:27:44.881 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected checksum is 47189924 and actual is checksum 3640637009. expected length is 5235629 and actual length is 48679
2020-04-27 07:27:45.335 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected checksum is 2305624674 and actual is checksum 704932512. expected length is 693 and actual length is 115
2020-04-27 07:27:45.939 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is 1157089232 and actual is checksum 2388254324. expected length is 23435 and actual length is 673
2020-04-27 07:27:45.941 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is 2811885732 and actual is checksum 2854129172. expected length is 508 and actual length is 508
2020-04-27 07:27:45.943 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected checksum is 1925626873 and actual is checksum 1921615605. expected length is 110 and actual length is 110
2020-04-27 07:27:45.945 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected checksum is 3627283142 and actual is checksum 3796132842. expected length is 34596 and actual length is 958
2020-04-27 07:27:45.947 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is 1291024098 and actual is checksum 2700484857. expected length is 107020 and actual length is 104209
2020-04-27 07:27:45.949 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected checksum is 2785033346 and actual is checksum 1911387906. expected length is 816 and actual length is 760
2020-04-27 07:27:45.951 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected checksum is 942596692 and actual is checksum 3039336477. expected length is 5557044 and actual length is 4847564
2020-04-27 07:27:45.974 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is 143536399 and actual is checksum 2962535892. expected length is 116 and actual length is 111
2020-04-27 07:27:45.976 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected checksum is 3884436961 and actual is checksum 1345841996. expected length is 339801 and actual length is 336809
2020-04-27 07:27:45.980 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is 3902419774 and actual is checksum 3791519088. expected length is 23306 and actual length is 21047
2020-04-27 07:27:45.981 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is 45470103 and actual is checksum 235515609. expected length is 508 and actual length is 508
2020-04-27 07:27:45.983 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected checksum is 408594907 and actual is checksum 2650340971. expected length is 110 and actual length is 110
2020-04-27 07:27:45.985 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected checksum is 761408352 and actual is checksum 685575313. expected length is 35256 and actual length is 31644
2020-04-27 07:27:45.987 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is 811204953 and actual is checksum 1244308773. expected length is 107460 and actual length is 107460
2020-04-27 07:27:45.989 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is 3813441918 and actual is checksum 2133371346. expected length is 107460 and actual length is 106360
2020-04-27 07:27:45.991 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is 4263416408 and actual is checksum 3722461284. expected length is 16698 and actual length is 21392
2020-04-27 07:27:45.993 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected checksum is 632422377 and actual is checksum 501799146. expected length is 110 and actual length is 110
2020-04-27 07:27:45.994 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected checksum is 3170165892 and actual is checksum 4152937488. expected length is 25090 and actual length is 32328
2020-04-27 07:27:46.000 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected checksum is 2679120655 and actual is checksum 3243853760. expected length is 553 and actual length is 726
2020-04-27 07:27:46.001 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected checksum is 2991034554 and actual is checksum 1980969905. expected length is 321785 and actual length is 322117
2020-04-27 07:27:46.004 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 2117172713 and actual is checksum 4064616498. expected length is 3479018 and actual length is 4488695
2020-04-27 07:27:46.017 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 1613761705 and actual is checksum 4154864516. expected length is 106 and actual length is 111
2020-04-27 07:27:46.019 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 3059265880 and actual is checksum 531516038. expected length is 508 and actual length is 508
2020-04-27 07:27:46.021 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 1804496408 and actual is checksum 2903373400. expected length is 627 and actual length is 621
2020-04-27 07:27:46.023 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 2241359534 and actual is checksum 4198491027. expected length is 508 and actual length is 508
2020-04-27 07:27:46.024 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 3097209578 and actual is checksum 1124925868. expected length is 312617 and actual length is 323465
2020-04-27 07:27:46.027 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 1909140548 and actual is checksum 2397297432. expected length is 106580 and actual length is 107460
2020-04-27 07:27:46.029 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 971818792 and actual is checksum 478139506. expected length is 110 and actual length is 110
2020-04-27 07:27:46.031 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 1304288778 and actual is checksum 1544273325. expected length is 18170 and actual length is 17756
2020-04-27 07:27:46.033 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 4033259458 and actual is checksum 4016593800. expected length is 27532 and actual length is 26636
2020-04-27 07:27:46.034 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 385809858 and actual is checksum 3497071885. expected length is 107 and actual length is 104
2020-04-27 07:27:46.036 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 3567656601 and actual is checksum 2800338830. expected length is 3622054 and actual length is 3792928
2020-04-27 07:27:46.051 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 2282222236 and actual is checksum 3183044106. expected length is 98 and actual length is 108
2020-04-27 07:27:46.053 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 150588833 and actual is checksum 3492244992. expected length is 508 and actual length is 508
2020-04-27 07:27:46.055 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 1457615503 and actual is checksum 4089018914. expected length is 110 and actual length is 110
2020-04-27 07:27:46.125 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 3866656431 and actual is checksum 2014845387. expected length is 16763 and actual length is 28481
2020-04-27 07:27:46.128 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 3008320208 and actual is checksum 808227383. expected length is 419 and actual length is 659
2020-04-27 07:27:46.130 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 2437844680 and actual is checksum 1756584754. expected length is 310713 and actual length is 315009
2020-04-27 07:27:46.133 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3036548497 and actual is checksum 343807338. expected length is 2245614 and actual length is 4011325
2020-04-27 07:27:46.146 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 4233343814 and actual is checksum 2984252206. expected length is 106580 and actual length is 107460
2020-04-27 07:27:46.148 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 2222747552 and actual is checksum 3771791266. expected length is 11198 and actual length is 18927
2020-04-27 07:27:46.150 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 2062419431 and actual is checksum 885929423. expected length is 107460 and actual length is 105632
2020-04-27 07:27:46.152 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1574973164 and actual is checksum 209201565. expected length is 516 and actual length is 138
2020-04-27 07:27:46.154 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 3606049617 and actual is checksum 3605002010. expected length is 14213 and actual length is 1590
2020-04-27 07:27:46.156 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 842753199 and actual is checksum 3527006266. expected length is 316073 and actual length is 279943
2020-04-27 07:27:46.159 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 2130892015 and actual is checksum 2031251178. expected length is 110 and actual length is 110
2020-04-27 07:27:46.161 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 1435858498 and actual is checksum 3731425121. expected length is 21284 and actual length is 2331
2020-04-27 07:27:46.163 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 20126402 and actual is checksum 1544152005. expected length is 508 and actual length is 508
2020-04-27 07:27:46.165 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1651289033 and actual is checksum 1035172250. expected length is 100 and actual length is 86
2020-04-27 07:27:46.166 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 3180610843 and actual is checksum 2027612627. expected length is 2818109 and actual length is 159550
2020-04-27 07:27:52.815 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=1471697 inSortedOrder=false
2020-04-27 07:27:53.592 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=211445345 inSortedOrder=false
2020-04-27 07:27:56.965 WARN (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348}
2020-04-27 07:27:57.329 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345}
2020-04-27 07:28:03.537 WARN (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945}
2020-04-27 07:28:07.353 WARN (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052}
2020-04-27 07:28:19.721 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697}
2020-04-27 07:33:26.278 WARN (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.c.RecoveryStrategy Stopping recovery for core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39]
2020-04-27 07:33:30.370 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
2020-04-27 07:33:37.918 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
2020-04-27 07:33:37.945 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is 204474971 and actual is checksum 3417943085. expected length is 437 and actual length is 546
2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
2020-04-28 07:53:30.910 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10528ms for sessionid 0x171a6fb51310008
2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
Re: SolrCloud degraded during backup and batch CSV update
Posted by matthew sporleder <ms...@gmail.com>.
You can add something like this to SOLR_OPTS -DzkClientTimeout=30000
in your init script or adjust solr.xml <int
name="zkClientTimeout">${zkClientTimeout:15000}</int>
On Wed, Apr 29, 2020 at 5:41 PM Sethuraman, Ganesh
<Se...@dnb.com.invalid> wrote:
>
> 3 Zookeeper ensemble are all in 3 separate boxes (EC2 instances). Each have separate transactional logs directory (separate EBS volume, separate disk), as this was zookeeper best practices.
>
> It feels ZK timeout is more of symptom and Solr slowness is the cause. Having said that, do you increase the timeout setting in Solr or Zookeeper, if you can share parameters it will certainly help
>
> Regards
> Ganesh
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Wednesday, April 29, 2020 11:47 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud degraded during backup and batch CSV update
>
> CAUTION: This email originated from outside of D&B. Please do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
> FWIW I've had some luck with strategy 3 (increase zk timeout) when you
> overwhelm the connection to zk or the disk on zk.
>
> Is zk on the same boxes as solr?
>
> On Tue, Apr 28, 2020 at 10:15 PM Sethuraman, Ganesh
> <Se...@dnb.com.invalid> wrote:
> >
> > Hi
> >
> > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: The issue is SolrCloud server went into a degraded collections (for few collections) when the Solr backup and the Solr batch CSV update load happened at the same time as backup. The CSV data load was about ~5 GB per shard/replica. We think this happened after zkClient disconnect happened as noted below. We had to restart Solr to bring it back to normal.
> >
> >
> > 1. Is it not suggested to run backup and Solr batch CSV update large load at the same time?
> > 2. In the past we have seen two CSV batch update load in parallel causes issues, is this also not suggested (this issue is not related to that)?
> > 3. Do you think we should increase Zookeeper timeout?
> > 4. How do we know if we need to up the JVM Max memory, and by how much?
> > 5. We also see that once the Solr goes into degraded collection and recovery failed, it NEVER get back to normal, even after when there is no load. Is this a bug?
> >
> > The GC information and Solr Log below
> >
> > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C8e47fdc157ce4bf8425c08d7ec548d6c%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637237720241486583&sdata=MlstokkgBAX7joUpljJnQZjrbQ7cZZAoSPfQebx2q5I%3D&reserved=0
> >
> >
> > 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> > ....
> > 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> >
> >
> >
> >
> > SOLR Log Below (Curtailed WARN log)
> > --------------------
> > 2020-04-27 07:26:45.402 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.405 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.406 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.408 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.410 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error fetching file, doing one retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.414 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> > java.io.EOFException
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.414 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> > java.nio.file.NoSuchFileException: /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> > at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> > at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> > at java.nio.file.Files.delete(Files.java:1126)
> > at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> > at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> > at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> > at org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is 3934327071 and actual is checksum 3095753427. expected length is 33235371 and actual length is 54180642
> > 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is 138861107 and actual is checksum 188997082. expected length is 107460 and actual length is 107460
> > 2020-04-27 07:26:45.457 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected checksum is 2947323803 and actual is checksum 979431341. expected length is 32176 and actual length is 26089
> > 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected checksum is 2087731087 and actual is checksum 3540174119. expected length is 110 and actual length is 110
> > 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 777834759 and actual is checksum 289576304. expected length is 59980991 and actual length is 6236424
> > 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is 1321210176 and actual is checksum 224636564. expected length is 114 and actual length is 106
> > 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is 3790133021 and actual is checksum 2516047051. expected length is 1477755 and actual length is 1477850
> > 2020-04-27 07:26:45.578 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected checksum is 3690299979 and actual is checksum 411724865. expected length is 4392270 and actual length is 4396498
> > 2020-04-27 07:26:45.588 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected checksum is 937224718 and actual is checksum 1176441880. expected length is 110 and actual length is 110
> > 2020-04-27 07:26:45.733 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected checksum is 1429458386 and actual is checksum 4155623862. expected length is 2213909 and actual length is 2213939
> > 2020-04-27 07:26:45.754 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
> > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> > at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> > 2020-04-27 07:26:45.780 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected checksum is 1474431645 and actual is checksum 1271168044. expected length is 275570 and actual length is 274506
> > 2020-04-27 07:26:45.834 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is 4129522944 and actual is checksum 4080519415. expected length is 6626 and actual length is 6626
> > 2020-04-27 07:26:45.843 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is 2493366746 and actual is checksum 31587394. expected length is 546 and actual length is 546
> > 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 1392813257 and actual is checksum 3455988143. expected length is 103 and actual length is 111
> > 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected checksum is 2301226889 and actual is checksum 29696594. expected length is 86459 and actual length is 85786
> > 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected checksum is 112676849 and actual is checksum 354222195. expected length is 286474 and actual length is 286546
> > 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 2203892485 and actual is checksum 1165086749. expected length is 107460 and actual length is 106580
> > 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is 3175370827 and actual is checksum 520836483. expected length is 51396 and actual length is 51404
> > 2020-04-27 07:26:45.912 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is 967260408 and actual is checksum 969726862. expected length is 3180 and actual length is 3180
> > 2020-04-27 07:26:45.914 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 719562508 and actual is checksum 956062095. expected length is 16717 and actual length is 21454
> > 2020-04-27 07:26:45.915 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected checksum is 3833337361 and actual is checksum 493358852. expected length is 43996 and actual length is 44026
> > 2020-04-27 07:26:45.916 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 1533628328 and actual is checksum 3735455395. expected length is 110 and actual length is 110
> > 2020-04-27 07:26:45.917 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected checksum is 75152145 and actual is checksum 1665090946. expected length is 497241188 and actual length is 497254814
> > 2020-04-27 07:26:45.918 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1349491281 and actual is checksum 3808036883. expected length is 25153 and actual length is 32322
> > 2020-04-27 07:26:45.920 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 3820733560 and actual is checksum 2819887160. expected length is 571 and actual length is 725
> > 2020-04-27 07:26:46.034 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 1547030244 and actual is checksum 1122196351. expected length is 324313 and actual length is 323553
> > 2020-04-27 07:26:46.170 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 751850967 and actual is checksum 1026527313. expected length is 508 and actual length is 508
> > and actual length is 319133
> > 2020-04-27 07:27:30.567 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 3435029878 and actual is checksum 1202327807. expected length is 3337719 and actual length is 3375106
> > 2020-04-27 07:27:30.578 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 2264443519 and actual is checksum 2299249501. expected length is 104 and actual length is 102
> > 2020-04-27 07:27:30.580 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 1184027188 and actual is checksum 2810465617. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.622 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 2246788287 and actual is checksum 2682862227. expected length is 92 and actual length is 96
> > 2020-04-27 07:27:30.625 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 3227520043 and actual is checksum 2778425164. expected length is 107137 and actual length is 107137
> > 2020-04-27 07:27:30.627 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 1964200539 and actual is checksum 3997289541. expected length is 7398 and actual length is 9042
> > 2020-04-27 07:27:30.630 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 2165407577 and actual is checksum 2563543943. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.632 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1427836076 and actual is checksum 225462388. expected length is 11162 and actual length is 13623
> > 2020-04-27 07:27:30.634 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 2998276982 and actual is checksum 1966428087. expected length is 296 and actual length is 353
> > 2020-04-27 07:27:30.637 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 2717609594 and actual is checksum 673086945. expected length is 315491 and actual length is 312531
> > 2020-04-27 07:27:30.640 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 2885031416 and actual is checksum 762513018. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.643 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected checksum is 602612626 and actual is checksum 2826680041. expected length is 1292092 and actual length is 1655171
> > 2020-04-27 07:27:30.669 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 401385741 and actual is checksum 3373755803. expected length is 621 and actual length is 715
> > 2020-04-27 07:27:30.671 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 3708405765 and actual is checksum 1057460545. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.674 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 2755907323 and actual is checksum 1931314059. expected length is 327125 and actual length is 318737
> > 2020-04-27 07:27:30.677 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 77505520 and actual is checksum 2602905677. expected length is 107240 and actual length is 107460
> > 2020-04-27 07:27:30.680 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 4204960097 and actual is checksum 2276943505. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.682 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 3414743450 and actual is checksum 3890346633. expected length is 18415 and actual length is 21922
> > 2020-04-27 07:27:30.685 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 2879829129 and actual is checksum 790212338. expected length is 27860 and actual length is 33235
> > 2020-04-27 07:27:30.687 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 3356980932 and actual is checksum 1544608917. expected length is 108 and actual length is 112
> > 2020-04-27 07:27:30.690 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 1979425912 and actual is checksum 584894070. expected length is 3726110 and actual length is 4689653
> > 2020-04-27 07:27:30.700 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is 3754169108 and actual is checksum 527001228. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.702 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected checksum is 3981710860 and actual is checksum 1749011618. expected length is 1938123 and actual length is 33694
> > 2020-04-27 07:27:30.710 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is 1515379146 and actual is checksum 3212722427. expected length is 98 and actual length is 84
> > 2020-04-27 07:27:30.712 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is 1819400085 and actual is checksum 3372088644. expected length is 107020 and actual length is 103887
> > 2020-04-27 07:27:30.715 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected checksum is 1401562358 and actual is checksum 2224139078. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.718 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected checksum is 3386701421 and actual is checksum 1434467863. expected length is 15944 and actual length is 685
> > 2020-04-27 07:27:30.720 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is 4259543754 and actual is checksum 3203989793. expected length is 10616 and actual length is 439
> > 2020-04-27 07:27:30.722 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected checksum is 725882371 and actual is checksum 3650374401. expected length is 383 and actual length is 114
> > 2020-04-27 07:27:30.725 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected checksum is 3717717087 and actual is checksum 3712473868. expected length is 303729 and actual length is 117582
> > 2020-04-27 07:27:30.728 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 3506718582 and actual is checksum 342046803. expected length is 93 and actual length is 104
> > 2020-04-27 07:27:30.731 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 2920971663 and actual is checksum 2142622028. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.733 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 723455769 and actual is checksum 2731538349. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.735 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 2624342764 and actual is checksum 1283943229. expected length is 11424 and actual length is 26688
> > 2020-04-27 07:27:30.738 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 978416968 and actual is checksum 1807767534. expected length is 330 and actual length is 597
> > 2020-04-27 07:27:30.740 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 3599774596 and actual is checksum 2764190632. expected length is 318017 and actual length is 311617
> > 2020-04-27 07:27:30.743 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3141314417 and actual is checksum 3847788889. expected length is 1258075 and actual length is 3539994
> > 2020-04-27 07:27:30.749 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 1404317797 and actual is checksum 2471843701. expected length is 105295 and actual length is 107020
> > 2020-04-27 07:27:30.752 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 3412288246 and actual is checksum 922481525. expected length is 7580 and actual length is 17733
> > 2020-04-27 07:27:30.754 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected checksum is 2675540020 and actual is checksum 1452209674. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.757 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected checksum is 665797990 and actual is checksum 2134373365. expected length is 10715 and actual length is 3418
> > 2020-04-27 07:27:30.759 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is 2006427884 and actual is checksum 527297847. expected length is 7178 and actual length is 2326
> > 2020-04-27 07:27:30.761 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is 200552893 and actual is checksum 3290522141. expected length is 106580 and actual length is 105206
> > 2020-04-27 07:27:30.764 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected checksum is 1511033560 and actual is checksum 3689083691. expected length is 278 and actual length is 154
> > 2020-04-27 07:27:30.767 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected checksum is 1435050119 and actual is checksum 6864243. expected length is 323217 and actual length is 338063
> > 2020-04-27 07:27:30.770 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is 1107038327 and actual is checksum 3068870234. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.772 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is 2853806777 and actual is checksum 459866651. expected length is 93 and actual length is 87
> > 2020-04-27 07:27:30.775 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected checksum is 1356537478 and actual is checksum 2594306289. expected length is 1179215 and actual length is 249458
> > 2020-04-27 07:27:30.780 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 1785795479 and actual is checksum 65014613. expected length is 106580 and actual length is 106800
> > 2020-04-27 07:27:30.783 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1824666616 and actual is checksum 2857527783. expected length is 507 and actual length is 637
> > 2020-04-27 07:27:30.785 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 1384446542 and actual is checksum 3981590407. expected length is 13686 and actual length is 18516
> > 2020-04-27 07:27:30.788 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 3380549662 and actual is checksum 2002436369. expected length is 317777 and actual length is 319213
> > 2020-04-27 07:27:30.791 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 1594954666 and actual is checksum 1892723425. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.794 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 305271241 and actual is checksum 854278263. expected length is 20522 and actual length is 27836
> > 2020-04-27 07:27:30.796 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 2220527769 and actual is checksum 2016379681. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.799 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1648054590 and actual is checksum 137025071. expected length is 100 and actual length is 105
> > 2020-04-27 07:27:30.801 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 262113808 and actual is checksum 327562225. expected length is 2731808 and actual length is 3909012
> > 2020-04-27 07:27:30.809 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected checksum is 2157667750 and actual is checksum 3043505208. expected length is 36906 and actual length is 163443
> > 2020-04-27 07:27:30.812 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is 3456088243 and actual is checksum 1715362430. expected length is 453 and actual length is 1636
> > 2020-04-27 07:27:30.814 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is 4028738113 and actual is checksum 4245480574. expected length is 104876 and actual length is 105515
> > 2020-04-27 07:27:30.817 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected checksum is 3120632072 and actual is checksum 1945528118. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.819 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected checksum is 3303166970 and actual is checksum 1740028553. expected length is 114 and actual length is 137
> > 2020-04-27 07:27:30.822 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected checksum is 126700369 and actual is checksum 162509958. expected length is 700 and actual length is 2398
> > 2020-04-27 07:27:30.825 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is 3157051968 and actual is checksum 137679212. expected length is 84 and actual length is 86
> > 2020-04-27 07:27:30.827 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected checksum is 3969090178 and actual is checksum 2364978843. expected length is 127576 and actual length is 266581
> > 2020-04-27 07:27:30.830 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is 3730892058 and actual is checksum 2734980133. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:30.832 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is 3259266514 and actual is checksum 2051283039. expected length is 104855 and actual length is 106819
> > 2020-04-27 07:27:30.835 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected checksum is 380153843 and actual is checksum 580580220. expected length is 1822 and actual length is 1737
> > 2020-04-27 07:27:30.838 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected checksum is 4094454161 and actual is checksum 421930789. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:30.840 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is 911575783 and actual is checksum 2090965125. expected length is 1244 and actual length is 1201
> > 2020-04-27 07:27:30.842 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected checksum is 1279063865 and actual is checksum 765869778. expected length is 128 and actual length is 130
> > 2020-04-27 07:27:30.845 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected checksum is 1689040450 and actual is checksum 627554920. expected length is 223473 and actual length is 230157
> > 2020-04-27 07:27:30.848 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is 1278110969 and actual is checksum 1444534251. expected length is 86 and actual length is 86
> > 2020-04-27 07:27:30.850 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected checksum is 1992192696 and actual is checksum 3455180678. expected length is 119844 and actual length is 117860
> > 2020-04-27 07:27:30.853 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is 2776236859 and actual is checksum 2572132550. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:35.866 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is 2573424453 and actual is checksum 3612352961. expected length is 546 and actual length is 437
> > 2020-04-27 07:27:35.868 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is 2592320603 and actual is checksum 1374033039. expected length is 289 and actual length is 289
> > 2020-04-27 07:27:35.872 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is 2911257988 and actual is checksum 2011202433. expected length is 437 and actual length is 437
> > 2020-04-27 07:27:35.874 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is 1304802713 and actual is checksum 454030092. expected length is 7598425 and actual length is 49265169
> > 2020-04-27 07:27:37.234 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is 1422458436 and actual is checksum 703149874. expected length is 437 and actual length is 437
> > 2020-04-27 07:27:38.117 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 1892273984 and actual is checksum 2087644984. expected length is 50710602 and actual length is 10402264
> > 2020-04-27 07:27:38.532 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is 3864394245 and actual is checksum 1846889307. expected length is 289 and actual length is 289
> > 2020-04-27 07:27:38.575 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is 660375856 and actual is checksum 571052523. expected length is 114 and actual length is 84
> > 2020-04-27 07:27:38.590 WARN (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> > 2020-04-27 07:27:39.259 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected checksum is 421440641 and actual is checksum 2075160524. expected length is 337361 and actual length is 141820
> > 2020-04-27 07:27:44.881 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected checksum is 47189924 and actual is checksum 3640637009. expected length is 5235629 and actual length is 48679
> > 2020-04-27 07:27:45.335 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected checksum is 2305624674 and actual is checksum 704932512. expected length is 693 and actual length is 115
> > 2020-04-27 07:27:45.939 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is 1157089232 and actual is checksum 2388254324. expected length is 23435 and actual length is 673
> > 2020-04-27 07:27:45.941 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is 2811885732 and actual is checksum 2854129172. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:45.943 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected checksum is 1925626873 and actual is checksum 1921615605. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:45.945 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected checksum is 3627283142 and actual is checksum 3796132842. expected length is 34596 and actual length is 958
> > 2020-04-27 07:27:45.947 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is 1291024098 and actual is checksum 2700484857. expected length is 107020 and actual length is 104209
> > 2020-04-27 07:27:45.949 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected checksum is 2785033346 and actual is checksum 1911387906. expected length is 816 and actual length is 760
> > 2020-04-27 07:27:45.951 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected checksum is 942596692 and actual is checksum 3039336477. expected length is 5557044 and actual length is 4847564
> > 2020-04-27 07:27:45.974 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is 143536399 and actual is checksum 2962535892. expected length is 116 and actual length is 111
> > 2020-04-27 07:27:45.976 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected checksum is 3884436961 and actual is checksum 1345841996. expected length is 339801 and actual length is 336809
> > 2020-04-27 07:27:45.980 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is 3902419774 and actual is checksum 3791519088. expected length is 23306 and actual length is 21047
> > 2020-04-27 07:27:45.981 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is 45470103 and actual is checksum 235515609. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:45.983 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected checksum is 408594907 and actual is checksum 2650340971. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:45.985 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected checksum is 761408352 and actual is checksum 685575313. expected length is 35256 and actual length is 31644
> > 2020-04-27 07:27:45.987 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is 811204953 and actual is checksum 1244308773. expected length is 107460 and actual length is 107460
> > 2020-04-27 07:27:45.989 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is 3813441918 and actual is checksum 2133371346. expected length is 107460 and actual length is 106360
> > 2020-04-27 07:27:45.991 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is 4263416408 and actual is checksum 3722461284. expected length is 16698 and actual length is 21392
> > 2020-04-27 07:27:45.993 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected checksum is 632422377 and actual is checksum 501799146. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:45.994 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected checksum is 3170165892 and actual is checksum 4152937488. expected length is 25090 and actual length is 32328
> > 2020-04-27 07:27:46.000 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected checksum is 2679120655 and actual is checksum 3243853760. expected length is 553 and actual length is 726
> > 2020-04-27 07:27:46.001 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected checksum is 2991034554 and actual is checksum 1980969905. expected length is 321785 and actual length is 322117
> > 2020-04-27 07:27:46.004 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 2117172713 and actual is checksum 4064616498. expected length is 3479018 and actual length is 4488695
> > 2020-04-27 07:27:46.017 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 1613761705 and actual is checksum 4154864516. expected length is 106 and actual length is 111
> > 2020-04-27 07:27:46.019 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 3059265880 and actual is checksum 531516038. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:46.021 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 1804496408 and actual is checksum 2903373400. expected length is 627 and actual length is 621
> > 2020-04-27 07:27:46.023 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 2241359534 and actual is checksum 4198491027. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:46.024 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 3097209578 and actual is checksum 1124925868. expected length is 312617 and actual length is 323465
> > 2020-04-27 07:27:46.027 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 1909140548 and actual is checksum 2397297432. expected length is 106580 and actual length is 107460
> > 2020-04-27 07:27:46.029 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 971818792 and actual is checksum 478139506. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:46.031 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 1304288778 and actual is checksum 1544273325. expected length is 18170 and actual length is 17756
> > 2020-04-27 07:27:46.033 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 4033259458 and actual is checksum 4016593800. expected length is 27532 and actual length is 26636
> > 2020-04-27 07:27:46.034 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 385809858 and actual is checksum 3497071885. expected length is 107 and actual length is 104
> > 2020-04-27 07:27:46.036 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 3567656601 and actual is checksum 2800338830. expected length is 3622054 and actual length is 3792928
> > 2020-04-27 07:27:46.051 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 2282222236 and actual is checksum 3183044106. expected length is 98 and actual length is 108
> > 2020-04-27 07:27:46.053 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 150588833 and actual is checksum 3492244992. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:46.055 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 1457615503 and actual is checksum 4089018914. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:46.125 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 3866656431 and actual is checksum 2014845387. expected length is 16763 and actual length is 28481
> > 2020-04-27 07:27:46.128 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 3008320208 and actual is checksum 808227383. expected length is 419 and actual length is 659
> > 2020-04-27 07:27:46.130 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 2437844680 and actual is checksum 1756584754. expected length is 310713 and actual length is 315009
> > 2020-04-27 07:27:46.133 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3036548497 and actual is checksum 343807338. expected length is 2245614 and actual length is 4011325
> > 2020-04-27 07:27:46.146 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 4233343814 and actual is checksum 2984252206. expected length is 106580 and actual length is 107460
> > 2020-04-27 07:27:46.148 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 2222747552 and actual is checksum 3771791266. expected length is 11198 and actual length is 18927
> > 2020-04-27 07:27:46.150 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 2062419431 and actual is checksum 885929423. expected length is 107460 and actual length is 105632
> > 2020-04-27 07:27:46.152 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1574973164 and actual is checksum 209201565. expected length is 516 and actual length is 138
> > 2020-04-27 07:27:46.154 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 3606049617 and actual is checksum 3605002010. expected length is 14213 and actual length is 1590
> > 2020-04-27 07:27:46.156 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 842753199 and actual is checksum 3527006266. expected length is 316073 and actual length is 279943
> > 2020-04-27 07:27:46.159 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 2130892015 and actual is checksum 2031251178. expected length is 110 and actual length is 110
> > 2020-04-27 07:27:46.161 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 1435858498 and actual is checksum 3731425121. expected length is 21284 and actual length is 2331
> > 2020-04-27 07:27:46.163 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 20126402 and actual is checksum 1544152005. expected length is 508 and actual length is 508
> > 2020-04-27 07:27:46.165 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1651289033 and actual is checksum 1035172250. expected length is 100 and actual length is 86
> > 2020-04-27 07:27:46.166 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 3180610843 and actual is checksum 2027612627. expected length is 2818109 and actual length is 159550
> > 2020-04-27 07:27:52.815 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=1471697 inSortedOrder=false
> > 2020-04-27 07:27:53.592 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=211445345 inSortedOrder=false
> > 2020-04-27 07:27:56.965 WARN (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348}
> > 2020-04-27 07:27:57.329 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345}
> > 2020-04-27 07:28:03.537 WARN (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945}
> > 2020-04-27 07:28:07.353 WARN (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052}
> > 2020-04-27 07:28:19.721 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697}
> > 2020-04-27 07:33:26.278 WARN (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.c.RecoveryStrategy Stopping recovery for core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39]
> > 2020-04-27 07:33:30.370 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> > 2020-04-27 07:33:37.918 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> > 2020-04-27 07:33:37.945 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is 204474971 and actual is checksum 3417943085. expected length is 437 and actual length is 546
> > 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> > 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> > 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> > 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> > 2020-04-28 07:53:30.910 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10528ms for sessionid 0x171a6fb51310008
> > 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> > 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
RE: SolrCloud degraded during backup and batch CSV update
Posted by "Sethuraman, Ganesh" <Se...@DNB.com.INVALID>.
3 Zookeeper ensemble are all in 3 separate boxes (EC2 instances). Each have separate transactional logs directory (separate EBS volume, separate disk), as this was zookeeper best practices.
It feels ZK timeout is more of symptom and Solr slowness is the cause. Having said that, do you increase the timeout setting in Solr or Zookeeper, if you can share parameters it will certainly help
Regards
Ganesh
-----Original Message-----
From: matthew sporleder <ms...@gmail.com>
Sent: Wednesday, April 29, 2020 11:47 AM
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud degraded during backup and batch CSV update
CAUTION: This email originated from outside of D&B. Please do not click links or open attachments unless you recognize the sender and know the content is safe.
FWIW I've had some luck with strategy 3 (increase zk timeout) when you
overwhelm the connection to zk or the disk on zk.
Is zk on the same boxes as solr?
On Tue, Apr 28, 2020 at 10:15 PM Sethuraman, Ganesh
<Se...@dnb.com.invalid> wrote:
>
> Hi
>
> We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: The issue is SolrCloud server went into a degraded collections (for few collections) when the Solr backup and the Solr batch CSV update load happened at the same time as backup. The CSV data load was about ~5 GB per shard/replica. We think this happened after zkClient disconnect happened as noted below. We had to restart Solr to bring it back to normal.
>
>
> 1. Is it not suggested to run backup and Solr batch CSV update large load at the same time?
> 2. In the past we have seen two CSV batch update load in parallel causes issues, is this also not suggested (this issue is not related to that)?
> 3. Do you think we should increase Zookeeper timeout?
> 4. How do we know if we need to up the JVM Max memory, and by how much?
> 5. We also see that once the Solr goes into degraded collection and recovery failed, it NEVER get back to normal, even after when there is no load. Is this a bug?
>
> The GC information and Solr Log below
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C8e47fdc157ce4bf8425c08d7ec548d6c%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637237720241486583&sdata=MlstokkgBAX7joUpljJnQZjrbQ7cZZAoSPfQebx2q5I%3D&reserved=0
>
>
> 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> ....
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
>
>
>
>
> SOLR Log Below (Curtailed WARN log)
> --------------------
> 2020-04-27 07:26:45.402 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.405 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.406 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.408 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.410 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error fetching file, doing one retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.414 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.414 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> java.nio.file.NoSuchFileException: /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> at org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is 3934327071 and actual is checksum 3095753427. expected length is 33235371 and actual length is 54180642
> 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is 138861107 and actual is checksum 188997082. expected length is 107460 and actual length is 107460
> 2020-04-27 07:26:45.457 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected checksum is 2947323803 and actual is checksum 979431341. expected length is 32176 and actual length is 26089
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected checksum is 2087731087 and actual is checksum 3540174119. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 777834759 and actual is checksum 289576304. expected length is 59980991 and actual length is 6236424
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is 1321210176 and actual is checksum 224636564. expected length is 114 and actual length is 106
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is 3790133021 and actual is checksum 2516047051. expected length is 1477755 and actual length is 1477850
> 2020-04-27 07:26:45.578 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected checksum is 3690299979 and actual is checksum 411724865. expected length is 4392270 and actual length is 4396498
> 2020-04-27 07:26:45.588 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected checksum is 937224718 and actual is checksum 1176441880. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.733 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected checksum is 1429458386 and actual is checksum 4155623862. expected length is 2213909 and actual length is 2213939
> 2020-04-27 07:26:45.754 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> 2020-04-27 07:26:45.780 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected checksum is 1474431645 and actual is checksum 1271168044. expected length is 275570 and actual length is 274506
> 2020-04-27 07:26:45.834 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is 4129522944 and actual is checksum 4080519415. expected length is 6626 and actual length is 6626
> 2020-04-27 07:26:45.843 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is 2493366746 and actual is checksum 31587394. expected length is 546 and actual length is 546
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 1392813257 and actual is checksum 3455988143. expected length is 103 and actual length is 111
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected checksum is 2301226889 and actual is checksum 29696594. expected length is 86459 and actual length is 85786
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected checksum is 112676849 and actual is checksum 354222195. expected length is 286474 and actual length is 286546
> 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 2203892485 and actual is checksum 1165086749. expected length is 107460 and actual length is 106580
> 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is 3175370827 and actual is checksum 520836483. expected length is 51396 and actual length is 51404
> 2020-04-27 07:26:45.912 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is 967260408 and actual is checksum 969726862. expected length is 3180 and actual length is 3180
> 2020-04-27 07:26:45.914 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 719562508 and actual is checksum 956062095. expected length is 16717 and actual length is 21454
> 2020-04-27 07:26:45.915 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected checksum is 3833337361 and actual is checksum 493358852. expected length is 43996 and actual length is 44026
> 2020-04-27 07:26:45.916 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 1533628328 and actual is checksum 3735455395. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.917 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected checksum is 75152145 and actual is checksum 1665090946. expected length is 497241188 and actual length is 497254814
> 2020-04-27 07:26:45.918 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1349491281 and actual is checksum 3808036883. expected length is 25153 and actual length is 32322
> 2020-04-27 07:26:45.920 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 3820733560 and actual is checksum 2819887160. expected length is 571 and actual length is 725
> 2020-04-27 07:26:46.034 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 1547030244 and actual is checksum 1122196351. expected length is 324313 and actual length is 323553
> 2020-04-27 07:26:46.170 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 751850967 and actual is checksum 1026527313. expected length is 508 and actual length is 508
> and actual length is 319133
> 2020-04-27 07:27:30.567 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 3435029878 and actual is checksum 1202327807. expected length is 3337719 and actual length is 3375106
> 2020-04-27 07:27:30.578 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 2264443519 and actual is checksum 2299249501. expected length is 104 and actual length is 102
> 2020-04-27 07:27:30.580 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 1184027188 and actual is checksum 2810465617. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.622 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 2246788287 and actual is checksum 2682862227. expected length is 92 and actual length is 96
> 2020-04-27 07:27:30.625 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 3227520043 and actual is checksum 2778425164. expected length is 107137 and actual length is 107137
> 2020-04-27 07:27:30.627 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 1964200539 and actual is checksum 3997289541. expected length is 7398 and actual length is 9042
> 2020-04-27 07:27:30.630 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 2165407577 and actual is checksum 2563543943. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.632 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1427836076 and actual is checksum 225462388. expected length is 11162 and actual length is 13623
> 2020-04-27 07:27:30.634 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 2998276982 and actual is checksum 1966428087. expected length is 296 and actual length is 353
> 2020-04-27 07:27:30.637 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 2717609594 and actual is checksum 673086945. expected length is 315491 and actual length is 312531
> 2020-04-27 07:27:30.640 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 2885031416 and actual is checksum 762513018. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.643 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected checksum is 602612626 and actual is checksum 2826680041. expected length is 1292092 and actual length is 1655171
> 2020-04-27 07:27:30.669 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 401385741 and actual is checksum 3373755803. expected length is 621 and actual length is 715
> 2020-04-27 07:27:30.671 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 3708405765 and actual is checksum 1057460545. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.674 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 2755907323 and actual is checksum 1931314059. expected length is 327125 and actual length is 318737
> 2020-04-27 07:27:30.677 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 77505520 and actual is checksum 2602905677. expected length is 107240 and actual length is 107460
> 2020-04-27 07:27:30.680 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 4204960097 and actual is checksum 2276943505. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.682 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 3414743450 and actual is checksum 3890346633. expected length is 18415 and actual length is 21922
> 2020-04-27 07:27:30.685 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 2879829129 and actual is checksum 790212338. expected length is 27860 and actual length is 33235
> 2020-04-27 07:27:30.687 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 3356980932 and actual is checksum 1544608917. expected length is 108 and actual length is 112
> 2020-04-27 07:27:30.690 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 1979425912 and actual is checksum 584894070. expected length is 3726110 and actual length is 4689653
> 2020-04-27 07:27:30.700 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is 3754169108 and actual is checksum 527001228. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.702 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected checksum is 3981710860 and actual is checksum 1749011618. expected length is 1938123 and actual length is 33694
> 2020-04-27 07:27:30.710 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is 1515379146 and actual is checksum 3212722427. expected length is 98 and actual length is 84
> 2020-04-27 07:27:30.712 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is 1819400085 and actual is checksum 3372088644. expected length is 107020 and actual length is 103887
> 2020-04-27 07:27:30.715 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected checksum is 1401562358 and actual is checksum 2224139078. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.718 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected checksum is 3386701421 and actual is checksum 1434467863. expected length is 15944 and actual length is 685
> 2020-04-27 07:27:30.720 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is 4259543754 and actual is checksum 3203989793. expected length is 10616 and actual length is 439
> 2020-04-27 07:27:30.722 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected checksum is 725882371 and actual is checksum 3650374401. expected length is 383 and actual length is 114
> 2020-04-27 07:27:30.725 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected checksum is 3717717087 and actual is checksum 3712473868. expected length is 303729 and actual length is 117582
> 2020-04-27 07:27:30.728 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 3506718582 and actual is checksum 342046803. expected length is 93 and actual length is 104
> 2020-04-27 07:27:30.731 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 2920971663 and actual is checksum 2142622028. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.733 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 723455769 and actual is checksum 2731538349. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.735 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 2624342764 and actual is checksum 1283943229. expected length is 11424 and actual length is 26688
> 2020-04-27 07:27:30.738 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 978416968 and actual is checksum 1807767534. expected length is 330 and actual length is 597
> 2020-04-27 07:27:30.740 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 3599774596 and actual is checksum 2764190632. expected length is 318017 and actual length is 311617
> 2020-04-27 07:27:30.743 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3141314417 and actual is checksum 3847788889. expected length is 1258075 and actual length is 3539994
> 2020-04-27 07:27:30.749 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 1404317797 and actual is checksum 2471843701. expected length is 105295 and actual length is 107020
> 2020-04-27 07:27:30.752 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 3412288246 and actual is checksum 922481525. expected length is 7580 and actual length is 17733
> 2020-04-27 07:27:30.754 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected checksum is 2675540020 and actual is checksum 1452209674. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.757 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected checksum is 665797990 and actual is checksum 2134373365. expected length is 10715 and actual length is 3418
> 2020-04-27 07:27:30.759 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is 2006427884 and actual is checksum 527297847. expected length is 7178 and actual length is 2326
> 2020-04-27 07:27:30.761 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is 200552893 and actual is checksum 3290522141. expected length is 106580 and actual length is 105206
> 2020-04-27 07:27:30.764 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected checksum is 1511033560 and actual is checksum 3689083691. expected length is 278 and actual length is 154
> 2020-04-27 07:27:30.767 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected checksum is 1435050119 and actual is checksum 6864243. expected length is 323217 and actual length is 338063
> 2020-04-27 07:27:30.770 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is 1107038327 and actual is checksum 3068870234. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.772 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is 2853806777 and actual is checksum 459866651. expected length is 93 and actual length is 87
> 2020-04-27 07:27:30.775 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected checksum is 1356537478 and actual is checksum 2594306289. expected length is 1179215 and actual length is 249458
> 2020-04-27 07:27:30.780 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 1785795479 and actual is checksum 65014613. expected length is 106580 and actual length is 106800
> 2020-04-27 07:27:30.783 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1824666616 and actual is checksum 2857527783. expected length is 507 and actual length is 637
> 2020-04-27 07:27:30.785 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 1384446542 and actual is checksum 3981590407. expected length is 13686 and actual length is 18516
> 2020-04-27 07:27:30.788 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 3380549662 and actual is checksum 2002436369. expected length is 317777 and actual length is 319213
> 2020-04-27 07:27:30.791 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 1594954666 and actual is checksum 1892723425. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.794 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 305271241 and actual is checksum 854278263. expected length is 20522 and actual length is 27836
> 2020-04-27 07:27:30.796 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 2220527769 and actual is checksum 2016379681. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.799 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1648054590 and actual is checksum 137025071. expected length is 100 and actual length is 105
> 2020-04-27 07:27:30.801 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 262113808 and actual is checksum 327562225. expected length is 2731808 and actual length is 3909012
> 2020-04-27 07:27:30.809 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected checksum is 2157667750 and actual is checksum 3043505208. expected length is 36906 and actual length is 163443
> 2020-04-27 07:27:30.812 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is 3456088243 and actual is checksum 1715362430. expected length is 453 and actual length is 1636
> 2020-04-27 07:27:30.814 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is 4028738113 and actual is checksum 4245480574. expected length is 104876 and actual length is 105515
> 2020-04-27 07:27:30.817 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected checksum is 3120632072 and actual is checksum 1945528118. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.819 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected checksum is 3303166970 and actual is checksum 1740028553. expected length is 114 and actual length is 137
> 2020-04-27 07:27:30.822 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected checksum is 126700369 and actual is checksum 162509958. expected length is 700 and actual length is 2398
> 2020-04-27 07:27:30.825 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is 3157051968 and actual is checksum 137679212. expected length is 84 and actual length is 86
> 2020-04-27 07:27:30.827 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected checksum is 3969090178 and actual is checksum 2364978843. expected length is 127576 and actual length is 266581
> 2020-04-27 07:27:30.830 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is 3730892058 and actual is checksum 2734980133. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.832 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is 3259266514 and actual is checksum 2051283039. expected length is 104855 and actual length is 106819
> 2020-04-27 07:27:30.835 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected checksum is 380153843 and actual is checksum 580580220. expected length is 1822 and actual length is 1737
> 2020-04-27 07:27:30.838 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected checksum is 4094454161 and actual is checksum 421930789. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.840 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is 911575783 and actual is checksum 2090965125. expected length is 1244 and actual length is 1201
> 2020-04-27 07:27:30.842 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected checksum is 1279063865 and actual is checksum 765869778. expected length is 128 and actual length is 130
> 2020-04-27 07:27:30.845 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected checksum is 1689040450 and actual is checksum 627554920. expected length is 223473 and actual length is 230157
> 2020-04-27 07:27:30.848 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is 1278110969 and actual is checksum 1444534251. expected length is 86 and actual length is 86
> 2020-04-27 07:27:30.850 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected checksum is 1992192696 and actual is checksum 3455180678. expected length is 119844 and actual length is 117860
> 2020-04-27 07:27:30.853 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is 2776236859 and actual is checksum 2572132550. expected length is 508 and actual length is 508
> 2020-04-27 07:27:35.866 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is 2573424453 and actual is checksum 3612352961. expected length is 546 and actual length is 437
> 2020-04-27 07:27:35.868 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is 2592320603 and actual is checksum 1374033039. expected length is 289 and actual length is 289
> 2020-04-27 07:27:35.872 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is 2911257988 and actual is checksum 2011202433. expected length is 437 and actual length is 437
> 2020-04-27 07:27:35.874 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is 1304802713 and actual is checksum 454030092. expected length is 7598425 and actual length is 49265169
> 2020-04-27 07:27:37.234 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is 1422458436 and actual is checksum 703149874. expected length is 437 and actual length is 437
> 2020-04-27 07:27:38.117 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 1892273984 and actual is checksum 2087644984. expected length is 50710602 and actual length is 10402264
> 2020-04-27 07:27:38.532 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is 3864394245 and actual is checksum 1846889307. expected length is 289 and actual length is 289
> 2020-04-27 07:27:38.575 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is 660375856 and actual is checksum 571052523. expected length is 114 and actual length is 84
> 2020-04-27 07:27:38.590 WARN (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> 2020-04-27 07:27:39.259 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected checksum is 421440641 and actual is checksum 2075160524. expected length is 337361 and actual length is 141820
> 2020-04-27 07:27:44.881 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected checksum is 47189924 and actual is checksum 3640637009. expected length is 5235629 and actual length is 48679
> 2020-04-27 07:27:45.335 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected checksum is 2305624674 and actual is checksum 704932512. expected length is 693 and actual length is 115
> 2020-04-27 07:27:45.939 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is 1157089232 and actual is checksum 2388254324. expected length is 23435 and actual length is 673
> 2020-04-27 07:27:45.941 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is 2811885732 and actual is checksum 2854129172. expected length is 508 and actual length is 508
> 2020-04-27 07:27:45.943 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected checksum is 1925626873 and actual is checksum 1921615605. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.945 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected checksum is 3627283142 and actual is checksum 3796132842. expected length is 34596 and actual length is 958
> 2020-04-27 07:27:45.947 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is 1291024098 and actual is checksum 2700484857. expected length is 107020 and actual length is 104209
> 2020-04-27 07:27:45.949 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected checksum is 2785033346 and actual is checksum 1911387906. expected length is 816 and actual length is 760
> 2020-04-27 07:27:45.951 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected checksum is 942596692 and actual is checksum 3039336477. expected length is 5557044 and actual length is 4847564
> 2020-04-27 07:27:45.974 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is 143536399 and actual is checksum 2962535892. expected length is 116 and actual length is 111
> 2020-04-27 07:27:45.976 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected checksum is 3884436961 and actual is checksum 1345841996. expected length is 339801 and actual length is 336809
> 2020-04-27 07:27:45.980 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is 3902419774 and actual is checksum 3791519088. expected length is 23306 and actual length is 21047
> 2020-04-27 07:27:45.981 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is 45470103 and actual is checksum 235515609. expected length is 508 and actual length is 508
> 2020-04-27 07:27:45.983 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected checksum is 408594907 and actual is checksum 2650340971. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.985 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected checksum is 761408352 and actual is checksum 685575313. expected length is 35256 and actual length is 31644
> 2020-04-27 07:27:45.987 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is 811204953 and actual is checksum 1244308773. expected length is 107460 and actual length is 107460
> 2020-04-27 07:27:45.989 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is 3813441918 and actual is checksum 2133371346. expected length is 107460 and actual length is 106360
> 2020-04-27 07:27:45.991 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is 4263416408 and actual is checksum 3722461284. expected length is 16698 and actual length is 21392
> 2020-04-27 07:27:45.993 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected checksum is 632422377 and actual is checksum 501799146. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.994 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected checksum is 3170165892 and actual is checksum 4152937488. expected length is 25090 and actual length is 32328
> 2020-04-27 07:27:46.000 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected checksum is 2679120655 and actual is checksum 3243853760. expected length is 553 and actual length is 726
> 2020-04-27 07:27:46.001 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected checksum is 2991034554 and actual is checksum 1980969905. expected length is 321785 and actual length is 322117
> 2020-04-27 07:27:46.004 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 2117172713 and actual is checksum 4064616498. expected length is 3479018 and actual length is 4488695
> 2020-04-27 07:27:46.017 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 1613761705 and actual is checksum 4154864516. expected length is 106 and actual length is 111
> 2020-04-27 07:27:46.019 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 3059265880 and actual is checksum 531516038. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.021 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 1804496408 and actual is checksum 2903373400. expected length is 627 and actual length is 621
> 2020-04-27 07:27:46.023 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 2241359534 and actual is checksum 4198491027. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.024 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 3097209578 and actual is checksum 1124925868. expected length is 312617 and actual length is 323465
> 2020-04-27 07:27:46.027 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 1909140548 and actual is checksum 2397297432. expected length is 106580 and actual length is 107460
> 2020-04-27 07:27:46.029 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 971818792 and actual is checksum 478139506. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.031 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 1304288778 and actual is checksum 1544273325. expected length is 18170 and actual length is 17756
> 2020-04-27 07:27:46.033 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 4033259458 and actual is checksum 4016593800. expected length is 27532 and actual length is 26636
> 2020-04-27 07:27:46.034 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 385809858 and actual is checksum 3497071885. expected length is 107 and actual length is 104
> 2020-04-27 07:27:46.036 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 3567656601 and actual is checksum 2800338830. expected length is 3622054 and actual length is 3792928
> 2020-04-27 07:27:46.051 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 2282222236 and actual is checksum 3183044106. expected length is 98 and actual length is 108
> 2020-04-27 07:27:46.053 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 150588833 and actual is checksum 3492244992. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.055 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 1457615503 and actual is checksum 4089018914. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.125 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 3866656431 and actual is checksum 2014845387. expected length is 16763 and actual length is 28481
> 2020-04-27 07:27:46.128 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 3008320208 and actual is checksum 808227383. expected length is 419 and actual length is 659
> 2020-04-27 07:27:46.130 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 2437844680 and actual is checksum 1756584754. expected length is 310713 and actual length is 315009
> 2020-04-27 07:27:46.133 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3036548497 and actual is checksum 343807338. expected length is 2245614 and actual length is 4011325
> 2020-04-27 07:27:46.146 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 4233343814 and actual is checksum 2984252206. expected length is 106580 and actual length is 107460
> 2020-04-27 07:27:46.148 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 2222747552 and actual is checksum 3771791266. expected length is 11198 and actual length is 18927
> 2020-04-27 07:27:46.150 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 2062419431 and actual is checksum 885929423. expected length is 107460 and actual length is 105632
> 2020-04-27 07:27:46.152 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1574973164 and actual is checksum 209201565. expected length is 516 and actual length is 138
> 2020-04-27 07:27:46.154 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 3606049617 and actual is checksum 3605002010. expected length is 14213 and actual length is 1590
> 2020-04-27 07:27:46.156 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 842753199 and actual is checksum 3527006266. expected length is 316073 and actual length is 279943
> 2020-04-27 07:27:46.159 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 2130892015 and actual is checksum 2031251178. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.161 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 1435858498 and actual is checksum 3731425121. expected length is 21284 and actual length is 2331
> 2020-04-27 07:27:46.163 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 20126402 and actual is checksum 1544152005. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.165 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1651289033 and actual is checksum 1035172250. expected length is 100 and actual length is 86
> 2020-04-27 07:27:46.166 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 3180610843 and actual is checksum 2027612627. expected length is 2818109 and actual length is 159550
> 2020-04-27 07:27:52.815 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=1471697 inSortedOrder=false
> 2020-04-27 07:27:53.592 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=211445345 inSortedOrder=false
> 2020-04-27 07:27:56.965 WARN (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348}
> 2020-04-27 07:27:57.329 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345}
> 2020-04-27 07:28:03.537 WARN (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945}
> 2020-04-27 07:28:07.353 WARN (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052}
> 2020-04-27 07:28:19.721 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697}
> 2020-04-27 07:33:26.278 WARN (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.c.RecoveryStrategy Stopping recovery for core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39]
> 2020-04-27 07:33:30.370 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> 2020-04-27 07:33:37.918 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> 2020-04-27 07:33:37.945 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is 204474971 and actual is checksum 3417943085. expected length is 437 and actual length is 546
> 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> 2020-04-28 07:53:30.910 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10528ms for sessionid 0x171a6fb51310008
> 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
Re: SolrCloud degraded during backup and batch CSV update
Posted by matthew sporleder <ms...@gmail.com>.
FWIW I've had some luck with strategy 3 (increase zk timeout) when you
overwhelm the connection to zk or the disk on zk.
Is zk on the same boxes as solr?
On Tue, Apr 28, 2020 at 10:15 PM Sethuraman, Ganesh
<Se...@dnb.com.invalid> wrote:
>
> Hi
>
> We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: The issue is SolrCloud server went into a degraded collections (for few collections) when the Solr backup and the Solr batch CSV update load happened at the same time as backup. The CSV data load was about ~5 GB per shard/replica. We think this happened after zkClient disconnect happened as noted below. We had to restart Solr to bring it back to normal.
>
>
> 1. Is it not suggested to run backup and Solr batch CSV update large load at the same time?
> 2. In the past we have seen two CSV batch update load in parallel causes issues, is this also not suggested (this issue is not related to that)?
> 3. Do you think we should increase Zookeeper timeout?
> 4. How do we know if we need to up the JVM Max memory, and by how much?
> 5. We also see that once the Solr goes into degraded collection and recovery failed, it NEVER get back to normal, even after when there is no load. Is this a bug?
>
> The GC information and Solr Log below
>
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
>
>
> 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> ....
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
>
>
>
>
> SOLR Log Below (Curtailed WARN log)
> --------------------
> 2020-04-27 07:26:45.402 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.405 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.406 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.408 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.410 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error fetching file, doing one retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.414 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes)
> java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.414 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> java.nio.file.NoSuchFileException: /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> at org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is 3934327071 and actual is checksum 3095753427. expected length is 33235371 and actual length is 54180642
> 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is 138861107 and actual is checksum 188997082. expected length is 107460 and actual length is 107460
> 2020-04-27 07:26:45.457 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected checksum is 2947323803 and actual is checksum 979431341. expected length is 32176 and actual length is 26089
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected checksum is 2087731087 and actual is checksum 3540174119. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 777834759 and actual is checksum 289576304. expected length is 59980991 and actual length is 6236424
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is 1321210176 and actual is checksum 224636564. expected length is 114 and actual length is 106
> 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is 3790133021 and actual is checksum 2516047051. expected length is 1477755 and actual length is 1477850
> 2020-04-27 07:26:45.578 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected checksum is 3690299979 and actual is checksum 411724865. expected length is 4392270 and actual length is 4396498
> 2020-04-27 07:26:45.588 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected checksum is 937224718 and actual is checksum 1176441880. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.733 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected checksum is 1429458386 and actual is checksum 4155623862. expected length is 2213909 and actual length is 2213939
> 2020-04-27 07:26:45.754 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
> at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> 2020-04-27 07:26:45.780 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected checksum is 1474431645 and actual is checksum 1271168044. expected length is 275570 and actual length is 274506
> 2020-04-27 07:26:45.834 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is 4129522944 and actual is checksum 4080519415. expected length is 6626 and actual length is 6626
> 2020-04-27 07:26:45.843 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is 2493366746 and actual is checksum 31587394. expected length is 546 and actual length is 546
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 1392813257 and actual is checksum 3455988143. expected length is 103 and actual length is 111
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected checksum is 2301226889 and actual is checksum 29696594. expected length is 86459 and actual length is 85786
> 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected checksum is 112676849 and actual is checksum 354222195. expected length is 286474 and actual length is 286546
> 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 2203892485 and actual is checksum 1165086749. expected length is 107460 and actual length is 106580
> 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is 3175370827 and actual is checksum 520836483. expected length is 51396 and actual length is 51404
> 2020-04-27 07:26:45.912 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is 967260408 and actual is checksum 969726862. expected length is 3180 and actual length is 3180
> 2020-04-27 07:26:45.914 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 719562508 and actual is checksum 956062095. expected length is 16717 and actual length is 21454
> 2020-04-27 07:26:45.915 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected checksum is 3833337361 and actual is checksum 493358852. expected length is 43996 and actual length is 44026
> 2020-04-27 07:26:45.916 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 1533628328 and actual is checksum 3735455395. expected length is 110 and actual length is 110
> 2020-04-27 07:26:45.917 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected checksum is 75152145 and actual is checksum 1665090946. expected length is 497241188 and actual length is 497254814
> 2020-04-27 07:26:45.918 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1349491281 and actual is checksum 3808036883. expected length is 25153 and actual length is 32322
> 2020-04-27 07:26:45.920 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 3820733560 and actual is checksum 2819887160. expected length is 571 and actual length is 725
> 2020-04-27 07:26:46.034 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 1547030244 and actual is checksum 1122196351. expected length is 324313 and actual length is 323553
> 2020-04-27 07:26:46.170 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 751850967 and actual is checksum 1026527313. expected length is 508 and actual length is 508
> and actual length is 319133
> 2020-04-27 07:27:30.567 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 3435029878 and actual is checksum 1202327807. expected length is 3337719 and actual length is 3375106
> 2020-04-27 07:27:30.578 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 2264443519 and actual is checksum 2299249501. expected length is 104 and actual length is 102
> 2020-04-27 07:27:30.580 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 1184027188 and actual is checksum 2810465617. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.622 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 2246788287 and actual is checksum 2682862227. expected length is 92 and actual length is 96
> 2020-04-27 07:27:30.625 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 3227520043 and actual is checksum 2778425164. expected length is 107137 and actual length is 107137
> 2020-04-27 07:27:30.627 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 1964200539 and actual is checksum 3997289541. expected length is 7398 and actual length is 9042
> 2020-04-27 07:27:30.630 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 2165407577 and actual is checksum 2563543943. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.632 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1427836076 and actual is checksum 225462388. expected length is 11162 and actual length is 13623
> 2020-04-27 07:27:30.634 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 2998276982 and actual is checksum 1966428087. expected length is 296 and actual length is 353
> 2020-04-27 07:27:30.637 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 2717609594 and actual is checksum 673086945. expected length is 315491 and actual length is 312531
> 2020-04-27 07:27:30.640 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 2885031416 and actual is checksum 762513018. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.643 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected checksum is 602612626 and actual is checksum 2826680041. expected length is 1292092 and actual length is 1655171
> 2020-04-27 07:27:30.669 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 401385741 and actual is checksum 3373755803. expected length is 621 and actual length is 715
> 2020-04-27 07:27:30.671 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 3708405765 and actual is checksum 1057460545. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.674 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 2755907323 and actual is checksum 1931314059. expected length is 327125 and actual length is 318737
> 2020-04-27 07:27:30.677 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 77505520 and actual is checksum 2602905677. expected length is 107240 and actual length is 107460
> 2020-04-27 07:27:30.680 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 4204960097 and actual is checksum 2276943505. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.682 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 3414743450 and actual is checksum 3890346633. expected length is 18415 and actual length is 21922
> 2020-04-27 07:27:30.685 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 2879829129 and actual is checksum 790212338. expected length is 27860 and actual length is 33235
> 2020-04-27 07:27:30.687 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 3356980932 and actual is checksum 1544608917. expected length is 108 and actual length is 112
> 2020-04-27 07:27:30.690 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 1979425912 and actual is checksum 584894070. expected length is 3726110 and actual length is 4689653
> 2020-04-27 07:27:30.700 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is 3754169108 and actual is checksum 527001228. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.702 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected checksum is 3981710860 and actual is checksum 1749011618. expected length is 1938123 and actual length is 33694
> 2020-04-27 07:27:30.710 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is 1515379146 and actual is checksum 3212722427. expected length is 98 and actual length is 84
> 2020-04-27 07:27:30.712 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is 1819400085 and actual is checksum 3372088644. expected length is 107020 and actual length is 103887
> 2020-04-27 07:27:30.715 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected checksum is 1401562358 and actual is checksum 2224139078. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.718 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected checksum is 3386701421 and actual is checksum 1434467863. expected length is 15944 and actual length is 685
> 2020-04-27 07:27:30.720 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is 4259543754 and actual is checksum 3203989793. expected length is 10616 and actual length is 439
> 2020-04-27 07:27:30.722 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected checksum is 725882371 and actual is checksum 3650374401. expected length is 383 and actual length is 114
> 2020-04-27 07:27:30.725 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected checksum is 3717717087 and actual is checksum 3712473868. expected length is 303729 and actual length is 117582
> 2020-04-27 07:27:30.728 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 3506718582 and actual is checksum 342046803. expected length is 93 and actual length is 104
> 2020-04-27 07:27:30.731 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 2920971663 and actual is checksum 2142622028. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.733 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 723455769 and actual is checksum 2731538349. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.735 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 2624342764 and actual is checksum 1283943229. expected length is 11424 and actual length is 26688
> 2020-04-27 07:27:30.738 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 978416968 and actual is checksum 1807767534. expected length is 330 and actual length is 597
> 2020-04-27 07:27:30.740 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 3599774596 and actual is checksum 2764190632. expected length is 318017 and actual length is 311617
> 2020-04-27 07:27:30.743 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3141314417 and actual is checksum 3847788889. expected length is 1258075 and actual length is 3539994
> 2020-04-27 07:27:30.749 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 1404317797 and actual is checksum 2471843701. expected length is 105295 and actual length is 107020
> 2020-04-27 07:27:30.752 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 3412288246 and actual is checksum 922481525. expected length is 7580 and actual length is 17733
> 2020-04-27 07:27:30.754 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected checksum is 2675540020 and actual is checksum 1452209674. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.757 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected checksum is 665797990 and actual is checksum 2134373365. expected length is 10715 and actual length is 3418
> 2020-04-27 07:27:30.759 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is 2006427884 and actual is checksum 527297847. expected length is 7178 and actual length is 2326
> 2020-04-27 07:27:30.761 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is 200552893 and actual is checksum 3290522141. expected length is 106580 and actual length is 105206
> 2020-04-27 07:27:30.764 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected checksum is 1511033560 and actual is checksum 3689083691. expected length is 278 and actual length is 154
> 2020-04-27 07:27:30.767 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected checksum is 1435050119 and actual is checksum 6864243. expected length is 323217 and actual length is 338063
> 2020-04-27 07:27:30.770 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is 1107038327 and actual is checksum 3068870234. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.772 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is 2853806777 and actual is checksum 459866651. expected length is 93 and actual length is 87
> 2020-04-27 07:27:30.775 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected checksum is 1356537478 and actual is checksum 2594306289. expected length is 1179215 and actual length is 249458
> 2020-04-27 07:27:30.780 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 1785795479 and actual is checksum 65014613. expected length is 106580 and actual length is 106800
> 2020-04-27 07:27:30.783 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1824666616 and actual is checksum 2857527783. expected length is 507 and actual length is 637
> 2020-04-27 07:27:30.785 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 1384446542 and actual is checksum 3981590407. expected length is 13686 and actual length is 18516
> 2020-04-27 07:27:30.788 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 3380549662 and actual is checksum 2002436369. expected length is 317777 and actual length is 319213
> 2020-04-27 07:27:30.791 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 1594954666 and actual is checksum 1892723425. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.794 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 305271241 and actual is checksum 854278263. expected length is 20522 and actual length is 27836
> 2020-04-27 07:27:30.796 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 2220527769 and actual is checksum 2016379681. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.799 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1648054590 and actual is checksum 137025071. expected length is 100 and actual length is 105
> 2020-04-27 07:27:30.801 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 262113808 and actual is checksum 327562225. expected length is 2731808 and actual length is 3909012
> 2020-04-27 07:27:30.809 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected checksum is 2157667750 and actual is checksum 3043505208. expected length is 36906 and actual length is 163443
> 2020-04-27 07:27:30.812 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is 3456088243 and actual is checksum 1715362430. expected length is 453 and actual length is 1636
> 2020-04-27 07:27:30.814 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is 4028738113 and actual is checksum 4245480574. expected length is 104876 and actual length is 105515
> 2020-04-27 07:27:30.817 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected checksum is 3120632072 and actual is checksum 1945528118. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.819 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected checksum is 3303166970 and actual is checksum 1740028553. expected length is 114 and actual length is 137
> 2020-04-27 07:27:30.822 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected checksum is 126700369 and actual is checksum 162509958. expected length is 700 and actual length is 2398
> 2020-04-27 07:27:30.825 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is 3157051968 and actual is checksum 137679212. expected length is 84 and actual length is 86
> 2020-04-27 07:27:30.827 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected checksum is 3969090178 and actual is checksum 2364978843. expected length is 127576 and actual length is 266581
> 2020-04-27 07:27:30.830 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is 3730892058 and actual is checksum 2734980133. expected length is 508 and actual length is 508
> 2020-04-27 07:27:30.832 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is 3259266514 and actual is checksum 2051283039. expected length is 104855 and actual length is 106819
> 2020-04-27 07:27:30.835 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected checksum is 380153843 and actual is checksum 580580220. expected length is 1822 and actual length is 1737
> 2020-04-27 07:27:30.838 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected checksum is 4094454161 and actual is checksum 421930789. expected length is 110 and actual length is 110
> 2020-04-27 07:27:30.840 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is 911575783 and actual is checksum 2090965125. expected length is 1244 and actual length is 1201
> 2020-04-27 07:27:30.842 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected checksum is 1279063865 and actual is checksum 765869778. expected length is 128 and actual length is 130
> 2020-04-27 07:27:30.845 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected checksum is 1689040450 and actual is checksum 627554920. expected length is 223473 and actual length is 230157
> 2020-04-27 07:27:30.848 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is 1278110969 and actual is checksum 1444534251. expected length is 86 and actual length is 86
> 2020-04-27 07:27:30.850 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected checksum is 1992192696 and actual is checksum 3455180678. expected length is 119844 and actual length is 117860
> 2020-04-27 07:27:30.853 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is 2776236859 and actual is checksum 2572132550. expected length is 508 and actual length is 508
> 2020-04-27 07:27:35.866 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is 2573424453 and actual is checksum 3612352961. expected length is 546 and actual length is 437
> 2020-04-27 07:27:35.868 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is 2592320603 and actual is checksum 1374033039. expected length is 289 and actual length is 289
> 2020-04-27 07:27:35.872 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is 2911257988 and actual is checksum 2011202433. expected length is 437 and actual length is 437
> 2020-04-27 07:27:35.874 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is 1304802713 and actual is checksum 454030092. expected length is 7598425 and actual length is 49265169
> 2020-04-27 07:27:37.234 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is 1422458436 and actual is checksum 703149874. expected length is 437 and actual length is 437
> 2020-04-27 07:27:38.117 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 1892273984 and actual is checksum 2087644984. expected length is 50710602 and actual length is 10402264
> 2020-04-27 07:27:38.532 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is 3864394245 and actual is checksum 1846889307. expected length is 289 and actual length is 289
> 2020-04-27 07:27:38.575 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is 660375856 and actual is checksum 571052523. expected length is 114 and actual length is 84
> 2020-04-27 07:27:38.590 WARN (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> 2020-04-27 07:27:39.259 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected checksum is 421440641 and actual is checksum 2075160524. expected length is 337361 and actual length is 141820
> 2020-04-27 07:27:44.881 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected checksum is 47189924 and actual is checksum 3640637009. expected length is 5235629 and actual length is 48679
> 2020-04-27 07:27:45.335 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected checksum is 2305624674 and actual is checksum 704932512. expected length is 693 and actual length is 115
> 2020-04-27 07:27:45.939 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is 1157089232 and actual is checksum 2388254324. expected length is 23435 and actual length is 673
> 2020-04-27 07:27:45.941 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is 2811885732 and actual is checksum 2854129172. expected length is 508 and actual length is 508
> 2020-04-27 07:27:45.943 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected checksum is 1925626873 and actual is checksum 1921615605. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.945 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected checksum is 3627283142 and actual is checksum 3796132842. expected length is 34596 and actual length is 958
> 2020-04-27 07:27:45.947 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is 1291024098 and actual is checksum 2700484857. expected length is 107020 and actual length is 104209
> 2020-04-27 07:27:45.949 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected checksum is 2785033346 and actual is checksum 1911387906. expected length is 816 and actual length is 760
> 2020-04-27 07:27:45.951 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected checksum is 942596692 and actual is checksum 3039336477. expected length is 5557044 and actual length is 4847564
> 2020-04-27 07:27:45.974 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is 143536399 and actual is checksum 2962535892. expected length is 116 and actual length is 111
> 2020-04-27 07:27:45.976 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected checksum is 3884436961 and actual is checksum 1345841996. expected length is 339801 and actual length is 336809
> 2020-04-27 07:27:45.980 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is 3902419774 and actual is checksum 3791519088. expected length is 23306 and actual length is 21047
> 2020-04-27 07:27:45.981 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is 45470103 and actual is checksum 235515609. expected length is 508 and actual length is 508
> 2020-04-27 07:27:45.983 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected checksum is 408594907 and actual is checksum 2650340971. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.985 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected checksum is 761408352 and actual is checksum 685575313. expected length is 35256 and actual length is 31644
> 2020-04-27 07:27:45.987 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is 811204953 and actual is checksum 1244308773. expected length is 107460 and actual length is 107460
> 2020-04-27 07:27:45.989 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is 3813441918 and actual is checksum 2133371346. expected length is 107460 and actual length is 106360
> 2020-04-27 07:27:45.991 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is 4263416408 and actual is checksum 3722461284. expected length is 16698 and actual length is 21392
> 2020-04-27 07:27:45.993 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected checksum is 632422377 and actual is checksum 501799146. expected length is 110 and actual length is 110
> 2020-04-27 07:27:45.994 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected checksum is 3170165892 and actual is checksum 4152937488. expected length is 25090 and actual length is 32328
> 2020-04-27 07:27:46.000 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected checksum is 2679120655 and actual is checksum 3243853760. expected length is 553 and actual length is 726
> 2020-04-27 07:27:46.001 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected checksum is 2991034554 and actual is checksum 1980969905. expected length is 321785 and actual length is 322117
> 2020-04-27 07:27:46.004 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 2117172713 and actual is checksum 4064616498. expected length is 3479018 and actual length is 4488695
> 2020-04-27 07:27:46.017 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 1613761705 and actual is checksum 4154864516. expected length is 106 and actual length is 111
> 2020-04-27 07:27:46.019 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 3059265880 and actual is checksum 531516038. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.021 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 1804496408 and actual is checksum 2903373400. expected length is 627 and actual length is 621
> 2020-04-27 07:27:46.023 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 2241359534 and actual is checksum 4198491027. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.024 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 3097209578 and actual is checksum 1124925868. expected length is 312617 and actual length is 323465
> 2020-04-27 07:27:46.027 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 1909140548 and actual is checksum 2397297432. expected length is 106580 and actual length is 107460
> 2020-04-27 07:27:46.029 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 971818792 and actual is checksum 478139506. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.031 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 1304288778 and actual is checksum 1544273325. expected length is 18170 and actual length is 17756
> 2020-04-27 07:27:46.033 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 4033259458 and actual is checksum 4016593800. expected length is 27532 and actual length is 26636
> 2020-04-27 07:27:46.034 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 385809858 and actual is checksum 3497071885. expected length is 107 and actual length is 104
> 2020-04-27 07:27:46.036 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 3567656601 and actual is checksum 2800338830. expected length is 3622054 and actual length is 3792928
> 2020-04-27 07:27:46.051 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 2282222236 and actual is checksum 3183044106. expected length is 98 and actual length is 108
> 2020-04-27 07:27:46.053 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 150588833 and actual is checksum 3492244992. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.055 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 1457615503 and actual is checksum 4089018914. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.125 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 3866656431 and actual is checksum 2014845387. expected length is 16763 and actual length is 28481
> 2020-04-27 07:27:46.128 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 3008320208 and actual is checksum 808227383. expected length is 419 and actual length is 659
> 2020-04-27 07:27:46.130 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 2437844680 and actual is checksum 1756584754. expected length is 310713 and actual length is 315009
> 2020-04-27 07:27:46.133 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3036548497 and actual is checksum 343807338. expected length is 2245614 and actual length is 4011325
> 2020-04-27 07:27:46.146 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 4233343814 and actual is checksum 2984252206. expected length is 106580 and actual length is 107460
> 2020-04-27 07:27:46.148 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 2222747552 and actual is checksum 3771791266. expected length is 11198 and actual length is 18927
> 2020-04-27 07:27:46.150 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 2062419431 and actual is checksum 885929423. expected length is 107460 and actual length is 105632
> 2020-04-27 07:27:46.152 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1574973164 and actual is checksum 209201565. expected length is 516 and actual length is 138
> 2020-04-27 07:27:46.154 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 3606049617 and actual is checksum 3605002010. expected length is 14213 and actual length is 1590
> 2020-04-27 07:27:46.156 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 842753199 and actual is checksum 3527006266. expected length is 316073 and actual length is 279943
> 2020-04-27 07:27:46.159 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 2130892015 and actual is checksum 2031251178. expected length is 110 and actual length is 110
> 2020-04-27 07:27:46.161 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 1435858498 and actual is checksum 3731425121. expected length is 21284 and actual length is 2331
> 2020-04-27 07:27:46.163 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 20126402 and actual is checksum 1544152005. expected length is 508 and actual length is 508
> 2020-04-27 07:27:46.165 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1651289033 and actual is checksum 1035172250. expected length is 100 and actual length is 86
> 2020-04-27 07:27:46.166 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 3180610843 and actual is checksum 2027612627. expected length is 2818109 and actual length is 159550
> 2020-04-27 07:27:52.815 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=1471697 inSortedOrder=false
> 2020-04-27 07:27:53.592 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=211445345 inSortedOrder=false
> 2020-04-27 07:27:56.965 WARN (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348}
> 2020-04-27 07:27:57.329 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345}
> 2020-04-27 07:28:03.537 WARN (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945}
> 2020-04-27 07:28:07.353 WARN (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052}
> 2020-04-27 07:28:19.721 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697}
> 2020-04-27 07:33:26.278 WARN (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.c.RecoveryStrategy Stopping recovery for core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39]
> 2020-04-27 07:33:30.370 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> 2020-04-27 07:33:37.918 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508
> 2020-04-27 07:33:37.945 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is 204474971 and actual is checksum 3417943085. expected length is 437 and actual length is 546
> 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> 2020-04-28 07:53:30.910 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10528ms for sessionid 0x171a6fb51310008
> 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None
> 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
Re: SolrCloud degraded during backup and batch CSV update
Posted by matthew sporleder <ms...@gmail.com>.
If the errors happen with garbage collection then potentially, yes.
You should never pause longer than your zk timeout (both sides).
On Thu, Apr 30, 2020 at 11:03 PM Ganesh Sethuraman
<ga...@gmail.com> wrote:
>
> Any other JVM settings change possible?
>
> On Tue, Apr 28, 2020, 10:15 PM Sethuraman, Ganesh
> <Se...@dnb.com.invalid> wrote:
>
> > Hi
> >
> > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92
> > collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes,
> > with JVM size of 18GB (G1 GC). We need your help with the Issue we faced
> > today: The issue is SolrCloud server went into a degraded collections (for
> > few collections) when the Solr backup and the Solr batch CSV update load
> > happened at the same time as backup. The CSV data load was about ~5 GB per
> > shard/replica. We think this happened after zkClient disconnect happened as
> > noted below. We had to restart Solr to bring it back to normal.
> >
> >
> > 1. Is it not suggested to run backup and Solr batch CSV update large
> > load at the same time?
> > 2. In the past we have seen two CSV batch update load in parallel
> > causes issues, is this also not suggested (this issue is not related to
> > that)?
> > 3. Do you think we should increase Zookeeper timeout?
> > 4. How do we know if we need to up the JVM Max memory, and by how much?
> > 5. We also see that once the Solr goes into degraded collection and
> > recovery failed, it NEVER get back to normal, even after when there is no
> > load. Is this a bug?
> >
> > The GC information and Solr Log below
> >
> >
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
> >
> >
> > 2020-04-27 07:34:07.322 WARN
> > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181))
> > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server
> > in 10775ms for sessionid 0x171a6fb51310008
> > ....
> > 2020-04-27 07:34:07.426 WARN
> > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread)
> > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> >
> >
> >
> >
> > SOLR Log Below (Curtailed WARN log)
> > --------------------
> > 2020-04-27 07:26:45.402 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.405 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.406 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.408 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.410 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.412 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error fetching file, doing one
> > retry...:org.apache.solr.common.SolrException: Unable to download
> > _1kr_r.liv completely. Downloaded 0!=587
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.414 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> > 587 bytes)
> > java.io.EOFException
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> > at
> > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.414 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> > java.nio.file.NoSuchFileException:
> > /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> > at
> > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> > at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> > at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> > at
> > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> > at
> > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> > at java.nio.file.Files.delete(Files.java:1126)
> > at
> > org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> > at
> > org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> > at
> > org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> > at
> > org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2020-04-27 07:26:45.449 WARN
> > (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> > o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is
> > 3934327071 and actual is checksum 3095753427. expected length is 33235371
> > and actual length is 54180642
> > 2020-04-27 07:26:45.449 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is
> > 138861107 and actual is checksum 188997082. expected length is 107460 and
> > actual length is 107460
> > 2020-04-27 07:26:45.457 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected
> > checksum is 2947323803 and actual is checksum 979431341. expected length is
> > 32176 and actual length is 26089
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected
> > checksum is 2087731087 and actual is checksum 3540174119. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3)
> > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1]
> > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> > 777834759 and actual is checksum 289576304. expected length is 59980991 and
> > actual length is 6236424
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is
> > 1321210176 and actual is checksum 224636564. expected length is 114 and
> > actual length is 106
> > 2020-04-27 07:26:45.521 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is
> > 3790133021 and actual is checksum 2516047051. expected length is 1477755
> > and actual length is 1477850
> > 2020-04-27 07:26:45.578 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected
> > checksum is 3690299979 and actual is checksum 411724865. expected length is
> > 4392270 and actual length is 4396498
> > 2020-04-27 07:26:45.588 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected
> > checksum is 937224718 and actual is checksum 1176441880. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:26:45.733 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected
> > checksum is 1429458386 and actual is checksum 4155623862. expected length
> > is 2213909 and actual length is 2213939
> > 2020-04-27 07:26:45.754 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.ReplicationHandler Index fetch failed
> > :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv
> > completely. Downloaded 0!=587
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> > at
> > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> > at
> > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> > at
> > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.c.RecoveryStrategy Error while trying to
> > recover:org.apache.solr.common.SolrException: Replication for recovery
> > failed.
> > at
> > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2020-04-27 07:26:45.755 ERROR
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> > 2020-04-27 07:26:45.780 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected
> > checksum is 1474431645 and actual is checksum 1271168044. expected length
> > is 275570 and actual length is 274506
> > 2020-04-27 07:26:45.834 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is
> > 4129522944 and actual is checksum 4080519415. expected length is 6626 and
> > actual length is 6626
> > 2020-04-27 07:26:45.843 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is
> > 2493366746 and actual is checksum 31587394. expected length is 546 and
> > actual length is 546
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> > 1392813257 and actual is checksum 3455988143. expected length is 103 and
> > actual length is 111
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected
> > checksum is 2301226889 and actual is checksum 29696594. expected length is
> > 86459 and actual length is 85786
> > 2020-04-27 07:26:45.882 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected
> > checksum is 112676849 and actual is checksum 354222195. expected length is
> > 286474 and actual length is 286546
> > 2020-04-27 07:26:45.911 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> > 2203892485 and actual is checksum 1165086749. expected length is 107460 and
> > actual length is 106580
> > 2020-04-27 07:26:45.911 WARN
> > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> > o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is
> > 3175370827 and actual is checksum 520836483. expected length is 51396 and
> > actual length is 51404
> > 2020-04-27 07:26:45.912 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is
> > 967260408 and actual is checksum 969726862. expected length is 3180 and
> > actual length is 3180
> > 2020-04-27 07:26:45.914 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> > 719562508 and actual is checksum 956062095. expected length is 16717 and
> > actual length is 21454
> > 2020-04-27 07:26:45.915 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected
> > checksum is 3833337361 and actual is checksum 493358852. expected length is
> > 43996 and actual length is 44026
> > 2020-04-27 07:26:45.916 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> > checksum is 1533628328 and actual is checksum 3735455395. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:26:45.917 WARN
> > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected
> > checksum is 75152145 and actual is checksum 1665090946. expected length is
> > 497241188 and actual length is 497254814
> > 2020-04-27 07:26:45.918 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> > checksum is 1349491281 and actual is checksum 3808036883. expected length
> > is 25153 and actual length is 32322
> > 2020-04-27 07:26:45.920 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> > checksum is 3820733560 and actual is checksum 2819887160. expected length
> > is 571 and actual length is 725
> > 2020-04-27 07:26:46.034 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> > checksum is 1547030244 and actual is checksum 1122196351. expected length
> > is 324313 and actual length is 323553
> > 2020-04-27 07:26:46.170 WARN
> > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> > 751850967 and actual is checksum 1026527313. expected length is 508 and
> > actual length is 508
> > and actual length is 319133
> > 2020-04-27 07:27:30.567 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected
> > checksum is 3435029878 and actual is checksum 1202327807. expected length
> > is 3337719 and actual length is 3375106
> > 2020-04-27 07:27:30.578 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is
> > 2264443519 and actual is checksum 2299249501. expected length is 104 and
> > actual length is 102
> > 2020-04-27 07:27:30.580 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is
> > 1184027188 and actual is checksum 2810465617. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.622 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> > 2246788287 and actual is checksum 2682862227. expected length is 92 and
> > actual length is 96
> > 2020-04-27 07:27:30.625 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> > 3227520043 and actual is checksum 2778425164. expected length is 107137 and
> > actual length is 107137
> > 2020-04-27 07:27:30.627 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> > 1964200539 and actual is checksum 3997289541. expected length is 7398 and
> > actual length is 9042
> > 2020-04-27 07:27:30.630 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> > checksum is 2165407577 and actual is checksum 2563543943. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.632 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> > checksum is 1427836076 and actual is checksum 225462388. expected length is
> > 11162 and actual length is 13623
> > 2020-04-27 07:27:30.634 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> > checksum is 2998276982 and actual is checksum 1966428087. expected length
> > is 296 and actual length is 353
> > 2020-04-27 07:27:30.637 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> > checksum is 2717609594 and actual is checksum 673086945. expected length is
> > 315491 and actual length is 312531
> > 2020-04-27 07:27:30.640 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> > 2885031416 and actual is checksum 762513018. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.643 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected
> > checksum is 602612626 and actual is checksum 2826680041. expected length is
> > 1292092 and actual length is 1655171
> > 2020-04-27 07:27:30.669 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected
> > checksum is 401385741 and actual is checksum 3373755803. expected length is
> > 621 and actual length is 715
> > 2020-04-27 07:27:30.671 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is
> > 3708405765 and actual is checksum 1057460545. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.674 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected
> > checksum is 2755907323 and actual is checksum 1931314059. expected length
> > is 327125 and actual length is 318737
> > 2020-04-27 07:27:30.677 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is
> > 77505520 and actual is checksum 2602905677. expected length is 107240 and
> > actual length is 107460
> > 2020-04-27 07:27:30.680 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected
> > checksum is 4204960097 and actual is checksum 2276943505. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.682 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is
> > 3414743450 and actual is checksum 3890346633. expected length is 18415 and
> > actual length is 21922
> > 2020-04-27 07:27:30.685 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected
> > checksum is 2879829129 and actual is checksum 790212338. expected length is
> > 27860 and actual length is 33235
> > 2020-04-27 07:27:30.687 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is
> > 3356980932 and actual is checksum 1544608917. expected length is 108 and
> > actual length is 112
> > 2020-04-27 07:27:30.690 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected
> > checksum is 1979425912 and actual is checksum 584894070. expected length is
> > 3726110 and actual length is 4689653
> > 2020-04-27 07:27:30.700 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is
> > 3754169108 and actual is checksum 527001228. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.702 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected
> > checksum is 3981710860 and actual is checksum 1749011618. expected length
> > is 1938123 and actual length is 33694
> > 2020-04-27 07:27:30.710 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is
> > 1515379146 and actual is checksum 3212722427. expected length is 98 and
> > actual length is 84
> > 2020-04-27 07:27:30.712 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is
> > 1819400085 and actual is checksum 3372088644. expected length is 107020 and
> > actual length is 103887
> > 2020-04-27 07:27:30.715 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected
> > checksum is 1401562358 and actual is checksum 2224139078. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.718 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected
> > checksum is 3386701421 and actual is checksum 1434467863. expected length
> > is 15944 and actual length is 685
> > 2020-04-27 07:27:30.720 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is
> > 4259543754 and actual is checksum 3203989793. expected length is 10616 and
> > actual length is 439
> > 2020-04-27 07:27:30.722 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected
> > checksum is 725882371 and actual is checksum 3650374401. expected length is
> > 383 and actual length is 114
> > 2020-04-27 07:27:30.725 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected
> > checksum is 3717717087 and actual is checksum 3712473868. expected length
> > is 303729 and actual length is 117582
> > 2020-04-27 07:27:30.728 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is
> > 3506718582 and actual is checksum 342046803. expected length is 93 and
> > actual length is 104
> > 2020-04-27 07:27:30.731 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is
> > 2920971663 and actual is checksum 2142622028. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.733 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected
> > checksum is 723455769 and actual is checksum 2731538349. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:30.735 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected
> > checksum is 2624342764 and actual is checksum 1283943229. expected length
> > is 11424 and actual length is 26688
> > 2020-04-27 07:27:30.738 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected
> > checksum is 978416968 and actual is checksum 1807767534. expected length is
> > 330 and actual length is 597
> > 2020-04-27 07:27:30.740 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected
> > checksum is 3599774596 and actual is checksum 2764190632. expected length
> > is 318017 and actual length is 311617
> > 2020-04-27 07:27:30.743 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected
> > checksum is 3141314417 and actual is checksum 3847788889. expected length
> > is 1258075 and actual length is 3539994
> > 2020-04-27 07:27:30.749 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is
> > 1404317797 and actual is checksum 2471843701. expected length is 105295 and
> > actual length is 107020
> > 2020-04-27 07:27:30.752 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is
> > 3412288246 and actual is checksum 922481525. expected length is 7580 and
> > actual length is 17733
> > 2020-04-27 07:27:30.754 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected
> > checksum is 2675540020 and actual is checksum 1452209674. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.757 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected
> > checksum is 665797990 and actual is checksum 2134373365. expected length is
> > 10715 and actual length is 3418
> > 2020-04-27 07:27:30.759 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is
> > 2006427884 and actual is checksum 527297847. expected length is 7178 and
> > actual length is 2326
> > 2020-04-27 07:27:30.761 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is
> > 200552893 and actual is checksum 3290522141. expected length is 106580 and
> > actual length is 105206
> > 2020-04-27 07:27:30.764 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected
> > checksum is 1511033560 and actual is checksum 3689083691. expected length
> > is 278 and actual length is 154
> > 2020-04-27 07:27:30.767 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected
> > checksum is 1435050119 and actual is checksum 6864243. expected length is
> > 323217 and actual length is 338063
> > 2020-04-27 07:27:30.770 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is
> > 1107038327 and actual is checksum 3068870234. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.772 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is
> > 2853806777 and actual is checksum 459866651. expected length is 93 and
> > actual length is 87
> > 2020-04-27 07:27:30.775 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected
> > checksum is 1356537478 and actual is checksum 2594306289. expected length
> > is 1179215 and actual length is 249458
> > 2020-04-27 07:27:30.780 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is
> > 1785795479 and actual is checksum 65014613. expected length is 106580 and
> > actual length is 106800
> > 2020-04-27 07:27:30.783 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected
> > checksum is 1824666616 and actual is checksum 2857527783. expected length
> > is 507 and actual length is 637
> > 2020-04-27 07:27:30.785 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is
> > 1384446542 and actual is checksum 3981590407. expected length is 13686 and
> > actual length is 18516
> > 2020-04-27 07:27:30.788 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected
> > checksum is 3380549662 and actual is checksum 2002436369. expected length
> > is 317777 and actual length is 319213
> > 2020-04-27 07:27:30.791 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected
> > checksum is 1594954666 and actual is checksum 1892723425. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.794 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected
> > checksum is 305271241 and actual is checksum 854278263. expected length is
> > 20522 and actual length is 27836
> > 2020-04-27 07:27:30.796 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is
> > 2220527769 and actual is checksum 2016379681. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.799 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is
> > 1648054590 and actual is checksum 137025071. expected length is 100 and
> > actual length is 105
> > 2020-04-27 07:27:30.801 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected
> > checksum is 262113808 and actual is checksum 327562225. expected length is
> > 2731808 and actual length is 3909012
> > 2020-04-27 07:27:30.809 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected
> > checksum is 2157667750 and actual is checksum 3043505208. expected length
> > is 36906 and actual length is 163443
> > 2020-04-27 07:27:30.812 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is
> > 3456088243 and actual is checksum 1715362430. expected length is 453 and
> > actual length is 1636
> > 2020-04-27 07:27:30.814 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is
> > 4028738113 and actual is checksum 4245480574. expected length is 104876 and
> > actual length is 105515
> > 2020-04-27 07:27:30.817 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected
> > checksum is 3120632072 and actual is checksum 1945528118. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:30.819 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected
> > checksum is 3303166970 and actual is checksum 1740028553. expected length
> > is 114 and actual length is 137
> > 2020-04-27 07:27:30.822 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected
> > checksum is 126700369 and actual is checksum 162509958. expected length is
> > 700 and actual length is 2398
> > 2020-04-27 07:27:30.825 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is
> > 3157051968 and actual is checksum 137679212. expected length is 84 and
> > actual length is 86
> > 2020-04-27 07:27:30.827 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected
> > checksum is 3969090178 and actual is checksum 2364978843. expected length
> > is 127576 and actual length is 266581
> > 2020-04-27 07:27:30.830 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is
> > 3730892058 and actual is checksum 2734980133. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:30.832 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is
> > 3259266514 and actual is checksum 2051283039. expected length is 104855 and
> > actual length is 106819
> > 2020-04-27 07:27:30.835 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected
> > checksum is 380153843 and actual is checksum 580580220. expected length is
> > 1822 and actual length is 1737
> > 2020-04-27 07:27:30.838 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected
> > checksum is 4094454161 and actual is checksum 421930789. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:30.840 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is
> > 911575783 and actual is checksum 2090965125. expected length is 1244 and
> > actual length is 1201
> > 2020-04-27 07:27:30.842 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected
> > checksum is 1279063865 and actual is checksum 765869778. expected length is
> > 128 and actual length is 130
> > 2020-04-27 07:27:30.845 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected
> > checksum is 1689040450 and actual is checksum 627554920. expected length is
> > 223473 and actual length is 230157
> > 2020-04-27 07:27:30.848 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is
> > 1278110969 and actual is checksum 1444534251. expected length is 86 and
> > actual length is 86
> > 2020-04-27 07:27:30.850 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected
> > checksum is 1992192696 and actual is checksum 3455180678. expected length
> > is 119844 and actual length is 117860
> > 2020-04-27 07:27:30.853 WARN
> > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> > o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is
> > 2776236859 and actual is checksum 2572132550. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:35.866 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is
> > 2573424453 and actual is checksum 3612352961. expected length is 546 and
> > actual length is 437
> > 2020-04-27 07:27:35.868 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is
> > 2592320603 and actual is checksum 1374033039. expected length is 289 and
> > actual length is 289
> > 2020-04-27 07:27:35.872 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is
> > 2911257988 and actual is checksum 2011202433. expected length is 437 and
> > actual length is 437
> > 2020-04-27 07:27:35.874 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is
> > 1304802713 and actual is checksum 454030092. expected length is 7598425 and
> > actual length is 49265169
> > 2020-04-27 07:27:37.234 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is
> > 1422458436 and actual is checksum 703149874. expected length is 437 and
> > actual length is 437
> > 2020-04-27 07:27:38.117 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> > 1892273984 and actual is checksum 2087644984. expected length is 50710602
> > and actual length is 10402264
> > 2020-04-27 07:27:38.532 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is
> > 3864394245 and actual is checksum 1846889307. expected length is 289 and
> > actual length is 289
> > 2020-04-27 07:27:38.575 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is
> > 660375856 and actual is checksum 571052523. expected length is 114 and
> > actual length is 84
> > 2020-04-27 07:27:38.590 WARN
> > (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744
> > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> > 2020-04-27 07:27:39.259 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected
> > checksum is 421440641 and actual is checksum 2075160524. expected length is
> > 337361 and actual length is 141820
> > 2020-04-27 07:27:44.881 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected
> > checksum is 47189924 and actual is checksum 3640637009. expected length is
> > 5235629 and actual length is 48679
> > 2020-04-27 07:27:45.335 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected
> > checksum is 2305624674 and actual is checksum 704932512. expected length is
> > 693 and actual length is 115
> > 2020-04-27 07:27:45.939 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is
> > 1157089232 and actual is checksum 2388254324. expected length is 23435 and
> > actual length is 673
> > 2020-04-27 07:27:45.941 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is
> > 2811885732 and actual is checksum 2854129172. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:45.943 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected
> > checksum is 1925626873 and actual is checksum 1921615605. expected length
> > is 110 and actual length is 110
> > 2020-04-27 07:27:45.945 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected
> > checksum is 3627283142 and actual is checksum 3796132842. expected length
> > is 34596 and actual length is 958
> > 2020-04-27 07:27:45.947 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is
> > 1291024098 and actual is checksum 2700484857. expected length is 107020 and
> > actual length is 104209
> > 2020-04-27 07:27:45.949 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected
> > checksum is 2785033346 and actual is checksum 1911387906. expected length
> > is 816 and actual length is 760
> > 2020-04-27 07:27:45.951 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected
> > checksum is 942596692 and actual is checksum 3039336477. expected length is
> > 5557044 and actual length is 4847564
> > 2020-04-27 07:27:45.974 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is
> > 143536399 and actual is checksum 2962535892. expected length is 116 and
> > actual length is 111
> > 2020-04-27 07:27:45.976 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected
> > checksum is 3884436961 and actual is checksum 1345841996. expected length
> > is 339801 and actual length is 336809
> > 2020-04-27 07:27:45.980 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is
> > 3902419774 and actual is checksum 3791519088. expected length is 23306 and
> > actual length is 21047
> > 2020-04-27 07:27:45.981 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is
> > 45470103 and actual is checksum 235515609. expected length is 508 and
> > actual length is 508
> > 2020-04-27 07:27:45.983 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected
> > checksum is 408594907 and actual is checksum 2650340971. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:45.985 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected
> > checksum is 761408352 and actual is checksum 685575313. expected length is
> > 35256 and actual length is 31644
> > 2020-04-27 07:27:45.987 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is
> > 811204953 and actual is checksum 1244308773. expected length is 107460 and
> > actual length is 107460
> > 2020-04-27 07:27:45.989 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is
> > 3813441918 and actual is checksum 2133371346. expected length is 107460 and
> > actual length is 106360
> > 2020-04-27 07:27:45.991 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is
> > 4263416408 and actual is checksum 3722461284. expected length is 16698 and
> > actual length is 21392
> > 2020-04-27 07:27:45.993 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected
> > checksum is 632422377 and actual is checksum 501799146. expected length is
> > 110 and actual length is 110
> > 2020-04-27 07:27:45.994 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected
> > checksum is 3170165892 and actual is checksum 4152937488. expected length
> > is 25090 and actual length is 32328
> > 2020-04-27 07:27:46.000 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected
> > checksum is 2679120655 and actual is checksum 3243853760. expected length
> > is 553 and actual length is 726
> > 2020-04-27 07:27:46.001 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected
> > checksum is 2991034554 and actual is checksum 1980969905. expected length
> > is 321785 and actual length is 322117
> > 2020-04-27 07:27:46.004 WARN
> > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> > o.a.s.h.IndexFetcher File _
Re: SolrCloud degraded during backup and batch CSV update
Posted by Ganesh Sethuraman <ga...@gmail.com>.
Any other JVM settings change possible?
On Tue, Apr 28, 2020, 10:15 PM Sethuraman, Ganesh
<Se...@dnb.com.invalid> wrote:
> Hi
>
> We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92
> collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes,
> with JVM size of 18GB (G1 GC). We need your help with the Issue we faced
> today: The issue is SolrCloud server went into a degraded collections (for
> few collections) when the Solr backup and the Solr batch CSV update load
> happened at the same time as backup. The CSV data load was about ~5 GB per
> shard/replica. We think this happened after zkClient disconnect happened as
> noted below. We had to restart Solr to bring it back to normal.
>
>
> 1. Is it not suggested to run backup and Solr batch CSV update large
> load at the same time?
> 2. In the past we have seen two CSV batch update load in parallel
> causes issues, is this also not suggested (this issue is not related to
> that)?
> 3. Do you think we should increase Zookeeper timeout?
> 4. How do we know if we need to up the JVM Max memory, and by how much?
> 5. We also see that once the Solr goes into degraded collection and
> recovery failed, it NEVER get back to normal, even after when there is no
> load. Is this a bug?
>
> The GC information and Solr Log below
>
>
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB
>
>
> 2020-04-27 07:34:07.322 WARN
> (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181))
> [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server
> in 10775ms for sessionid 0x171a6fb51310008
> ....
> 2020-04-27 07:34:07.426 WARN
> (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread)
> [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
>
>
>
>
> SOLR Log Below (Curtailed WARN log)
> --------------------
> 2020-04-27 07:26:45.402 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.405 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.406 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.408 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.410 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.412 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error fetching file, doing one
> retry...:org.apache.solr.common.SolrException: Unable to download
> _1kr_r.liv completely. Downloaded 0!=587
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.414 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of
> 587 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.414 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv
> java.nio.file.NoSuchFileException:
> /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
> at
> org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
> at
> org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119)
> at
> org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-04-27 07:26:45.449 WARN
> (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is
> 3934327071 and actual is checksum 3095753427. expected length is 33235371
> and actual length is 54180642
> 2020-04-27 07:26:45.449 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is
> 138861107 and actual is checksum 188997082. expected length is 107460 and
> actual length is 107460
> 2020-04-27 07:26:45.457 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected
> checksum is 2947323803 and actual is checksum 979431341. expected length is
> 32176 and actual length is 26089
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected
> checksum is 2087731087 and actual is checksum 3540174119. expected length
> is 110 and actual length is 110
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3)
> [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1]
> o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> 777834759 and actual is checksum 289576304. expected length is 59980991 and
> actual length is 6236424
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is
> 1321210176 and actual is checksum 224636564. expected length is 114 and
> actual length is 106
> 2020-04-27 07:26:45.521 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is
> 3790133021 and actual is checksum 2516047051. expected length is 1477755
> and actual length is 1477850
> 2020-04-27 07:26:45.578 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected
> checksum is 3690299979 and actual is checksum 411724865. expected length is
> 4392270 and actual length is 4396498
> 2020-04-27 07:26:45.588 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected
> checksum is 937224718 and actual is checksum 1176441880. expected length is
> 110 and actual length is 110
> 2020-04-27 07:26:45.733 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected
> checksum is 1429458386 and actual is checksum 4155623862. expected length
> is 2213909 and actual length is 2213939
> 2020-04-27 07:26:45.754 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.ReplicationHandler Index fetch failed
> :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv
> completely. Downloaded 0!=587
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.c.RecoveryStrategy Error while trying to
> recover:org.apache.solr.common.SolrException: Replication for recovery
> failed.
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228)
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-27 07:26:45.755 ERROR
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> 2020-04-27 07:26:45.780 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected
> checksum is 1474431645 and actual is checksum 1271168044. expected length
> is 275570 and actual length is 274506
> 2020-04-27 07:26:45.834 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is
> 4129522944 and actual is checksum 4080519415. expected length is 6626 and
> actual length is 6626
> 2020-04-27 07:26:45.843 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is
> 2493366746 and actual is checksum 31587394. expected length is 546 and
> actual length is 546
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> 1392813257 and actual is checksum 3455988143. expected length is 103 and
> actual length is 111
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected
> checksum is 2301226889 and actual is checksum 29696594. expected length is
> 86459 and actual length is 85786
> 2020-04-27 07:26:45.882 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected
> checksum is 112676849 and actual is checksum 354222195. expected length is
> 286474 and actual length is 286546
> 2020-04-27 07:26:45.911 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> 2203892485 and actual is checksum 1165086749. expected length is 107460 and
> actual length is 106580
> 2020-04-27 07:26:45.911 WARN
> (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23)
> [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20]
> o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is
> 3175370827 and actual is checksum 520836483. expected length is 51396 and
> actual length is 51404
> 2020-04-27 07:26:45.912 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is
> 967260408 and actual is checksum 969726862. expected length is 3180 and
> actual length is 3180
> 2020-04-27 07:26:45.914 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> 719562508 and actual is checksum 956062095. expected length is 16717 and
> actual length is 21454
> 2020-04-27 07:26:45.915 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected
> checksum is 3833337361 and actual is checksum 493358852. expected length is
> 43996 and actual length is 44026
> 2020-04-27 07:26:45.916 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> checksum is 1533628328 and actual is checksum 3735455395. expected length
> is 110 and actual length is 110
> 2020-04-27 07:26:45.917 WARN
> (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19)
> [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16]
> o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected
> checksum is 75152145 and actual is checksum 1665090946. expected length is
> 497241188 and actual length is 497254814
> 2020-04-27 07:26:45.918 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> checksum is 1349491281 and actual is checksum 3808036883. expected length
> is 25153 and actual length is 32322
> 2020-04-27 07:26:45.920 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> checksum is 3820733560 and actual is checksum 2819887160. expected length
> is 571 and actual length is 725
> 2020-04-27 07:26:46.034 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> checksum is 1547030244 and actual is checksum 1122196351. expected length
> is 324313 and actual length is 323553
> 2020-04-27 07:26:46.170 WARN
> (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35)
> [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32]
> o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> 751850967 and actual is checksum 1026527313. expected length is 508 and
> actual length is 508
> and actual length is 319133
> 2020-04-27 07:27:30.567 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected
> checksum is 3435029878 and actual is checksum 1202327807. expected length
> is 3337719 and actual length is 3375106
> 2020-04-27 07:27:30.578 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is
> 2264443519 and actual is checksum 2299249501. expected length is 104 and
> actual length is 102
> 2020-04-27 07:27:30.580 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is
> 1184027188 and actual is checksum 2810465617. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.622 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is
> 2246788287 and actual is checksum 2682862227. expected length is 92 and
> actual length is 96
> 2020-04-27 07:27:30.625 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is
> 3227520043 and actual is checksum 2778425164. expected length is 107137 and
> actual length is 107137
> 2020-04-27 07:27:30.627 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is
> 1964200539 and actual is checksum 3997289541. expected length is 7398 and
> actual length is 9042
> 2020-04-27 07:27:30.630 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected
> checksum is 2165407577 and actual is checksum 2563543943. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.632 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected
> checksum is 1427836076 and actual is checksum 225462388. expected length is
> 11162 and actual length is 13623
> 2020-04-27 07:27:30.634 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected
> checksum is 2998276982 and actual is checksum 1966428087. expected length
> is 296 and actual length is 353
> 2020-04-27 07:27:30.637 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected
> checksum is 2717609594 and actual is checksum 673086945. expected length is
> 315491 and actual length is 312531
> 2020-04-27 07:27:30.640 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is
> 2885031416 and actual is checksum 762513018. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.643 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected
> checksum is 602612626 and actual is checksum 2826680041. expected length is
> 1292092 and actual length is 1655171
> 2020-04-27 07:27:30.669 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected
> checksum is 401385741 and actual is checksum 3373755803. expected length is
> 621 and actual length is 715
> 2020-04-27 07:27:30.671 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is
> 3708405765 and actual is checksum 1057460545. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.674 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected
> checksum is 2755907323 and actual is checksum 1931314059. expected length
> is 327125 and actual length is 318737
> 2020-04-27 07:27:30.677 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is
> 77505520 and actual is checksum 2602905677. expected length is 107240 and
> actual length is 107460
> 2020-04-27 07:27:30.680 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected
> checksum is 4204960097 and actual is checksum 2276943505. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.682 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is
> 3414743450 and actual is checksum 3890346633. expected length is 18415 and
> actual length is 21922
> 2020-04-27 07:27:30.685 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected
> checksum is 2879829129 and actual is checksum 790212338. expected length is
> 27860 and actual length is 33235
> 2020-04-27 07:27:30.687 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is
> 3356980932 and actual is checksum 1544608917. expected length is 108 and
> actual length is 112
> 2020-04-27 07:27:30.690 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected
> checksum is 1979425912 and actual is checksum 584894070. expected length is
> 3726110 and actual length is 4689653
> 2020-04-27 07:27:30.700 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is
> 3754169108 and actual is checksum 527001228. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.702 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected
> checksum is 3981710860 and actual is checksum 1749011618. expected length
> is 1938123 and actual length is 33694
> 2020-04-27 07:27:30.710 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is
> 1515379146 and actual is checksum 3212722427. expected length is 98 and
> actual length is 84
> 2020-04-27 07:27:30.712 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is
> 1819400085 and actual is checksum 3372088644. expected length is 107020 and
> actual length is 103887
> 2020-04-27 07:27:30.715 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected
> checksum is 1401562358 and actual is checksum 2224139078. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.718 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected
> checksum is 3386701421 and actual is checksum 1434467863. expected length
> is 15944 and actual length is 685
> 2020-04-27 07:27:30.720 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is
> 4259543754 and actual is checksum 3203989793. expected length is 10616 and
> actual length is 439
> 2020-04-27 07:27:30.722 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected
> checksum is 725882371 and actual is checksum 3650374401. expected length is
> 383 and actual length is 114
> 2020-04-27 07:27:30.725 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected
> checksum is 3717717087 and actual is checksum 3712473868. expected length
> is 303729 and actual length is 117582
> 2020-04-27 07:27:30.728 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is
> 3506718582 and actual is checksum 342046803. expected length is 93 and
> actual length is 104
> 2020-04-27 07:27:30.731 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is
> 2920971663 and actual is checksum 2142622028. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.733 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected
> checksum is 723455769 and actual is checksum 2731538349. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:30.735 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected
> checksum is 2624342764 and actual is checksum 1283943229. expected length
> is 11424 and actual length is 26688
> 2020-04-27 07:27:30.738 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected
> checksum is 978416968 and actual is checksum 1807767534. expected length is
> 330 and actual length is 597
> 2020-04-27 07:27:30.740 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected
> checksum is 3599774596 and actual is checksum 2764190632. expected length
> is 318017 and actual length is 311617
> 2020-04-27 07:27:30.743 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected
> checksum is 3141314417 and actual is checksum 3847788889. expected length
> is 1258075 and actual length is 3539994
> 2020-04-27 07:27:30.749 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is
> 1404317797 and actual is checksum 2471843701. expected length is 105295 and
> actual length is 107020
> 2020-04-27 07:27:30.752 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is
> 3412288246 and actual is checksum 922481525. expected length is 7580 and
> actual length is 17733
> 2020-04-27 07:27:30.754 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected
> checksum is 2675540020 and actual is checksum 1452209674. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.757 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected
> checksum is 665797990 and actual is checksum 2134373365. expected length is
> 10715 and actual length is 3418
> 2020-04-27 07:27:30.759 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is
> 2006427884 and actual is checksum 527297847. expected length is 7178 and
> actual length is 2326
> 2020-04-27 07:27:30.761 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is
> 200552893 and actual is checksum 3290522141. expected length is 106580 and
> actual length is 105206
> 2020-04-27 07:27:30.764 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected
> checksum is 1511033560 and actual is checksum 3689083691. expected length
> is 278 and actual length is 154
> 2020-04-27 07:27:30.767 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected
> checksum is 1435050119 and actual is checksum 6864243. expected length is
> 323217 and actual length is 338063
> 2020-04-27 07:27:30.770 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is
> 1107038327 and actual is checksum 3068870234. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.772 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is
> 2853806777 and actual is checksum 459866651. expected length is 93 and
> actual length is 87
> 2020-04-27 07:27:30.775 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected
> checksum is 1356537478 and actual is checksum 2594306289. expected length
> is 1179215 and actual length is 249458
> 2020-04-27 07:27:30.780 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is
> 1785795479 and actual is checksum 65014613. expected length is 106580 and
> actual length is 106800
> 2020-04-27 07:27:30.783 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected
> checksum is 1824666616 and actual is checksum 2857527783. expected length
> is 507 and actual length is 637
> 2020-04-27 07:27:30.785 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is
> 1384446542 and actual is checksum 3981590407. expected length is 13686 and
> actual length is 18516
> 2020-04-27 07:27:30.788 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected
> checksum is 3380549662 and actual is checksum 2002436369. expected length
> is 317777 and actual length is 319213
> 2020-04-27 07:27:30.791 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected
> checksum is 1594954666 and actual is checksum 1892723425. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.794 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected
> checksum is 305271241 and actual is checksum 854278263. expected length is
> 20522 and actual length is 27836
> 2020-04-27 07:27:30.796 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is
> 2220527769 and actual is checksum 2016379681. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.799 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is
> 1648054590 and actual is checksum 137025071. expected length is 100 and
> actual length is 105
> 2020-04-27 07:27:30.801 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected
> checksum is 262113808 and actual is checksum 327562225. expected length is
> 2731808 and actual length is 3909012
> 2020-04-27 07:27:30.809 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected
> checksum is 2157667750 and actual is checksum 3043505208. expected length
> is 36906 and actual length is 163443
> 2020-04-27 07:27:30.812 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is
> 3456088243 and actual is checksum 1715362430. expected length is 453 and
> actual length is 1636
> 2020-04-27 07:27:30.814 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is
> 4028738113 and actual is checksum 4245480574. expected length is 104876 and
> actual length is 105515
> 2020-04-27 07:27:30.817 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected
> checksum is 3120632072 and actual is checksum 1945528118. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:30.819 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected
> checksum is 3303166970 and actual is checksum 1740028553. expected length
> is 114 and actual length is 137
> 2020-04-27 07:27:30.822 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected
> checksum is 126700369 and actual is checksum 162509958. expected length is
> 700 and actual length is 2398
> 2020-04-27 07:27:30.825 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is
> 3157051968 and actual is checksum 137679212. expected length is 84 and
> actual length is 86
> 2020-04-27 07:27:30.827 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected
> checksum is 3969090178 and actual is checksum 2364978843. expected length
> is 127576 and actual length is 266581
> 2020-04-27 07:27:30.830 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is
> 3730892058 and actual is checksum 2734980133. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:30.832 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is
> 3259266514 and actual is checksum 2051283039. expected length is 104855 and
> actual length is 106819
> 2020-04-27 07:27:30.835 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected
> checksum is 380153843 and actual is checksum 580580220. expected length is
> 1822 and actual length is 1737
> 2020-04-27 07:27:30.838 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected
> checksum is 4094454161 and actual is checksum 421930789. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:30.840 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is
> 911575783 and actual is checksum 2090965125. expected length is 1244 and
> actual length is 1201
> 2020-04-27 07:27:30.842 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected
> checksum is 1279063865 and actual is checksum 765869778. expected length is
> 128 and actual length is 130
> 2020-04-27 07:27:30.845 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected
> checksum is 1689040450 and actual is checksum 627554920. expected length is
> 223473 and actual length is 230157
> 2020-04-27 07:27:30.848 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is
> 1278110969 and actual is checksum 1444534251. expected length is 86 and
> actual length is 86
> 2020-04-27 07:27:30.850 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected
> checksum is 1992192696 and actual is checksum 3455180678. expected length
> is 119844 and actual length is 117860
> 2020-04-27 07:27:30.853 WARN
> (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55)
> [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52]
> o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is
> 2776236859 and actual is checksum 2572132550. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:35.866 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is
> 2573424453 and actual is checksum 3612352961. expected length is 546 and
> actual length is 437
> 2020-04-27 07:27:35.868 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is
> 2592320603 and actual is checksum 1374033039. expected length is 289 and
> actual length is 289
> 2020-04-27 07:27:35.872 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is
> 2911257988 and actual is checksum 2011202433. expected length is 437 and
> actual length is 437
> 2020-04-27 07:27:35.874 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is
> 1304802713 and actual is checksum 454030092. expected length is 7598425 and
> actual length is 49265169
> 2020-04-27 07:27:37.234 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is
> 1422458436 and actual is checksum 703149874. expected length is 437 and
> actual length is 437
> 2020-04-27 07:27:38.117 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is
> 1892273984 and actual is checksum 2087644984. expected length is 50710602
> and actual length is 10402264
> 2020-04-27 07:27:38.532 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is
> 3864394245 and actual is checksum 1846889307. expected length is 289 and
> actual length is 289
> 2020-04-27 07:27:38.575 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is
> 660375856 and actual is checksum 571052523. expected length is 114 and
> actual length is 84
> 2020-04-27 07:27:38.590 WARN
> (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7)
> [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4]
> o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744
> deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536}
> 2020-04-27 07:27:39.259 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected
> checksum is 421440641 and actual is checksum 2075160524. expected length is
> 337361 and actual length is 141820
> 2020-04-27 07:27:44.881 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected
> checksum is 47189924 and actual is checksum 3640637009. expected length is
> 5235629 and actual length is 48679
> 2020-04-27 07:27:45.335 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected
> checksum is 2305624674 and actual is checksum 704932512. expected length is
> 693 and actual length is 115
> 2020-04-27 07:27:45.939 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is
> 1157089232 and actual is checksum 2388254324. expected length is 23435 and
> actual length is 673
> 2020-04-27 07:27:45.941 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is
> 2811885732 and actual is checksum 2854129172. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:45.943 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected
> checksum is 1925626873 and actual is checksum 1921615605. expected length
> is 110 and actual length is 110
> 2020-04-27 07:27:45.945 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected
> checksum is 3627283142 and actual is checksum 3796132842. expected length
> is 34596 and actual length is 958
> 2020-04-27 07:27:45.947 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is
> 1291024098 and actual is checksum 2700484857. expected length is 107020 and
> actual length is 104209
> 2020-04-27 07:27:45.949 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected
> checksum is 2785033346 and actual is checksum 1911387906. expected length
> is 816 and actual length is 760
> 2020-04-27 07:27:45.951 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected
> checksum is 942596692 and actual is checksum 3039336477. expected length is
> 5557044 and actual length is 4847564
> 2020-04-27 07:27:45.974 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is
> 143536399 and actual is checksum 2962535892. expected length is 116 and
> actual length is 111
> 2020-04-27 07:27:45.976 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected
> checksum is 3884436961 and actual is checksum 1345841996. expected length
> is 339801 and actual length is 336809
> 2020-04-27 07:27:45.980 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is
> 3902419774 and actual is checksum 3791519088. expected length is 23306 and
> actual length is 21047
> 2020-04-27 07:27:45.981 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is
> 45470103 and actual is checksum 235515609. expected length is 508 and
> actual length is 508
> 2020-04-27 07:27:45.983 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected
> checksum is 408594907 and actual is checksum 2650340971. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:45.985 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected
> checksum is 761408352 and actual is checksum 685575313. expected length is
> 35256 and actual length is 31644
> 2020-04-27 07:27:45.987 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is
> 811204953 and actual is checksum 1244308773. expected length is 107460 and
> actual length is 107460
> 2020-04-27 07:27:45.989 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is
> 3813441918 and actual is checksum 2133371346. expected length is 107460 and
> actual length is 106360
> 2020-04-27 07:27:45.991 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is
> 4263416408 and actual is checksum 3722461284. expected length is 16698 and
> actual length is 21392
> 2020-04-27 07:27:45.993 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected
> checksum is 632422377 and actual is checksum 501799146. expected length is
> 110 and actual length is 110
> 2020-04-27 07:27:45.994 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected
> checksum is 3170165892 and actual is checksum 4152937488. expected length
> is 25090 and actual length is 32328
> 2020-04-27 07:27:46.000 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected
> checksum is 2679120655 and actual is checksum 3243853760. expected length
> is 553 and actual length is 726
> 2020-04-27 07:27:46.001 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected
> checksum is 2991034554 and actual is checksum 1980969905. expected length
> is 321785 and actual length is 322117
> 2020-04-27 07:27:46.004 WARN
> (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr
> x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51)
> [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48]
> o.a.s.h.IndexFetcher File _